Publication and related bias in quantitative health services and delivery research: a multimethod study

Article history

The research reported in this issue of the journal was funded by the HS&DR programme or one of its preceding programmes as project number 15/71/06. The contractual start date was in January 2017. The final report began editorial review in July 2019 and was accepted for publication in November 2019. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Iestyn Williams was a member of the Health Services and Delivery Research Prioritisation Committee (Commissioned) (2015–19). Magdalena Skrybant and Richard J Lilford are also supported by the National Institute for Health Research Applied Research Collaboration West Midlands.

Copyright statement

© Queen’s Printer and Controller of HMSO 2020. This work was produced by Ayorinde et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2020 Queen’s Printer and Controller of HMSO

Publication and related bias

Publication bias refers to the phenomenon by which research findings that are statistically significant and/or are perceived to be interesting or favourable are more likely to be published than those that are not statistically significant or perceived to be interesting. Publication bias occurs at study level (i.e. when a study is not published at all owing to the nature of its findings). Similar bias may manifest at outcome level (i.e. a study may have evaluated several outcomes, but only the findings related to a subset of the outcomes that are statistically significant or judged to be worth noting are reported in its publication). This is termed ‘outcome reporting bias’. 1 Bias can also occur when researchers carry out analyses of data collected from research. For example, multiple analyses may be performed using different techniques or different subsets of the data until statistically significant results are obtained, which are then published. This is referred to as ‘data dredging’ or ‘p-hacking’. 2,3 These biases are a major threat to evidence-based decision-making, as they distort the full picture of the ‘true’ evidence gathered from research and may lead to misinformed decisions. Here we use the term ‘publication and related bias’ to refer to these biases collectively (Figure 1). Further bias may occur following the publication of research findings, such as citation bias and media attention bias, which alongside publication and related bias are collectively known as dissemination bias. 4 For the present study, we focus on publication and related bias that occurs up to the stage of publication (see Figure 1), as biases that occur following publication can largely be overcome through systematic literature search, which is becoming a standard practice when evidence is synthesised to support decision-making.

FIGURE 1.

Various forms of bias in the process of research publication. See Glossary for the definition of ‘winner’s curse’.

A large body of literature has demonstrated the existence of publication and related bias in clinical research. 1,4,5 In particular, several high-profile cases highlighting the non-publication and/or under-reporting of unfavourable results from clinical trials of new pharmaceuticals have raised ethical concerns over publication bias and have highlighted the potential harm it may cause to patients. 6–8 These concerns have led to mandatory clinical trial registration9,10 and, more recently, mandatory reporting of findings of registered trials. 11 By contrast, health services and delivery research (HSDR) has not been subject to similar levels of regulation and scrutiny, and the issue of publication bias appears to be largely ignored. Our preliminary scoping of the health services research literature found only a small number studies addressing this topic, which suggests the possible existence of publication bias. 12–15 The paucity of documented evidence was affirmed in our discussions with some leading experts in health services research.

The lack of published literature on publication and related bias in HSDR compared with clinical research is surprising, given that publication bias has also been documented in many other scientific disciplines, such as social science,16 management research17 and ecology. 18 As there is no obvious reason to believe that HSDR is immune to publication and related bias, and such bias could have substantial implications for health service decisions relying on HSDR, we set out to explore the existence and extent of publication and related bias in this field in order to inform best practice in the reporting and synthesis of evidence from HSDR.

Aims and objectives

The primary aims of this study were to gather prima facie evidence of the existence and potential impact of publication and related bias in HSDR, to examine current practice in systematic reviews of HSDR literature, and to explore common methods in relation to the detection and mitigation of the bias.

The above aims were to be achieved through five distinct but inter-related work packages (WPs), each with a specific objective:

1.  WP 1 – a systematic review of empirical and methodological studies concerning the occurrence, potential impact and/or methodology related to publication bias in HSDR, to provide a summary of what is known from current literature.

2.  WP 2 – an overview of systematic reviews of intervention and association studies in HSDR, to describe current practice and potential challenges in assessing publication bias during evidence synthesis.

3.  WP 3 – in-depth case studies to evaluate the applicability of different methods for detecting and mitigating publication bias in HSDR and to provide guidance for future research and practice.

4.  WP 4 – a retrospective study to follow up the publication status of cohorts of HSDR studies to directly observe publication bias in HSDR.

5.  WP 5 – semistructured interviews with health services researchers and commissioners, journal editors and service managers, and a focus group discussion with patient representatives to explore their perception and experience related to publication bias.

These WPs complemented each other to provide a full picture of both empirical evidence and current practices related to publication bias in HSDR (Figure 2). Detailed methods for the WPs are described in the next chapter.

FIGURE 2.

Overview of the five complementary WPs included in this study. HSRProj, Health Services Research Projects in Progress; NIHR, National Institute for Health Research.

Structure of the report

The rest of this report is organised as follows: Chapter 2 details the scope and methods of each of the five WPs described above. Chapters 3–7 present the findings from each of the WPs. Chapter 8 includes discussions of specific issues related to methods and findings of individual WPs, and provides some discourse of findings across the WPs and wider issues related to use of evidence to inform decision-making in health services. The chapter concludes with issues worth considering by different stakeholders of HSDR in relation to publication and related bias, and recommendations for future research.

Scope

The subject area of HSDR is very broad. In order to draw a boundary that allowed a focused investigation, we used the definition adopted by the National Institute for Health Research (NIHR) and defined HSDR as ‘research that produces evidence on the quality, accessibility and organisation of health services’. In addition, we targeted two types of quantitative studies: (1) intervention studies, which were carried out to evaluate interventions to improve and optimise the effectiveness and/or efficiency of the delivery of health services; and (2) association studies, which were carried out to evaluate associations between different variables along the service delivery causal chain. 19 A large number of variables can be covered in association studies in HSDR. These include structure variables (e.g. characteristics of a hospital, nurse–patient ratio); generic processes (e.g. continuous professional development, institutional human resource policy); intervening variables (e.g. safety culture, staff knowledge and morale, etc., that could be influenced by structure and generic processes and then impact on many downstream processes);19 targeted processes (e.g. door-to-balloon time for treating myocardial infarction, adherence to guidelines for management of patients with diabetes); and health service utilisation, patient, carer or health care provider outcomes and context (e.g. weekdays vs. weekends, low- and middle-income countries vs. high-income countries). We recognised that intervention studies cannot be completely separated from association studies, as the former is a special case of the latter. Nevertheless, such classification reflected how research questions are often asked in HSDR (e.g. whether or not an intervention works vs. whether or not certain factors affect one another or influence outcomes in the health system). We hypothesised that association studies may be more vulnerable to selective publication and reporting than intervention studies, as a causal relationship is assumed between an intervention and outcomes, whereas relationships between different factors examined in association studies are exploratory and not necessarily causal. In addition, evaluation of interventions may be more likely to be specifically funded with a mandate from the funder to disseminate results, whereas association studies may be carried out without specific funding and related incentive for publication.

The criteria for selecting intervention and association studies were applied in conjunction with the definition of HSDR described in the previous paragraph. Eligible studies may focus on any aspects of health systems and health policy, health-care organisations, people who organise and deliver the health services, and users and carers of the services, as well as related processes, outcomes and contextual factors. Studies concerning clinical research and health technology assessment (i.e. those focusing on interventions applied directly to individual patients), disease epidemiology and genetic associations have previously been examined in detail4 and, therefore, were not included in this study. We were aware of potential grey areas in which the boundary between HSDR and non-HSDR studies may be vague. These were dealt with by consulting members of the Project Management Group and Study Steering Committee.

A wide variety of research designs, including quantitative, qualitative and mixed-methods research, have been used in HSDR. 20,21 This study focused on quantitative research and mixed-methods research that incorporated an element of quantitative estimation of intervention effects or association, although we acknowledged that qualitative research can also be subject to publication bias. 22 As the mechanisms and manifestation of publication bias for qualitative research are likely to be different, and methods for evaluating its occurrence and impact are not well developed, we felt that issues related to qualitative research were beyond the scope of the study and warranted a separate investigation. 23

Methods for assessing publication and related bias

Many methods have been developed in order to detect publication and related bias and estimate and/or mitigate its potential impact. 4,24,25 Methods used to detect publication bias can be broadly classified as either making indirect inference or employing direct observation.

Several statistical methods, such as funnel plots and related regression tests, can suggest that publication and related bias may be present during the synthesis of evidence across many studies. 24,25 However, these widely used methods allow only indirect inference, as they rely on the identification of specific patterns in the findings across studies that are suggestive of publication bias, but cannot rule out alternative causes. 26 As a result, evidence on publication and related bias obtained using these methods is generally weak and could be misleading if assumptions underlying these methods do not hold.

Direct evidence on publication and related bias can be obtained in two ways:

1. Identifying a cohort of studies (usually through a registry) and then following these up over time to determine whether or not they are published; this is sometimes described as an inception cohort study and is often undertaken retrospectively given the length of time between the inception of a research and publication of its findings (if published).

2. Consulting stakeholders who are involved in generating and/or disseminating research evidence through surveys or interviews to find out their experience.

These methods of direct observation are labour intensive, but may provide the strongest evidence on the presence (or absence) of publication and related bias. A major challenge in HSDR lies with the difficulty in identifying study cohorts owing to the lack of registries.

Among the five WPs included in this study, WP 1 systematically reviewed previous studies that set out to investigate publication and related bias in any substantive areas of HSDR, using either direct or indirect methods. WP 2 examined a random sample of systematic reviews of HSDR topics with regard to their practice in assessing publication and related bias. WP 3 explored issues surrounding commonly used methods for detecting publication and related bias in more depth using case studies. WP 4 identified four cohorts of HSDR studies from various sources and followed them up to check their publication status and to explore if this was associated with the direction or strength of study findings. WP 5 gathered perceptions and experiences of HSDR stakeholders on this issue through interviews and a focus group discussion. The methods for each WP are described in detail below.

Work package 1: a systematic review of empirical evidence on publication bias in HSDR

Work package 2: overview of current practice and findings associated with publication and related bias in systematic reviews of intervention and association studies in HSDR

Systematic reviews have emerged in various fields, including HSDR, as a key tool for summarising the rapidly expanding evidence base in a way that maximises the completeness, while minimising potential bias in their coverage of relevant evidence. Steps to identify and reduce various types of bias are built into the process of a systematic review. The following steps are particularly relevant for publication and related bias:

• comprehensive search of literature, including attempts to locate unpublished studies

• assessment of outcome reporting bias of included studies

• assessment of potential publication bias using funnel plots, related regression methods or other techniques.

This WP examined a random sample of 200 HSDR systematic reviews and inspected each review with regard to whether or not the above measures for minimising and detecting publication and related bias were considered and/or performed. Data were also collected on the methods used and findings from the assessment of publication and related bias. We focused on systematic reviews covering two main types of quantitative study: intervention studies and association studies (see Scope). We explored whether or not the practice of assessing publication and related bias differed between these two types of studies, and whether or not this was associated with various characteristics of the review and the journal in which it was published.

Work package 3: case studies to explore the applicability of methods for detecting and dealing with publication and related bias

Several methods have been developed to facilitate the detection and potential adjustment of publication and related bias. Among these, funnel plots and related regression methods are the most widely used and have been adopted in many systematic reviews. The key assumption for these methods is that the precision of a study (mainly determined by its sample size) is not correlated with the actual size of the intervention effect or association being estimated and, hence, the results of smaller studies are scattered more widely due to random variation, forming an inverse funnel shape when plotted against precision. Asymmetry in a funnel plot would suggest possible publication bias. Figure 3 shows an example of an asymmetrical funnel plot compiled (by authors of this report) using data from a published systematic review of mortality risk associated with out-of-hours admissions in patients with myocardial infarction. 36

FIGURE 3.

An example of an asymmetrical funnel plot.

Although the assumption behind funnel plots and related regression methods holds for many clinical interventions, this is not necessarily true in many HSDR studies. For example, early evaluation of a quality improvement intervention in a small number of sites may observe a large intervention effect due to the expertise and dedication of the personnel and thoroughness of implementation, which may be difficult to maintain when the intervention is scaled up in a larger study. Alternatively, an intervention that appears to be highly effective in early small-scale studies may have an apparently diminished intervention effect by the time it is subject to a large-scale evaluation due to a system-wide improvement triggered by the same social pressure that prompted the intervention. 37 On the other hand, the availability of data from large databases covering nearly the whole population may render the influence of small studies negligible. These different types of heterogeneity arising from the complexity of HSDR interventions and associations, and the context in which they are deployed and observed, pose a potential threat for the validity of applying these conventional methods. In addition, funnel plots and related regression methods require a sufficiently large number of studies (e.g. ≥ 10), which may not be available for many topics in HSDR.

The WP 2 described above allowed us to obtain an overview of current practice of examining publication and related bias in systematic reviews of HSDR, including a description of if and what methods have been used. Nevertheless, when formal methods, such as funnel plots and related regression methods, have been used, there remain potential issues concerning the validity and applicability of these methods. 26,38 WP 3 aimed to address these issues through more detailed case studies. In addition, WP 3 offered an opportunity to explore novel methods, such as the p-curve for identifying p-hacking (see Investigation of p-hacking using p-curves), which could be very relevant for HSDR.

Work package 4: follow-up of publication status of cohorts of health services research studies

The previous three WPs drew on crucial evidence on issues concerning the extent of publication bias and methods of detecting it in HSDR from the literature. Nevertheless, most of the evidence gathered was indirect in nature, as observations made (such as asymmetry in funnel plots and significant tests) were indicative of the existence of such bias, rather than confirmatory. WP 4 consisted of a retrospective investigation of cohorts of HSDR studies, which were followed over time to ascertain whether their publication status was associated with the statistical significance, or perceived ‘positivity’ or interest of their findings. The main objective was to provide a direct observation of the presence or absence of publication bias in HSDR, as measured by the presence or absence of an association between the publication status of HSDR projects and the statistical significance and perceived ‘positivity’ (see Extraction of study information and classification of study findings) of their findings. In addition, if publication bias was observed, whether it was associated with study design, study type (intervention vs. association) and/or sample size would be explored.

Work package 5: semistructured interviews and a focus group discussion with health services researchers, journal editors and other stakeholders

Work package 5 sought to complement direct evidence collected from the retrospective cohort study in WP 4 by exploring the perceptions and first-hand experiences of health services researchers and commissioners, journal editors, service managers and users, with regard to the occurrence and impact of publication and related bias. It contributes to the overall aim of obtaining (qualitative) evidence on the extent and existence of publication bias. It was also intended to contribute to the development of methods for the detection and mitigation of publication bias in HSDR. As well as generating important data on the perspectives of key actors in the HSDR process, this WP was designed to support analysis of results deriving from prior WPs.

Patient and public involvement

This project involved two public contributors from its inception. One of the public representatives (MS) was a member of the Project Management Group and co-author of the report, and helped with planning and facilitating the focus group discussion for WP 5. Another public representative sat on the Study Steering Committee. Both members of the public regularly participated in project meetings and discussions, received meeting minutes, provided advice on all issues related to patient and public involvement (PPI) and dissemination of findings. The project also benefited from input from NIHR Collaboration for Leadership in Applied Health Research and Care West Midlands PPI Supervisory Committee Advisors.

Deviations from the original protocol

This 2-year project tackled a complex area of HSDR, which is very broad in scope and diverse in methods. Several challenges needed to be overcome in order to deliver the proposed research within the time and resources available. As a consequence, some amendments of the methodological approaches were made during the conduct of the project, which deviated from the original protocol. These are described below with rationale behind the changes explicated.

Literature search and study selection

Initial searches of electronic databases in March 2017 retrieved 7732 records. After removing duplicates, the titles and abstracts of 6155 records were screened. Of these, 422 records were considered potentially relevant and their full-text articles were retrieved. Six additional full-text articles were identified and obtained from other sources.

Of the 428 full-text articles examined, 188 were retained. Of these, four were methodological studies that set out specifically to investigate publication and related bias in HSDR13–15,50 and three were systematic reviews of substantive HSDR topics, in which evidence from published literature was compared with grey literature and unpublished studies (and thus provided direct evidence on publication bias). 51–53 These seven studies were examined and are described in detail in this chapter. The remaining 181 studies were systematic reviews of substantive HSDR topics, in which publication and outcome reporting bias was assessed as part of the review process. As these reviews provided only indirect evidence on publication bias, they are briefly described in this chapter and are summarised in Appendix 6.

Two hundred and forty retrieved full-text articles did not meet the inclusion criteria and were excluded. The primary reasons for exclusion were not mentioning publication and related bias at all (we examined systematic reviews that might have examined publication and related bias, even if this was not explicitly stated in the titles and abstracts); mentioning these biases but without assessing them; and topics being outside the definition of HSDR adopted for this project. A flow diagram for the literature retrieval and study selection process is shown in Figure 4. We carried out an updated search in July/August 2018. This retrieved 1328 new records, but no relevant methodological studies were identified.

FIGURE 4.

Flow diagram for literature search and study selection process for the systematic review. This figure has been reproduced from Ayorinde et al. 54 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original text.

This figure has been reproduced from Ayorinde et al. 54 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original text.

Methodological studies investigating publication and related bias in HSDR

Four studies specifically set out to investigate publication and related bias in a substantive topic area of HSDR. The objectives, methods, key findings and limitations of these studies are summarised in Table 2. Three studies investigated publication bias in health informatics research,14,15,50 and one study explored potential reporting bias or p-hacking arising from researchers competing for limited space of publication in high-impact journals in health economics and policy literature. 13

Of the four studies,13–15,50 only one was an inception cohort study that tracked individual research projects from the start and thus provided direct evidence of publication bias. 14 Studies included in this cohort were clinical trials of electronic health records registered with ClinicalTrials.gov during 2000–8. Findings from 76% (47/62) of completed studies were published. Of these, 74% (35/47) reported predominantly positive findings, 21% (10/47) reported neutral results (no significant effects) and 4% (2/47) reported negative or harmful results. Of the 15 unpublished trials, three had positive findings and four had neutral results based on information supplied by the investigators. Findings for the remaining eight studies were unknown. The authors found that trials with positive findings were more likely to be published than those with neutral findings (see Table 2), but cautioned that the sample included in the cohort may be atypical of general studies in the field.

Another study15 in health informatics was an e-mail-based survey of people who were likely to be involved in the evaluation of health information systems. Participants were asked about (1) what information systems they had evaluated in the past 3 years; (2) where they published the results of the evaluation; and (3) the reasons for non-publication of the results, if this was the case. A response rate of 19% (136/722) was achieved, with 118 respondents reporting the completion of a total of 217 evaluation studies. Most of these respondents were from an academic background (92/118), with a small number from information technology management, industry and government institutions. Approximately half of the identified evaluation studies were published in peer-reviewed journals, proceedings or books, whereas the remaining half either were published only locally (e.g. internal reports) or were unpublished (see Table 2). Reasons cited for not publishing included: not of global interest (35%), publication in preparation (31%), no time for publication (22%), limited scientific quality (17%), political and legal reasons (14%) and for internal use only (13%).

A low response rate was the major limitation of this study. Nevertheless, the survey provided some insights concerning reasons behind non-publication. Like most surveys, the study findings could be influenced by sampling, response and recall bias. It is also worth noting that publication was still being considered or under way for about one-third of the unpublished studies at the time of the survey, and, therefore, the actual publication rate might be higher.

The third methodological study50 in health informatics utilised evaluation studies identified from a specialist health informatics database that covered literature published between 1982 to 2002, and adopted three different approaches to assessing publication bias: (1) statistical analyses of the small study effect; (2) examination of the percentage of evaluation studies with positive findings compared with percentage of studies with mixed or negative findings; and (3) examination of the percentage of systematic reviews reporting positive, neutral or negative results. The authors did not identify sufficient number of studies with the same outcome measures to carry out statistical analyses of the small study effect. Although the percentages of primary studies with negative findings and systematic reviews reporting negative results were low (see Table 2), these were not good indicators for the existence of publication bias, as there is no estimate of what the ‘unbiased’ proportion of negative findings should be for evaluation studies and reviews of health informatics interventions.

The fourth methodological study13 included in this review examined quantitative estimates of income elasticity of health care and price elasticity of prescription drugs reported in the published health economics literature. 13 The authors identified a positive correlation between effect sizes and the standard errors of income and price elasticity estimates using funnel plots and meta-regression, which suggested potential existence of publication bias. Having adjusted for this, they also found an independent association between effect size and journal impact factor. This suggested that studies reporting larger effect sizes (i.e. more striking findings) were more likely to be published in ‘high-impact’ journals. However, the finding still needs to be interpreted with caution, as other confounding factors could not be ruled out for these observed associations. In addition, the authors acknowledged that studies in the field concerned were often reported in grey literature, which was not examined in this study.

Systematic reviews of substantive HSDR topics providing evidence on publication and related bias

In addition to the four methodological studies13–15,50 that set out to investigate publication and related bias, we identified 184 systematic reviews of substantive HSDR topics, in which findings from assessment of publication bias and outcome reporting bias were reported. Therefore, these systematic reviews, although not undertaken to specifically investigate publication and related bias, also provided empirical evidence on this topic. In particular, three of these reviews51–53 provided direct evidence on publication bias by comparing evidence from studies published in academic journals with evidence from grey literature or unpublished studies. These reviews are described in detail below. The remaining 181 reviews provided only indirect evidence and are summarised briefly in the following section (and further details can be found in Appendix 6).

Characteristics of included intervention and association reviews

As planned, 200 systematic reviews (100 intervention reviews and 100 association reviews) were included in this meta-epidemiological study. The characteristics of the included systematic reviews are shown for both intervention and association reviews in Table 4. The majority of the 200 systematic reviews (79%) included at least 10 studies, with association reviews more likely than intervention reviews to include ≥ 10 studies (86% vs. 71%; p = 0.01). Only 22% of all the systematic reviews performed a meta-analysis, which was more common in the intervention reviews than in the association reviews (33% vs. 10%; p < 0.0001). Of the 157 reviews that did not include meta-analysis, 90 (57%) provided reasons for this (mainly heterogeneity between studies and a small number of comparable studies).

Intervention and association reviews also differ significantly in several other characteristics, such as inclusion of study design beyond controlled trials, undertaking quality assessment of included studies, mentioning the use of systematic review reporting guidelines and GRADE, their A MeaSurement Tool to Assess systematic Reviews (AMSTAR) rating and the impact factors for the journals in which they were published (see Table 4). Just over half (52%) of the systematic reviews searched for grey and unpublished literature, and the proportions were similar in both association reviews (52%) and intervention reviews (51%). Seventy per cent of the journals in which the reviews were published endorsed a reporting guideline for systematic reviews, with similar proportions for both intervention reviews and association reviews (71% vs. 69%; p = 0.758) (see Table 4).

Publication bias

We found that 85 (43%) of the systematic reviews considered publication bias, and this was more common in intervention reviews than in association reviews (54% vs. 31%; p = 0.001). However, only about 10% (19/200) formally assessed publication bias through statistical analysis, mostly using funnel plots and related methods (Figure 5). Again, intervention reviews assessed publication bias more frequently than association reviews (14% vs. 5%; p = 0.03) (see Table 4). Five59–63 of the 19 reviews (26%) that assessed publication bias reported some evidence of publication bias. The remaining reviews mostly reported no risk or low risk of publication bias, and one reported that the funnel plot was not very informative owing to small numbers (the review included only four studies). Authors of reviews in which publication bias was mentioned but not assessed often reported conditions of using funnel plots not being met, especially insufficient number of studies and/or heterogeneity between included studies.

FIGURE 5.

Methods used to assess publication bias in sampled systematic reviews. a, Nineteen reviews assessed publication bias but some reviews used more than one method of assessment.

Factors associated with assessing publication bias

As with mentioning of publication bias, assessment of publication bias was most strongly associated with the inclusion of meta-analysis (OR 112.32, 95% CI 14.35 to 879.03) in the univariable analysis. Intervention reviews were more likely than association reviews to include such an assessment (OR 3.09, 95% CI 1.07 to 8.95). Inclusion of only RCTs and controlled trials, journal impact factor and reviewers reporting the use of systematic review guidelines were also associated with the assessment of publication bias (Table 6). There was no significant association between the assessment of publication bias and the number of included studies, search of grey and unpublished literature, quality assessment of individual studies or journal endorsement of systematic review guidelines. In the multivariable analysis, only the inclusion of meta-analysis and reviewers reporting the use of systematic review guidelines were significantly associated with the assessment of publication bias (see Table 6). The relationship between types of review and assessment of publication bias diminished after they were adjusted for other factors (OR 0.94, 95% CI 0.20 to 4.55). As current methods for assessing publication bias primarily rely on constructing funnel plots and/or performing related statistical tests as part of meta-analyses in which data are available from ≥ 10 studies, we further explored and illustrate the influence of these factors in Figure 6.

TABLE 6 - Factors associated with the assessment of publication bias

IQR, interquartile range.

This table has been adapted from Ayorinde et al. 58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The table includes minor additions and formatting changes to the original text.

Factor	Assessed publication bias, n (row %)		Univariable		Multivariable
	Yes (N = 19)	No (N = 181)	OR (95% CI)	p-value	OR (95% CI)	p-value
Number of included studies, n (%)
< 10 (n = 43, 22%)	2 (5)	41 (95)
≥ 10 (n = 157, 78%)	17 (11)	140 (89)	2.49 (0.55 to 11.22)	0.235	2.21 (0.32 to 15.27)	0.422
Meta-analysis included, n (%)
No (n = 157, 78%)	1 (1)	156 (99)
Yes (n = 43, 22%)	18 (42)	25 (58)	112.32 (14.35 to 879.03)	< 0.0001	84.65 (9.56 to 749.49)	< 0.0001
Design of included studies, n (%)
Mixed (n = 164, 82%)	12 (7)	152 (93)
RCT and controlled trials (n = 36, 18%)	7 (19)	29 (81)	3.06 (1.11 to 8.42)	0.031	Not included
Searched grey/unpublished literature, n (%)
No (n = 97, 49%)	13 (13)	84 (87)
Yes (n = 103, 51%)	6 (6)	97 (94)	0.40 (0.15 to 1.10)	0.075	0.34 (0.08 to 1.46)	0.148
Quality assessment performed, n (%)
No (n = 43, 22%)	1 (2)	42 (98)
Yes (n = 157, 78%)	18 (11)	139 (89)	5.44 (0.71 to 41.96)	0.104	5.29 (0.38 to 82.82)	0.236
Authors mentioned using GRADE
No (n = 177, 88%), n (%)	17 (10)	160 (90)
Yes (n = 23, 12%), n (%)	2 (9)	21 (91)	0.90 (0.19 to 4.16)	0.889	0.47 (0.07 to 3.38)	0.453
Journal impact factor, median (IQR)	3.85 (2.73–5.76)	2.94 (2.14–4.98)	1.09 (1.004 to 1.18)	0.040	1.01 (0.90 to 1.13)	0.848
Journal endorses systematic review guideline, n (%)
No (n = 60, 30%)	9 (15)	51 (85)
Yes (n = 140, 70%)	10 (7)	130 (93)	0.44 (0.17 to 1.34)	0.089	0.22 (0.04 to 1.09)	0.064
Reviewers reported using systematic review guideline, n (%)
No (n = 127, 63%)	5 (4)	122 (96)
Yes (n = 73, 37%)	14 (19)	59 (81)	5.79 (1.99 to 16.84)	0.001	5.38 (1.19 to 24.23)	0.029
Type of review, n (%)
Association (n = 100, 50%)	5 (5)	95 (95)
Intervention (n = 100, 50%)	14 (14)	86 (86)	3.09 (1.07 to 8.95)	0.037	0.94 (0.20 to 4.55)	0.941

FIGURE 6.

Assessment of publication bias in sampled reviews stratified by type of review, number of included studies and whether or not a meta-analysis was performed. (a) Intervention reviews; and (b) association reviews. PB, publication bias.

Outcome reporting bias

Outcome reporting bias was mentioned in 34 (17%) systematic reviews and, again, this was more frequent in intervention reviews than in association reviews (30% vs 4%; p < 0.0001). All of the 34 reviews stated that outcome reporting bias was assessed as part of quality assessment of included studies. The risk-of-bias tool used most often was the Cochrane’s risk-of-bias tool (28/34). 64 Two reviews used the Agency for Healthcare Research and Quality’s Methods Guide for Effectiveness and Comparative Effectiveness Reviews,65 one used the Amsterdam–Maastricht Consensus List for Quality Assessment, and the remaining three reviews used unspecified or bespoke tools. Three of the 34 reviews did not report the findings of outcome reporting bias assessment, despite having stated such an assessment in their methods section. Of the remaining 31 reviews that reported the findings, 35% (11/31) reported having at least one study with high risk of selective outcome reporting, 32% (10/31) judged all included studies as low risk, and the remaining 10 reviews (32%) had at least one study in which the authors were unable to judge the risk of bias and were classed as ‘unclear’.

Case study 1: association between weekend/weekday admissions and hospital mortality (Chen et al.66)

Comments and further exploration of different methods

As the Bayesian meta-analysis and data augmentation approach used by the review authors are technically demanding, we reapply other commonly used (frequentist) statistical methods to explore their applicability in HSDR scenarios.

Egger’s test, Begg and Mazumdar’s rank correlation test, and trim and fill method

Egger’s regression test40 (in which the standardised effect estimate is regressed against its standard error) suggested no funnel plot asymmetry (p = 0.686), whereas Begg and Mazumdar’s rank correlation test41 (which is a test of correlation between the ranks of effect sizes and the ranks of their variances) indicated a strong small study effect (p = 0.004). These results are surprising, as the statistical power is usually greater for Egger’s regression test than for Begg and Mazumdar’s rank correlation test. The above results are likely due to the fact that most of the studies have very small standard errors (very large sample sizes). The lack of studies with large standard errors (small sample sizes) and, hence, a spread of studies with varied sample sizes makes the regression test unreliable but Begg and Mazumdar’s rank correlation test is less affected owing to its non-parametric nature. Application of the trim and fill procedure resulted in little change in the overall estimate (meta-analysis with missing studies filled, OR 1.15, 95% CI 1.11 to 1.20); this is probably because any effects of ‘filling’ missing studies would be minor considering the weight given to studies with very large sample sizes based on data from population databases.

The above example highlights an important technical requirement for the appropriate use of funnel plot and related methods, that is, the studies cannot be mostly of similar sample sizes. 26 Failure to meet this requirement may occur in HSDR, for which studies often rely on analyses of data from large administrative databases.

A further issue arises when studies draw data from population databases that essentially cover the entire population or health system, such as the English Hospital Episode Statistics. The concept of sampling error that underpins a funnel plot becomes irrelevant if a study has access to and utilises whole-population data. Publication bias associated with smaller studies (e.g. based on data from a region or a few hospitals within the population) may seem irrelevant, as these studies represent repeated analyses of a fraction of the same data (admissions) and could be regarded as redundant. However, an important issue to consider here is that, unlike RCTs of interventions, in which large sample sizes tend to confer a high level of methodological rigour and hence a low level of bias, observational studies based on whole-population data do not necessarily confer the same advantage over smaller studies that are based on bespoke data collected locally. Indeed, the contrary may be true. For example, the weekend mortality effect is hypothesised to reflect the effect of differential care quality between weekend and weekday, but this assertion is valid only if potential confounders, such as differential case mix in patients admitted between weekends and weekdays, have been accounted for. It has been speculated that the weekend mortality effect demonstrated in studies based on large administrative databases may be, at least in part, attributed to unmeasured (and thus unadjusted) differences in the frailty and urgency of patients admitted at weekends and on weekdays, and emerging evidence conducted by independent research groups has shown that the estimated weekend effect attenuates when data (not usually available in administrative databases) reflecting patients’ acute physiology or admission context are taken into account. 71–73

In such a scenario, in which an association may exist between estimated effect sizes and sample sizes, but for a different reason and in a different form from what is expected of funnel plot asymmetry caused by publication bias, use of funnel plot and related methods and interpretation of their findings require further caution.

A further issue, which is commonplace in HSDR, and which is another important consideration in the application of funnel plot and related methods, is the existence of heterogeneity between studies. 26 The authors of this weekend effect review focused on hospital-wide studies and excluded studies that examined only condition-specific hospital admissions. This might have helped to reduce statistical heterogeneity between studies. Nevertheless, heterogeneity associated with differences in clinical (e.g. emergency vs. elective vs. maternity admissions; different geographic locations) and methodological factors (e.g. variables included in statistical adjustment; variation in outcome measures, such as inhospital, 7- or 30-day mortality, definition of weekends, etc.) between the included studies is still evident, as shown in Figure 7a, in which many studies lie outside the 95% confidence limits, indicating that they were not measuring a common effect.

FIGURE 7.

Funnel plots for the meta-analysis of weekend mortality effect. (a) Funnel plot showing 95% confidence limits (dashed lines), centred at the fixed-effect pooled estimate and a fitted regression line for Egger’s test; and (b) contour-enhanced funnel plot showing contours of different levels of statistical significance, centred at null effect (OR = 1, or log-OR = 0 as shown here).

Heterogeneity poses an issue for the use of funnel plots and related methods, as it confounds the association between effect size and sample size. For example, studies that include only elective admissions are likely to have smaller sample sizes than studies that cover all types of admissions (both emergency and elective). If the weekend effect is more pronounced among elective admissions than other types of admissions, an association between sample size and effect size may be observed, even if there is no publication bias. Attributing the funnel plot asymmetry to publication bias and attempts to ‘correct’ for this purported bias could therefore be misleading.

There are a couple of ways to further investigate whether apparent funnel plot asymmetry may be attributed to publication and reporting bias, or is likely to be caused by other factors, such as heterogeneity. One is to use a contour-enhanced funnel plot (as shown in Figure 7b). For asymmetry caused by publication bias, the ‘missing’ (presumably unpublished) studies that could have made the funnel plot symmetric would have lain within the area corresponding to non-significant results (p > 0.05, the dark grey and white areas in Figure 7b), which seems likely here. In addition, p-hacking and reporting bias could also result in a higher density of (published) studies appearing in the area corresponding to borderline statistical significance (the area within and surrounding 1% < p < 5%), which is not apparent here. Another way is to produce funnel plots by study-level features that might create distinct subgroups contributing to the heterogeneity. Figures 8 and 9 show funnel plots in which studies are sorted by whether or not they included elective admissions and maternity admissions, respectively. Overall, the asymmetry appears to be more profound in studies including elective admissions and in studies excluding maternity admissions. These suggest that the degree of funnel plot asymmetry may differ within different types of admissions, but the asymmetry observed in the pooled analysis is unlikely to be purely caused by inclusion of different types of admissions.

FIGURE 8.

Funnel plot with pseudo-95% confidence limits of the weekend effect, sorted by whether individual studies include or exclude elective admissions.

FIGURE 9.

Funnel plot with pseudo-95% confidence limits of the weekend effect, sorted by whether individual studies include or exclude maternity admissions.

Meta-regression

A further approach to investigating publication and related bias is to use meta-regression to examine the presence or absence of an association of between effect sizes and the sample sizes of individual studies (this is similar to Egger’s regression test), while simultaneously controlling for other study-level variables. The authors of the systematic review carried out a meta-regression, although sample size was not included as one of the covariates in the analysis. We replicated the meta-regression, including variables initially included by the review authors, and then added the total study sample size as a binary variable (categorised into two groups of greater or less than 3 million hospital admissions). The findings are shown in Table 8.

TABLE 8 - Meta-regression to explore the relationship between study sample sizes and estimated effect sizes, adjusting for other potential factors that could influence the estimated weekend effect

Study sample size refers to the total sample size of the study from which the samples for individual effect estimates were drawn, not the sample size based on which individual effect estimate was calculated. The study sample sizes were dichotomised into greater or less than 3 million admissions, which were approximate to the median sample sizes.

Statistical adjustment categories: 1 and 2a, adjusted for measures of acute physiology; 2b, not adjusted for acute physiology but adjustment was otherwise adequate; 3, partial adjustment; and 4, inadequate adjustment.

Notes

Total n = 119 weekend effect estimates.

One study can contribute to multiple estimates. Exp(β) is the regression coefficient exponentiated so that it can be interpreted as the OR compared with a reference category or per unit increment (e.g. for each later year).

Variable	Exponential (β)	Standard error	p-value	95% CI
Sample size > 3 million admissionsa	1.078	0.375	0.033	1.006 to 1.155
Statistical adjustment categoryb
1 and 2a	1			Reference
2b	1.047	0.091	0.601	0.881 to 1.243
3	1.125	0.964	0.171	0.950 to 1.334
4	1.123	0.094	0.168	0.952 to 1.325
Year	1.001	0.003	0.769	0.994 to 1.008
Inclusion of elective admissions	1.301	0.561	< 0.001	1.194 to 1.417
Inclusion of surgical admissions	0.984	0.072	0.822	0.850 to 1.138
Inclusion of maternity admissions	0.824	0.032	< 0.001	0.764 to 0.889
Intercept	1.015	0.114	0.896	0.812 to 1.268

The analysis suggested that estimates obtained from larger studies with a total sample size of > 3 million hospital admissions showed a weekend effect approximately 8% larger than those obtained from studies of smaller sample sizes. This is in contrast with evidence from randomised trials, which tend to show that larger studies tend to report smaller effects. 12 A possible explanation is that smaller studies were able to make better statistical adjustment, with additional data not available from administrative databases. However, the completeness of statistical adjustment (based on review authors’ judgement) was taken into account in the meta-regression reported in the review. Although the level of statistical adjustment did not appear to significantly influence effect size in the analysis, it remains a possibility that the review authors’ judgement did not properly capture adequacy of statistical adjustment and, hence, the existence of residual confounding between bias related to adjustment, sample size and effect size. Finally, the finding from this meta-regression is exploratory. It could simply be a chance finding and, therefore, needs to be treated with great caution.

Application and applicability of p-curve

The risk of p-hacking may be particularly high in studies involving analysis of large data sets with many different variables, such as the ones included in the weekend effect systematic review. We therefore used the p-curve technique to explore the potential occurrence of p-hacking. 2 First, we calculated corresponding p-values for the 119 estimates (adjusted ORs and their CIs) of the weekend effect included in the above meta-regression, using the approximation method described by Altman and Bland,74 and then created a p-curve using the online application provided by Simonsohn et al. 43 The p-curve (Figure 10) suggests that, based on the estimates from included studies, the weekend effect does exist (the curve peaks on the left), but there is no evidence of p-hacking (there is no peak on the right close to p = 0.05).

FIGURE 10.

p-curve for estimates of the weekend effect included in meta-regression. The observed p-curve includes 82 statistically significant (p < 0.05) results, of which 78 are p < 0.025. There were 37 additional results entered but excluded from p-curve because they were p > 0.05.

The p-curve is a relatively new tool and, thus, its validity and utility are still subject to debate and require further empirical testing. 75,76 Our exploratory analysis provides some assurance that p-hacking is not evident among the literature included in the weekend effect systematic review. It is plausible that with very large sample sizes available from administrative databases and myriad variables/groups that can be analysed, it would not be necessary to resort to p-hacking to obtain a statistically significant finding to report. On the other hand, the strong right-skewness does not necessarily confirm the existence of a true ‘weekend effect’; as Bruns and Ioannidis76 demonstrated in their simulation study, the p-curve cannot distinguish true effects from null effects in the presence of even a minimal level of unaccounted confounding, which is a likely scenario in the case of studies of the weekend effect.

Case study 2: relationships between organisational culture, organisational climate and nurse work outcomes (Fang52)

Case study 3: the effect of electronic prescribing on medication errors and adverse drug events (Ammenwerth et al.67)

Case study 4: effects of standardised hand-off protocols on information relay, patient, provider and organisational outcomes (Keebler et al.68)

Verification of publication status

After the initial online search for publications, we contacted investigators for 145 of the 300 studies in order to verify their publication status. Investigators for 34 of these studies were unreachable, as we were not able to find their contact details (n = 26) or the e-mails were undeliverable (n = 8). Thus, 111 investigators were contacted. The overall response rate was 60% (67/111) and this varied by cohorts: NIHR, 79% (33/42); HSRUK, 60% (6/10); HSRProj, 50% (23/46); and ISQua, 38% (5/13). Of the 26 investigators whose contact details we could not obtain, 23 were from the ISQua cohort. Organisers of the ISQua conference could not provide the contact details to us owing to data protection regulation, but agreed to forward our information request to the abstract presenters so that they could contact us directly. However, no response was received.

Characteristics of included cohorts of HSDR studies

The characteristics of included studies stratified by different cohorts are presented in Table 10. In summary, 140 (47%) of the selected studies were intervention studies (inception cohort, 44%; conference cohort, 53%). Of the intervention studies, just over one-third (36%) were RCTs (inception cohort, 44%; conference cohort, 23%), but nearly two-thirds (62%) had a concurrent control group (inception cohort, 77%; conference cohort, 38%). Two-thirds of studies (67%) used bespoke data sources rather than using data from databases. The majority of the studies (79%) reported statistically significant findings. Findings were judged to be positive in the same proportion (79%) of studies.

Publication status of included HSDR studies

Of all 300 studies, 62% were published in academic journals (inception cohort, 70%; conference cohort, 48%). Findings for 20% of the 300 studies were available as grey literature (inception cohort, 26%; conference cohort, 8%). Fifty-three studies (18%) were unpublished (inception cohort, 4.5%; conference cohort, 44%). The pattern of publication varied between individual cohorts (Figure 11). The highest academic journal publication rate was observed in the HSRProj cohort (75%) and the lowest was in the ISQua cohort (26%). All studies from the NIHR cohort were published either in academic journals or as grey literature (technical reports available through the NIHR’s website), whereas results for 68% of the ISQua were not published.

FIGURE 11.

Publication status of selected studies at follow-up.

Factors associated with publication in academic journals

We explored factors associated with publication in academic journals for all studies and separately for intervention studies, using logistic regression. The results are shown in Table 11. For the analysis including all studies, being a study from the ISQua cohort was the only statistically significant factor associated with a reduced odds of publishing in an academic journal in univariable analysis. Neither statistical significance nor positivity of study finding was associated with publication in academic journals.

In multivariable analysis for all studies, studies from both the NIHR cohort and the ISQua cohort were significantly less likely to be published in academic journals (compared with studies from HSRProj cohort). In addition, studies with bespoke data collection were more likely to be published in an academic journal than those utilising data from routine databases or surveys. Statistical significance and positivity of finding also appeared to be associated with increased odds of publication in academic journals, having adjusted for the other factors.

Findings from analyses focusing on intervention studies identified similar factors associated with publication in academic journals, with having a concurrent control group being an additional strong predictor in univariable analysis and approaching statistical significance in multivariable analysis.

Factors associated with non-publication

In this subsection we examine factors associated with non-publication (i.e. when study findings were neither published in academic journals nor available as grey literature). Given that non-publication did not occur in the NIHR cohort, these studies were excluded from the analyses. Findings from univariable and multivariable logistic regression for all studies and for intervention studies are presented in Table 12.

For the analyses across all (intervention and association) studies, the ISQua cohort was the only statistically significant predictor for non-publication in the univariable analysis. However, in the multivariable analysis, studies from both HSRUK and ISQua cohorts were significantly more likely than studies from the HSRProj cohort to remain unpublished, and statistical significance (or positivity of findings if the variable was entered in the model in place of statistical significance) also became a significant predictor of reduced risk of non-publication.

The influence of being from a conference cohort and bespoke data collection remains similar in the univariable and multivariable analyses focusing on intervention studies (although the analyses had less statistical power). However, having a concurrent control group or being a RCT appeared to be an additional independent factor associated with reduced odds of non-publication.

Is publication bias a problem in HSDR?

The first issue we explored with interviewees was to what extent they believe that publication bias was a feature of HSDR. A striking finding of the study is that many felt unable to respond with any certainty. This included researchers, for whom typical answers included ‘I don’t know the answer to the question’ and ‘I can’t tell really, to be honest’. Of the 16 interviewees who expressed a view, 11 believed that publication bias was prevalent and five that it was not. However, within each of these groups there was varying levels of confidence in this assessment. For example, although some argued it to be ‘rampant’, others were circumspect (‘I don’t think that it should be, but I think that it creeps in’, ‘I think it is important, but I don’t think it’s the central issue’). Those arguing against the prevalence of publication bias were also hesitant in their judgement (‘I don’t think publication bias is an issue, to be honest with you’). These findings appear to mirror the complexity emerging from previous work and further exploration helped to unpick the reasons behind this. Subsequent questioning focused on aspects or stages of the publication process of most relevance to the interviewee and focus group respondents.

Commissioning and funding of HSDR

Interviewees generally agreed that the circumstances in which HSDR is commissioned and funded influence the likelihood of publication bias. For example, many believed that highly formalised programmes of research funding were less susceptible. Some cited the NIHR HSDR programme as an example of commissioners adopting criteria-based decision-making by autonomous panels, based on external peer review:

With the NIHR or research councils, you know, you put in your research proposal and they let you get on with the research.

Interviewee 12, senior researcher/clinician, UK

A key reason why such funders were seen as associated with low publication bias was the presence of formal publication requirements (including, for example, mandatory full reporting and making data publicly available), as well as conflict policies and adherence to study protocols. This in turn was associated with substantial research projects led by teams with a strong academic track record and motivation to produce published academic outputs. Commissioners in these circumstances were described as being largely ‘blind’ to considerations other than the rigour of research outputs.

This approach to the commissioning and funding of research was contrasted with end user sponsored research, characterised by high levels of interest and involvement of the commissioners in the conduct and reporting of results. One respondent, for example, felt that government bodies directly commissioning evaluations of their own programmes were often ‘desperate for those to be shown to be positive’ and another spoke of charitable foundations requiring striking findings in order to ‘feed their communications machine’:

A lot of the research will be funded by apparently objective, non-commercial funding agencies, but there will still be the management consultancies and charities, there will still be government involvement. So I don’t think we can be too complacent.

Interviewee 2, editor, UK

Many of the interviewees who worked as researchers in such circumstances nevertheless asserted the relative autonomy afforded to them, indicating that any subsequent bias was unlikely to be direct or overly coercive:

What I worry about is this subtle, indirect pulling of punches; so softening the edges of the messaging. Because you get comments coming back from policy officials saying things like ‘you’ve given us a kind of glass half empty story. Can’t you turn it round?’. And of course, often it is a matter of language and emphasis.

Interviewee 11, editor/senior researcher, UK

Interviewees indicated that pressure from funders was of a lesser magnitude than that generated in much clinical research, from their perceptions or experience, and that there were less likely to be major conflicts of interests:

There’s a danger of framing it in the same way as in medical research because the stakes are very high in medical research and there’s big commercial involvement through the pharmaceutical companies, who’ve been repeatedly caught ‘off side’, as it were. Whereas I think the same issues don’t apply, at least universally, in health service research.

Interviewee 16, funder/senior researcher, UK

I think because so much of the funding we have is competitive and most of it is won through committees that actually make it very difficult, I would imagine, for the result of the previous study to affect the funding. Whereas traditionally, you do a trial and the company wants to give you more money to do more research in the area. I’ve never been in that situation, [laughing] sadly.

Interviewee 11, editor/senior researcher UK

Despite this, the frequent absence of a formal requirement for peer-reviewed publications from such government bodies was identified as a potential source of bias:

It effectively leaves the onus on getting peer-reviewed publications up to us which is exactly when publication bias is more likely.

Interviewee 14, senior researcher/funder, UK

Interviewees noted that locally commissioned evaluations and audits are rarely intended to be reported beyond the immediate organisational context, and that therefore it is inappropriate for them to be subjected to the rigours of formal peer review before publication. Publication bias was therefore seen as a result of publishing of such data beyond the intended scope and setting of the data collection exercise.

Many interviewees made a distinction between research and other forms of knowledge:

I think the boundary between research and service improvement is really blurred and if you’re just using someone’s routinely collect data, not purposely collected for a research study, but just kind of scraped off their systems, and you’re using it for a model which will help them improve their systems, then I don’t call that research personally.

Interviewee 4, editor/senior researcher, UK

I suppose then it becomes a question of what counts as publication bias and where the borderline is with something that actually really isn’t research, or shouldn’t be judged as such, shouldn’t be kind of assessed as such. I think it entirely depends on how the project’s been set up . . . many small-scale quality improvement projects are not generalisable analytically or statistically and therefore I’m not terribly sure why you’d try to publish them other than using them in a local setting.

Interviewee 9, junior mid-career researcher, UK

One interviewee considered that ‘editorial control’ was the primary means of differentiating research and consultancy:

At [name of organisation] we did both consultancy and research, and the distinction was research meant that there was an explicit agreement up front, that we the researchers would be able to have editorial control so that we could seek publication in a journal. In other words, that the funder could not stop us publishing.

Interviewee 15, consultant/senior researcher, UK

Publication bias in journal decision-making

Within the sample, researchers were most likely to argue that publication bias was pervasive in academic publishing and especially in editorial decisions over whether to accept or reject HSDR journal submissions:

I think [positive effects] are of interest to all the journals and some are more explicit about it than others . . . All of the journals, despite their veneer of academic impartiality, are highly news oriented and they are kind of pushing for space in that crowded market place.

Interviewee 16, funder/senior researcher, UK

You would never ever send something non-significant to a top journal, that’s my feeling.

Interviewee 3, junior mid-career researcher, Germany

You do quite often find editors will say, ‘This is too similar to another paper that’s already published’.

Interviewee 4, editor/senior researcher, UK

The responses of journal editors within the sample were more mixed. For example, some argued that authors were primarily responsible for any publication bias at the analysis and write-up phase of the research process (see Research design and researcher conduct in HSDR). However, most conceded that factors other than scientific quality influenced publication decisions, and many of these factors coalesced around the requirement for novelty of some form. One editor of a medical journal used the term ‘saturation bias’ to describe the diminishing likelihood of publication of papers once ‘the message is already out’, irrespective of scientific merit. Another identified study scale and certainty of results as being important:

If you have a very clear-cut answer on either side, positive or negative, obviously any journal will be very interested. We’re not shying away from a negative study by any means, if it’s definitively negative. I think it’s when it’s non-definitive because the study is too small, or inadequately powered. I think we’re a little bit more hesitant with those.

Interviewee 13, editor/mid-senior researcher, USA

Editors of medical journals acknowledged that HSDR submissions, in general, were less likely to be accepted, especially when they did not follow a trial-based design. Editors of journals in subdisciplines of HSDR described a range of characteristics predisposing journal editors to accept HSDR manuscripts for publication. Along with methodological rigour and study scale, these included extent of methodological and theoretical innovation, and the political salience of the research topic or intervention:

I think [strength of effects] wouldn’t affect publication in an OR [operational research] journal, but then usually they would be looking for you to show that your method was better than somebody else’s method. And if in a particular case you had shown that: you know, you’d applied your method to some problem and it hadn’t really made any difference, I think that would be harder to publish.

Interviewee 4, editor/senior researcher, UK

I do think that there is a very strong relationship between, if you like, the size of the funded project and its chances of being published, irrespective of its findings.

Interviewee 5, mid-senior researcher/editor, UK

In our kind of publishing, if you don’t see any change, for example, that’s not a very interesting study theoretically. Nobody is really very keen to publish articles that explain why things stay the same and so, to that extent, there is some bias in that we think we know why things stay the same.

Interviewee 2, editor/senior researcher, Canada

These views and experiences were echoed in interviews with researchers:

There are some systematic biases there and I think people, especially editors and peer reviewers, are sort of denying them in a way. They’re looking for the best quality research and they’ve got this idea of what best quality research looks like. And they do privilege the very novel, you know, highly rigorous, high impact studies over those that are sort of more mundane but still important.

Interviewee 19, senior researcher/editor, UK

I think it’s absolutely right that poor or null effects are less interesting, but they’re only less interesting if it’s a topic where people really want it to work; if it’s a topic where people don’t want it to work, then our hypothesis is great and, you know, the [name of journal] will publish it. So I don’t think it’s a simple as just positive results.

Interviewee 9, junior mid-career researcher, UK

These factors were seen as diluting the influence of the magnitude and statistical significance of quantitative findings. By comparison, the latter were generally attributed higher importance in medical journal decision-making. This suggests that although HSDR journals may be less likely to exacerbate publication bias through their editorial practices, they may be substituting this with other forms of bias associated with preference for novelty.

Responses were less consistent with regard to the influence of the editor and peer review roles in publication bias. Although some editors referred to peer reviewers as ‘human funnel plots’, other interviewees believed that publication bias was rarely a major consideration in the peer review process:

Certainly as a reviewer and as an editor I don’t see many questions or prompts asking about the presence of publication bias. I don’t see many reviewers’ comments about publication bias unless they’re aware of particular studies that have been missed.

Interviewee 5, mid-senior researcher/editor, UK

However, the proliferation of peer-reviewed journals was seen as mitigating the threat of publication bias, for example in HSDR systematic reviews. This meant that the status of the journal was a secondary consideration:

I don’t think it matters a toss where it’s published as long as it’s published.

Interviewee 16, funder/senior researcher, UK

Obviously, there’s a certain level of prestige that comes through getting it in a high impact factor journal but, nowadays, everything is findable in PubMed so as long as it’s there, I’m much less bothered about where it goes.

Interviewee 7, senior researcher, UK

Research design and researcher conduct in HSDR

Interviewee perspectives were again mixed in relation to the role played by researchers in publication bias. Although some argued that in trial-based research, behaviours, such as selective reporting and ‘data hacking’, were no longer possible, others believed this to be commonplace, especially in association studies:

The more subtle and more common type [of publication bias] is the issue around hypothesis testing when you’re the statistician doing the analysis and when you do a lot of what you would, in your mind, neatly label ‘preliminary analysis’; a lot of unofficial hypothesis testing – ‘see if this works’ – and subgroup analysis . . . That is a massive problem.

Interviewee 8, mid-senior researcher, UK

Others believed there to be a temptation to search for positive results when researchers were in some way connected to the intervention being evaluated:

It’s very hard to find adverse effects of service delivery and part of that is either that the people who are introducing those models are sort of strongly advocating them, you know, it’s sort of like airing your dirty washing. And I suspect the other thing is also it’s very hard to identify unintended consequences because they have a lens that focuses on the evaluation framework and don’t look more widely at knock-on effects. So that would be a variation of selective outcome reporting bias but one particularly unique to Health Service and Delivery Research.

Interviewee 5, mid-career researcher/editor, UK

It depends if you think bias is the absence of the results or it’s the change in what people report. So people may have been trying, for example, reduce [hospital infection] rates and if in fact, they stay the same, but they manage to cut readmissions for whatever reason, they may then report, ‘Oh, this is an effective intervention to cut readmissions rates’, although it wasn’t actually designed that way in the first place.

Interviewee 7, senior researcher, UK

For example, when I evaluated [name of government initiative]. I didn’t care whether it worked or not but if I have been part of developing a particular service, so for example I’m developing a medical adherence intervention at the moment, there is going to be part of me invested in wanting that to be positive.

Interviewee 6, senior researcher/editor, UK

There was general support for the view that research teams might take longer to submit negative or null results for publication. This was seen to be related to general levels of ‘excitement’ associated with positive findings:

I’ve been involved in a couple of large implementation research trials and neither of them came out with positive or significant outcome findings . . . To motivate yourself to get that negative finding trial out there, it’s a real challenge. I think that there is something about not wanting to share something that you feel failed in some way, which is ridiculous.

Interviewee 18, funder/senior researcher, UK

The psychology is, you know, ‘we looked for something, we didn’t find it, that’s a bit disappointing. Can we be bothered to write it up? It may not get published anyway. Let’s go ahead and look at something else’, kind of thing. I think that’s understandable and human nature.

Interviewee 2, editor, UK

Some respondents identified a risk of publication bias when evaluators are responsible for developing the intervention, and many felt that publication bias was a danger within much service improvement research:

There’s strong movement within quality improvement, which is a very enthusiastic kind of movement, one that celebrates success in lots of ways. And some relatively large claims are made for some work that’s often done with relatively simple ways of looking at data . . . Now that is all maturing rapidly but my instincts are that because of that strong enthusiasm and the desire for things to be seen to work, there’s a risk.

Interviewee 17, funder/junior mid-researcher, UK

This problem was seen by the two managers in our sample to be compounded by a health-care environment that rewarded ‘good news’ stories about health-care innovations and improvements:

It’s a fear of failure and therefore you cannot be seen to try something that doesn’t work because that’s not celebrated.

Interviewee 20, manager, UK

Table 13 summarises the factors seen to increase or decrease the risk of publication bias in HSDR.

It was clear from the interviews with researchers that, to the extent that they considered researchers guilty of publication bias, this was partly driven by the perceived expectations of journals and universities. Some argued that ‘job prospects’ are linked to high-impact journal publications. For example, junior researchers within our sample were more likely to believe publication bias to be widespread:

Even though we are all academics and we know about publication bias when it comes to our own work we’re thinking, ‘Well, which journal will be interested in these findings?’. If they’re positive ones it’s easier to get kind of excited about it, as it were, when thinking about likelihood of acceptance by a bigger journal. You know, if you’re evaluating some new policy and you find it doesn’t work then clearly that is vital but I think the threshold is higher for a negative finding.

Interviewee 8, mid-senior researcher, UK

A small number of respondents identified incentives to researcher-instigated publication that went beyond the individual’s interests:

Sometimes important parts of HSDR studies don’t get published because the organisations, universities, don’t see the value of that.

Interviewee 6, senior researcher/editor, UK

In the USA the amount of funding that’s available for health systems improvement work is quite limited so there’s definitely a competition aspect that plays in, which leads to issues with individual investigators; obviously they want to show that the work that they’re doing is worthwhile which increases the pressure to publish positive results.

Interviewee 23, editor/senior researcher, USA

For reasons outlined in the following section (see Distinctive features of HSDR), senior researchers were more likely to underplay the pressure to publish positive results:

Almost all of my studies show that whatever I’m looking at doesn’t work . . . The answer to your question is that, in my own personal research, it would not cross my mind not to publish something.

Interviewee 12, senior researcher/clinician, UK

Distinctive features of HSDR

As noted, a significant minority of researchers in the sample felt that, in broad terms, HSDR was less susceptible to publication bias than clinical research. They attributed this to characteristics they considered distinctive and typical of HSDR. These included the complex nature of interventions, which generated uncertainty in evaluation, and the context-specific determinants of outcomes, which meant that definitive conclusions were often difficult to draw:

I wonder whether epistemologically they are doing different things; essentially within the [Health Technology Assessment] community it’s much more of an aggregative process coming up with, you know, pooled overall effects, etc. But if you assume that there will be a great deal of heterogeneity in health service delivery research then perhaps epistemologically you’re trying to do much more about making sense of options and mapping different models. And linked to that is the interplay of context so that within an HTA [Health Technology Assessment] context there’s almost the assumption that a drug will work in a relatively similar way across multiple contexts, whereas a service delivery intervention is very context-specific. So maybe people are bringing different quality markers or a different sort of epistemological view.

Interviewee 5, mid-career researcher/editor, UK

They further argued that this complexity and contingency often prevented study randomisation and replication:

The sort of research I do, it is rarely possible to randomise the intervention, so if you’re evaluating the [name of national government initiative] for example, you can’t persuade NHS England to randomise it to practices. So much of the research designs have been observational.

Interviewee 12, senior researcher/clinician, UK

Allied to this was the ‘multifactorial’ and ‘multicomponent’ nature of much HSDR study design, and the simultaneous addressing of multiple research questions, so that strength and direction of findings was assigned less overall importance:

I think in a lot of clinical research the effectiveness question is the only question that people are really interested in. They still pretend to be interested in mechanisms of action, you know, why something didn’t work. There’s still a pretence that we’re interested in the cost-effectiveness but at the end of the day that effectiveness question is the absolute king. Whereas in health services research that’s less of an issue, I think. Usually, we do have an effectiveness question that we can rarely answer with a randomised control trial; we have to address with something lower down the hierarchy of evidence. And because those designs have less status than the RCT, there’s more of a balance in the status of the questions and the methods that are being used within health services research . . . You know, in meetings everybody’s talking about all the different components and how they’re going. You don’t have to fight to get any component onto the agenda.

Interviewee 6, senior researcher/editor, UK

However, some interviewees expressed frustration at the relative inattention paid to outcomes and the primacy given to implementation in HSDR publications, highlighting the challenges this posed for subsequent evidence synthesis:

We’ve done work on user engagement in major health service reforms and it’s kind of, ‘Well, what’s the outcome? Is it whether you actually got the change implemented or whether you did the right consultation?’. The fact that the change was or wasn’t implemented may not be the key outcome but that’s often the one that’s reported, with the description about how you got there. So you may not have information about what might be considered the key outcome.

Interviewee 10, senior researcher, UK

The equivocal nature of many HSDR results was cited as an additional reason for reduced levels of external scrutiny and verification from, for example, media, government and industry. This was seen as a further check on the pressure to find positive associations and effects.

Tackling publication bias

The range of views on the prevalence of the publication bias ‘problem’ was reflected in interviewee comments on potential ‘solutions’. All of those who considered publication bias to be widespread (and many of those who did not) expressed support for pre-registration of studies and the strict application of study protocols containing statements of pre-set outcome measures:

It’s very easy to do and I think it’s also a mark of sound intellectual practice to actually say what you’re going to study and why you’re going to study it, so we don’t end up seeing fishing expeditions.

Interviewee 1, editor/clinician, UK

A subgroup of interviewees believed this would result in unwarranted and inappropriate constraints on researcher conduct (‘feet of clay’). As well as the practical challenges in large, multistrand studies, interviewees argued that sensitivity to changes in context during the lifetime of a research programme would be compromised:

You’re not dealing with nature, you’re dealing with people. So you want to be able to take advantage of what you learn, not just think, ‘Alright, we’ll stick with the state of knowledge we had when we wrote the proposal’. But you need to be quite transparent about why you’ve changed what you’ve done.

Interviewee 15, consultant/senior researcher, UK

If you take a recent study which we’ve done in which we probably had about eight substudies. All using different methodologies. Are you going to register each of them? And difficult to register the programme as the whole, not least because the nature of the grant is that things change as your 5 years goes on. So I remain to be persuaded of the benefits of pre registration for HSR [health service research] trials principally because pre registration for drug trials is to get at the very clear problem which you’ve articulated, which I don’t think exists in quite the same way for us.

Interviewee 12, senior researcher/clinician, UK

Other strategies identified for tackling publication bias included mandatory publishing and development of repositories of null findings; training and awareness raising in publication bias; a general improvement in the rigour of HSDR studies (especially in the field of service improvement); strengthening of research teams (e.g. through partnering with senior academics); and greater incorporation of publication bias checks in journal peer review.

Impact of publication bias in HSDR

As noted, some interviewees believed publication bias to be common in HSDR and cited the potential distorting effects of this on syntheses of the evidence, for example through systematic review. However, others identified the proliferation of journals publishing HSDR as partly counteraction to this trend.

Interviews (including with service managers at senior and local levels) revealed some apparent contradictions in relation to the prevalence and impact of publication bias. On the one hand, there was a widespread view that decision-making was often carried out without due regard to the published evidence, such that publication bias was less of a concern. Interviewees were at pains to emphasise the weak relationship between evidence and the decision-making they observed:

You have to consider just how poor the access of NHS managers to research evidence is. That is my experience and when I came to realise there’s this whole world out there of people doing health services research it was a revelation. But it was something I had to discover for myself. There is a very low awareness of research work generally, I mean, in the management class. That’s a very broad statement, but that’s from my experience of having worked in it. The way you made decisions is very much based on what is in front of us and our conversations with colleagues, rather than through reference to literature.

Interviewee 24, manager/policy, UK

There might be a sad reason why publication bias would not be such a problem in health services research, because the decision-makers don’t pay attention to the evidence anyway.

Interviewee 22, senior researcher/clinician, Canada

On the other hand, respondents identified similar sources of bias generated from within the decision-making environment itself and perpetuated, for example, through professional periodicals and information sources:

It’s the same in my organisation. I can’t think of anyone who’s really stood up and said ‘We tried this and I’m gonna write it up and get it published because I don’t want anyone else going through this pain or spending energy on it’. We just don’t seem to have a culture of doing that . . . I think when you’re getting a decision from your governing body or your board to try something there is an expectation from them that it will work. My director says, ‘Oh, we’ve been asked for some good news stories’ and in my head I want to say ‘well the good news could be that we tried this and it didn’t work so we can’t go through it again’.

Interviewee 20, manager, UK

If you take something like [name of professional journal], most of the papers are subtitled ‘How my last two year project has transformed health care’ you know? They’re less frequently constructed as ‘How my last two year project proved that this didn’t work.

Interviewee 24, manager/policy, UK

If we run something like the [national health services initiative] and we’ve committed many billions of pounds to what is notionally a pilot project for other people to copy, it’s quite hard for the taxpayer to understand that at the end of spending many millions of pounds on testing something, we came to the conclusion we shouldn’t do it anymore because it’s a waste of money. Now, there is an acceptance in the drug industry that you have to kiss a lot of frogs before you find a prince and that’s not really the culture in management. The culture in management is everything needs to work every time.

Interviewee 24, manager/policy, UK

One researcher in the interview sample identified a further knock-on effect of this on the research community when deciding which interventions to evaluate:

There’s great reason to believe that the picture of what’s going on, for example, in the NHS is heavily influenced by reporting bias, in other words, people do a successful initiative and they inter-disseminate it. That doesn’t translate into quantitative bias but upstream if you think that that means that these interventions are then more likely to get funding to be explored, then that will lead to a sort of distortional bias in terms of interventions that are available.

Interviewee 5, mid-senior researcher/editor, UK

Patient and public views: findings from the focus group

The focus group was structured so as to enable discussion in three phases, starting with general discussion of evidence, followed by discussion of publication bias in clinical research and, finally, discussion of publication bias in HSDR. Examples were used to illustrate publication bias in the second and third phases. The over-riding aim was to gather perspectives on the importance of this topic from a patient and public perspective.

Participants described receiving an ‘avalanche’ of information, but said that they would search for more detailed information on the stories that piqued their interest (i.e. if it affected them, a family member or a friend). There was acknowledgement that news stories as presented might not be an accurate reflection of the research evidence and the need for critical reflection. Participants described the need to be ‘critical’ of sources and to consult and listen to varied perspectives. Participants also discussed different sources of information that they would consult, including social media (e.g. Twitter; Twitter, Inc., San Francisco, CA, USA), books (written by authors they trusted), to lectures and articles in peer-reviewed journals.

Key points of consensus included:

• There is a need to identify ‘trustworthy’ sources that accurately summarise evidence.

• Some people accept what is reported in the mainstream media unquestioningly, with associated risks and harms.

• When reporting research findings the mainstream media typically provide only limited information on how the study is funded and conducted, making it impossible to judge extent of potential bias.

• Some level of bias is unavoidable (e.g. ‘bias starts when you decide what you’re going to study’), so transparency of reporting is of the utmost importance.

The recent clinical example of reboxetine6 (Edronax®, Pfizer Inc., New York, NY, USA) (for anxiety and depression) was discussed. Participants used words, such as ‘worrying’, in relation to the example and raised ethical objections. For example, participants noted that patients participating in studies might not be aware that drug companies were not publishing all of the evidence. Participants also discussed the dangers of drug companies funding trials and the importance of patient involvement in setting research aims and outcome measures.

It’s worrying, yeah.

And it is worrying that there’s . . . because there’s always this issue of ethics, isn’t there, in medical studies and although people sign up and say, ‘This is an ethical study,’ where there is a benefit to, like, a drug company then I’m not sure you could ever think, ‘Well, they’d be unbiased,’ because I’d always think that, ‘Have they actually published all their findings from this study?’. So although you would hope such people, you would hope researchers would be ethical. If they’re being funded and their research is being funded then how difficult is it for them to be completely neutral?

A second example from HSDR, related to the use of mass mail-outs, was discussed (see Maglione et al. ,53 also described in Chapter 3, see Table 3). This prompted discussion of the need for public resources to be used as efficiently as possible, reducing waste and the threat posed to this by publication bias of this kind.

Does it matter if, for example, the NHS invests in things where the evidence is slightly biased . . . is that a problem?

It’s a waste of money.

Yeah.

It is if your resources are short, yes.

Across the two examples, participants made the following observations:

• Much extra attention is given to new interventions (‘the latest thing’) that promise to solve major problems and the tendency, therefore, is to be biased towards positive findings for new interventions.

• There are dangers in measuring and reporting only short-term outcomes and neglecting longer-term effects, whether positive or otherwise.

• Participants were aware that researchers might be under pressure to publish interesting results and might be deemed a ‘failure’ if they fail to do so.

This led to a series of recommendations put forward by participants:

• Critical appraisal – participants emphasised the importance of educating people at a young age to be critical of information they receive, especially in popular and social media.

• Clear reporting of information about who funds should be compulsory, with any conflicts declared.

• Addressing the pressures within universities to ‘get results and publish’ was seen as a priority. People should be allowed to ‘make mistakes’ (i.e. embark on studies resulting in null results) because these present ‘opportunities for learning’.

• Participants argued that other aspects of studies (e.g. methods) might be of interest and warrant publication where results are unremarkable.

Participants also agreed that research should be able to be replicated and, therefore, all sources of data should be made fully available for future researchers:

Is the issue not because of the company that we don’t have access to the data?

Yes, yeah.

And all this would be solved if it was a legal requirement that every time you did an experiment, the data was held in some sort of archive accessible by anybody.

Overall, participants argued that having strong PPI throughout a study is critical. This needs to be carefully planned and inclusive of all groups, including those typically under-represented. They argued that PPI advisors should have roles in designing studies and ‘challenging’ the experts so that bad practices associated with publication were less likely to happen.

This chapter summarises diverse views on publication and related bias obtained from our interviews with a wide range of stakeholders of HSDR, including researchers of varied seniorities and specialties, journal editors, research funders and service managers. In addition, we present findings from a focus group discussion, which was planned with PPI involvement and highlighted service users’s and the public’s concerns about the potential impact of publication and related bias in terms of research waste and the influence of evidence on health service decision-making. We discuss the implications of these findings in relevant sections in Chapter 8.

Work package 1: systematic review of empirical evidence on publication bias in HSDR

Our systematic review set out to examine available evidence on publication and related bias in HSDR. Despite an extensive literature search, we found only four methodological studies13–15,50 that specifically investigated this issue, with three14,15,50 of them focusing on health informatics. Although the evidence is very limited and there are methodological weaknesses in the identified studies, the available evidence does indicate the existence of publication bias: study findings were not always published; differences may exist between published and unpublished evidence; and examples, such as null or unfavourable findings being associated with non-publication,53 and magnitude of effect sizes being associated with publication in high-impact journals, have been reported. 13

Reasons for non-publication of HSDR findings described in the only survey we found concerning the publication of evaluation studies in health informatics appear to be similar to those for clinical research. 87 Lack of time and interest on the part of the researcher appears to be a major factor, which could be exacerbated when the study findings are perceived as uninteresting. Some of the reported reasons, such as ‘not of interest for others’ and ‘only meant for internal use’, highlighted the context-sensitive nature of evidence that is common for HSDR. These reasons for non-publication also highlight issues arising from the vague boundary between research and non-research (such as quality improvement projects and service evaluations) for many interventions and data collection activities undertaken in health-care organisations. Another reason for non-publication of particular relevance to HSDR is ‘political and legal reasons’. Publication bias arising from conflict of interest is well documented in clinical research,4 and similar situations may arise when funding for the evaluation is provided by commercial, charitable or health-care organisations which developed and/or implemented the service intervention. We did not identify methodological research specifically related to the impact of conflict of interest on publication of findings in HSDR, although anecdotal evidence of financial arrangements influencing editorial process exists88 and there are debates concerning the public’s access to information related to health services and policy. 89

Among systematic reviews of substantive HSDR topics that have reported findings of assessment of publication bias, most have used funnel plots and regression tests and approximately half of them found some evidence of bias. The required conditions for utilising these methods and caveats for interpretation of their findings,26,38 as illustrated and discussed in detail in case studies presented in Chapter 5, hold true here and warrant particular attention in view of heterogeneity commonly found among HSDR, resulting from the complexity and variability of service delivery interventions and the influence of contextual factors. 90,91

Searching grey literature and contacting researchers and funders who may be involved in research on the topic of interest remain important measures to reduce the risk of publication and related bias when undertaking evidence synthesis for HSDR, although these are often time and resource intensive, and to what extent the invested effort outweighs the negative impact of the bias requires careful evaluation. The findings reported by Batt et al. ,51 that published and grey literature from low- and middle-income countries may differ in volume, quality and geographical coverage, have important implications for stakeholders who are involved in the synthesis and use of HSDR related to global health.

This systematic review has several strengths, but also has some weaknesses. We used a wide range of search terms (see Appendix 1) and searched several databases and information sources. Nevertheless, we could not rule out the possibility that some evidence was missed in the process. In addition, we may not have covered all pertinent areas of HSDR owing to the definition that we adopted. We focused only on publication and related bias in quantitative studies and did not consider qualitative research, which plays an important role in HSDR. Given the limited evidence we found, which was mainly collected in the field of health informatics, the generalisability of the findings for other fields of HSDR is uncertain. Finally, despite our efforts in extensive literature search, the possibility that our findings are subject to publication and related bias cannot be ruled out.

Work package 2: overview of systematic reviews of intervention and association studies in HSDR

Work package 2 examined a random sample of 200 systematic reviews in HSDR in relation to reviewers’ practice in the assessment of publication bias and outcome reporting bias. Although 43% of the systematic reviews mentioned publication bias, only 10% formally assessed it. Outcome reporting bias was mentioned and assessed in 17% of the reviews. The proportions of reviews in which these biases were mentioned and assessed were significantly higher among intervention reviews than association reviews.

There are similarities and differences between our findings and those of previous meta-epidemiological studies on assessment of publication and outcome reporting biases in systematic reviews, mostly concerning clinical research (Table 14). Based on existing literature, approximately one-third to a half of systematic reviews mentioned publication bias and thus demonstrated awareness of this issue, with the notable exception of systematic reviews of genetic association studies, of which 70% mentioned publication bias. Mention of outcome reporting bias was lower than 30% across the board, with very low rates observed in reviews of HSDR association studies and reviews of epidemiological risk factors.

Although the findings suggest that there is still substantial scope for improvement in raising awareness of these potential biases and in explicitly acknowledging them in the conduct of HSDR systematic reviews, a few inter-related issues warrant further consideration when interpreting these findings and the making recommendations for further actions. These relate to research traditions and the nature of evidence, as well as the requirement for research registration, which we explicate below.

The research traditions and nature of evidence in different subject disciplines may influence the perceived importance and relevance for considering publication and outcome reporting biases in the review process. These might have contributed to the apparently low prevalence of assessing and documenting these biases in HSDR reviews. For example, both our study and the other HSDR-related study31 concerning health policy research mentioned earlier found that a meta-analysis was conducted in < 40% of all reviews, and in our study the prevalence was found to be as low as 10% for reviews of association. In contrast, at least 60% of reviews of clinical research included meta-analysis (see Table 14). There is a general recognition that HSDR requires consideration of multiple factors in a complex health system,19 and that evidence generated from HSDR tends to be context specific. It is therefore possible that HSDR systematic reviews, and particularly those examining associations between the myriads of structure, process, outcome measures and contextual factors, may tend to adopt a more configurative, descriptive approach (as opposed to the more aggregative, meta-analytical approach in reviews of various types of clinical research). 97 As generating an overall estimate of a ‘true effect’ is not the main focus, the issue of publication and outcome reporting biases may be perceived as unimportant or irrelevant in reviews adopting configurative approaches. This may partly explain the strong association between inclusion of meta-analysis and assessment of publication bias in our multivariable analysis.

The diverse, context-specific nature of evidence in HSDR may have further impeded formal assessment of publication bias. Funnel plots and related techniques, the methods most commonly used in HSDR, as in other fields, require that at least 10 studies that are sufficiently similar can be included in an analysis. 26 The requirement has probably prevented many HSDR reviews from carrying out formal statistical assessment of publication bias, as the level of heterogeneity among studies included in HSDR systematic reviews is often high. Irrespective of the feasibility of adopting statistical methods, these techniques provide only indirect evidence suggestive of the presence or absence of publication bias, and their limitations have been well documented. 26

Irrespective of research approaches and nature of evidence, a major barrier for evaluating both publication and outcome reporting biases in HSDR is the lack of prospective registration of study protocols. Comprehensive study registration and accessible study protocols are essential for identifying unpublished studies and unreported outcomes, which in turn is a prerequisite for directly assessing publication and outcome reporting biases. As mandatory registration of research protocols has been enforced among clinical studies on human subjects only, lack of study registry and protocols is likely to have contributed to the low prevalence of assessing these biases, particularly among reviews of observational studies (e.g. 4% among HSDR association reviews in our study and 7% among epidemiological risk factor reviews examined by Song et al. 4). However, irrespective of evidence synthesis approaches and challenges, the potential threats from these biases do not dissipate as long as the intention is to quantify an intervention effect or an association. Pre-registration of study protocols, which is the ultimate safeguard against publication and outcome reporting biases, may therefore be worth serious consideration for both HSDR intervention studies and at least some types of association studies. Accessibility to pre-registered study protocols will also alleviate current difficulties in assessing outcome reporting bias.

Calls for comprehensive registration of research studies and their protocols are not new, but making further progress beyond clinical trials is likely to require careful debates and assessments with regard to feasibility and practical value for registering different types of studies, weighing potential benefits against costs and potential harms. Meanwhile, it is important to continue raising awareness of these biases and improving the levels of documenting the awareness when evidence from quantitative HSDR is synthesised. Our findings show that systematic reviews mentioning the use of a systematic review guideline are five times more likely to include an assessment of publication bias than those without a mention. A previous study also found that the proportion of systematic reviews which assessed publication bias was significantly higher after the publication of PRISMA (53%) than before PRISMA (39%). 94 Methodological standards, such as Cochrane Collaboration’s Methodological Expectations of Cochrane Intervention Reviews, and systematic reviews reporting guidelines, such as PRISMA and Meta analysis of Observational Studies in Epidemiology,33 are therefore likely to play an important role. Nevertheless, the suboptimal level of documenting awareness found in our study and others highlights that additional mechanisms may be required to enforce them.

For the assessment of outcome reporting bias, we noted that all the reviews which assessed this did so as part of quality assessment of individual studies. Currently, outcome reporting bias is a standard item in the Cochrane’s risk-of-bias tool,64 which is most widely used in intervention reviews. However, this item is not included in tools commonly used for assessing observational studies, such as the Newcastle–Ottawa Scale. 98 Given that the risk of outcome reporting bias is substantially higher for observational studies, this is an important deficit which developers of quality assessment tools for observational studies need to address in the future.

Finally, the search and inclusion of grey and unpublished literature remains a potentially important strategy in minimising the potential effect of publication bias. However, the practice varies widely between different types of reviews. Interestingly, our study showed that reviewers who searched for grey literature do not always assess or discuss the potential effect of publication bias. This suggests that some review authors might have followed the good practice of searching the grey and unpublished literature to ensure comprehensiveness, without considering minimising publication bias as a rationale behind this. Alternatively, these authors may consider it unnecessary to assess and/or discuss the potential impact of publication bias in addition to the efforts in locating unpublished studies. Limited evidence suggests that data included in published HSDR studies differ in quality and nature from those included in grey literature. 51 More empirical evidence is needed to guide future practice regarding search of grey and unpublished literature, taking into account the trade-off between biases averted and additional resources required.

One of the strengths of the current study is that the systematic reviews were randomly selected from the HSE database, which covers multiple sources of literature, and our selection was neither limited by a single source of literature nor restricted to highly ranked journals. We therefore believe that the sample is representative of quantitative HSDR systematic reviews. In addition, study selection and data extraction were carried out by one person and checked by another in order to ensure accuracy and completeness. We also evaluated factors which may influence the assessment of publication and outcome reporting bias. The main limitation of this study is that the results of the multivariable logistic regression analyses produced ORs with fairly wide CIs. We are aware that variables which we examined may interact in various ways, but believe that collinearity is not a major issue, judging from the broad consistency between the results of univariable and multivariable analyses.

In conclusion, this study has shown that publication and outcome reporting biases are not consistently considered or assessed in HSDR systematic reviews. Until a comprehensive registration of HSDR studies and their protocols becomes available, formal assessment of publication bias and outcome reporting biases may not always be possible. In such cases, review authors could still consider and acknowledge the potential implications. Developers of quality assessment tools for observational studies should consider including items for outcome reporting bias. Adherence to existing systematic review guidelines may also improve the consistency in assessment of these biases. The findings of this study would enhance awareness of publication and outcome reporting biases in HSDR systematic reviews and inform future systematic review methodologies and reporting.

Work package 3: in-depth case studies on the applicability of methods for detecting and mitigating publication and related biases in HSDR

In case study 1, described in Chapter 5, we utilised data from a systematic review of studies on the weekend mortality effect to explore in detail the applicability of common methods that have been used in clinical research to detect and/or adjust for publication bias. We highlighted several issues pertinent to the use of these methods in evidence synthesis in HSDR, and the findings may be useful in similar scenarios in which HSDR mainly relies on evidence from analyses of observational data obtained from routine administrative databases.

The case study shows that, although funnel plot and related methods can be used in HSDR as a tool to explore the presence of small-study effects (which are a common manifestation of publication/reporting bias), several features of HSDR based on observational databases could contradict requirements for their appropriate use and, in particular, could invalidate the crucial assumption that any observed small-study effects are caused by publication or reporting bias. Special attention needs to be paid to the underlying heterogeneity of studies included in a review. Factors that could define distinctive subgroups and settings (e.g. types of admissions, availability of clinical data) may be potential confounding factors that are independently associated with effect sizes and sample sizes and may therefore confound the assumed publication bias. Any findings from these statistical techniques will thus remain highly speculative, unless these potential confounding factors can be ruled out, which is likely to be difficult.

Our detailed exploration in this case study suggests that some level of publication or reporting bias is possible in the literature included in the chosen systematic review, but its effect on the pooled estimate of the weekend effect is limited. A few points are worth mentioning. First, the authors of the review excluded grey literature, such as conference abstracts and institutional reports, because it was difficult to assess their quality. This precluded direct assessment of potential publication bias by comparing published and grey literature. Second, the impact of publication bias may be relatively small when data from population databases are available. Nevertheless, caution is required in such circumstances, as precise estimates generated by large study samples are not necessarily accurate. Evaluating other sources of bias, such as the potential impact of unaccounted confounding, may be of higher priority and importance than publication bias. Finally, a deeper epistemological consideration is the utility of a ‘pooled effect’ in HSDR in general. In contrast to the priority of pursuing an accurate, universal estimate of the effect of a treatment or exposure in clinical research (however, this might change in the era of ‘precision medicine’), a pooled effect estimate, such as the weekend effect across various patient populations, health systems and geographical locations, may be of interest to no more than a small number of persons. Examining the mediators and contextual moderators of an effect is likely to play a more prominent role in HSDR. Funnel plot and related methods that are designed to assess the impact of publication bias surrounding a pooled estimate may therefore be of limited value, except in examples such as multivariate meta-regression, in which investigation of the impact of multiple factors can be carried out. Issues identified in case study 1, such as heterogeneity and potential confounding, were mirrored in other case studies included in the WP.

Work package 4: follow-up of HSDR study cohorts for investigating publication bias in HSDR

This WP collected direct, and thus the strongest, evidence on publication bias. Rates of publication among the selected HSDR cohorts are relatively high, except the ISQua cohort. This may reflect the nature of these cohorts, most comprising studies being well-funded research carried out by researchers. We noted that many of the studies from the ISQua cohort were reported by practitioners and were conducted in a single institution, as exemplified by the larger proportion of before-and-after studies without a control group. Many of these were likely to be studies of in-house improvement project and service evaluation that were not intended to be published.

The significantly lower rate of publication in this cohort than in other cohorts therefore suggests that the motivation for conducting these studies is likely to be one of the key factors influencing subsequent publication of the findings. Nonetheless, our multivariable analysis provided some indication that statistical significance and perceived positivity of the findings did appear to influence the likelihood of publication having adjusted for other factors associated with motivation, funding and methodological rigour. This finding should be considered tentative, given our limited sample size and potential subjectivity in classifying research findings. In addition, we were unable to obtain further information from investigators for 34 studies, with 23 of these from the ISQua cohort. This might have introduced some bias, although it could be argued that from the perspective of accessibility of evidence, any publications that might have existed but were missed by our own searches may be difficult to reach by decision-makers and other users of HSDR. Given the differential motivation, publication bias may be more profound in cohorts, such as the ISQua, than in cohorts of independently funded research. However, our sample size was too small to investigate this.

The 100% accessibility of study findings from the NIHR cohort demonstrates the important influence that research funders could have. Archives of historical projects accessible online and the now established NIHR Journals Library facilitate the dissemination of research findings irrespective of whether or not they are published in ‘traditional’ academic journals. An unexpected finding from this work was that the rate of publication in academic journals is lower for the NIHR cohort than for the HSRProj cohort. We could offer two plausible explanations among others. First, for consistency, we have classified all full reports from NIHR-funded projects as ‘grey literature’. These included project reports from the previous SDO and Health Service Research programme available online, as well as more recent monographs published in the Health Services and Delivery Research series of the NIHR Journals Library. If the latter had been classified as an academic journal, the publication rate would have appeared higher. Second, given that the journals included in the NIHR Journals Library are indexed in MEDLINE and are openly accessible, researchers might not perceive it necessary to publish the findings in other academic journals. In addition, during our communications with HSDR researchers, we were made aware that some academic journals in the field appear to be more reluctant to publish findings that would also be described and made available in NIHR Journals Library. Furthermore, as the production of the project final report as a monograph in the NIHR Journals Library requires substantial effort, this might have incurred some opportunity costs in relation to preparation of publications in other academic journals.

Work package 5: key informant interviews and focus group discussion to explore publication bias in HSDR

A notable finding from WP 5 was that many respondents were uncertain as to how significant a problem publication bias presented in HSDR. Although this is perhaps unsurprising in the case of those not directly involved in publishing, for example patients and health-care managers, it was also a feature of interviews with researchers, journal editors and research funders. Although the majority of these believed publication bias in some form to be present, few were able to refer to instances when this had been directly observed. This lack of clarity may reflect the apparently equivocal findings of previous WPs and our interviews therefore provide some potentially valuable qualitative insights.

One area that influenced views as to the presence of publication bias was the subdiscipline of HSDR to which respondents belonged. The researchers in our sample spanned diverse scholarly subfields, and this was reflected in the journals they targeted to publish their work. Although respondents sought to publish HSDR results in medical journals, many did so as a response to institutional pressures to publish in journals with high impact scores. In these circumstances, there was some consensus that significant (if not necessarily positive) results would be more likely to be (1) submitted and (2) accepted for publication. In contrast, many believed that other criteria were more important when seeking to publish in their disciplinary ‘home’ journals. Here, strength of results was believed to be secondary to other potential sources of ‘novelty’ in shaping publication outcomes.

As well as journal variation, interviewees drew a distinction between externally funded and peer-reviewed research on the one hand, and end user-funded quality improvement projects on the other, with the latter category attributing higher susceptibility to publication bias instigated by the research teams. Respondents also identified risks in ex post facto decisions to submit for publication, data which have been gathered primarily for other purposes. In the former category, there was general agreement that HSDR studies often contain more than summative assessments or measurements of associations. The typical presence of multiple study objectives, complex interventions, higher levels of ‘mess and noise’ in the data and their interpretation, was seen as reducing the importance ascribed to effect sizes and significance of associations. For this reason, a substantial minority of respondents were resistant to the proposal that all HSDR research be pre-registered.

There was some support in interview findings for the claim that forms of bias are linked to study type, with, for example, association studies considered more susceptible to p-hacking and selective outcome reporting, and evaluation studies at greater risk of funder pressure. However, respondents contrasted the level of external scrutiny (e.g. from industry and health systems) in clinical research with the tendency towards ‘lower stakes’ in HSDR, which was again seen as mitigating the incentive towards publication and related bias. Overall, interviewees considered incentives towards publication and related bias to be present but to a lesser degree.

The research cultures described by interviewees confirm the previously reported pressure to publish in high-impact journals. 99 However, in comparison with clinical research, HSDR is fragmented, and ‘subcultural’ factors may attenuate the drivers of publication and related bias. Other variables that warrant further investigation include researcher seniority (e.g. senior researchers in our sample were more likely to believe that negative or null results did not present an impediment to them publishing their work).

The interviews and focus group discussions with those not directly involved in publishing research (health-care decision-makers, patients and citizens) raise issues that go beyond the specific question of publication and related bias. These include the relatively weak relationship between HSDR and decision-making in health care, and the wider ethical implications of research waste, especially for participating patients and the allocation of scarce public resources.

Overall learning from the project and study limitations

Having described lessons learned from individual WPs above, we offer below some learning that emerges from across the WPs and highlight key limitations of this project.

Gleaning from the evidence collected across all WPs, it is reasonably clear that publication and related bias can and does occur in HSDR. However, it is currently difficult to gauge the true scale and impact of the bias given the sparse high-quality evidence, which in turn may be associated with difficulties in identifying study cohorts due to lack of comprehensive study registration. Solid evidence on publication bias and selective outcome reporting in clinical research has primarily been obtained from RCTs, as study protocols are made available in the trial registration process. 1 Stakeholders in our interviews generally supported registration of HSDR, albeit with some reservations, and quality improvement projects and implementation studies were identified as being potentially at high risk of these bias. As repositories of quality improvement projects emerge,100 HSDR and quality improvement communities will need to consider and evaluate the feasibility and values of adopting these practices for studies of varied types and purposes.

The lack of prospective study registration poses further challenges in assessing outcome reporting bias, which could be a greater concern for HSDR than clinical research, given the more exploratory approaches to examining a larger number of variables and associations in HSDR. Interestingly, our hypothesis that association HSDR studies based on observational evidence are more susceptible to publication and reporting bias was supported neither by the quantitative evidence that we collected so far, nor by recent research from the field of organisational studies (described in more detail in Publication and related bias in other cognate fields). 101–104 Nevertheless, the potential risk could still not be dismissed, as most of available evidence relies on methods that have some weakness. It is possible that the focus on formative evaluations rather than summative measures may have mitigated the bias to some extent.

Like many issues in HSDR, publication and related bias is the product of complex social processes and is likely be context dependent. Qualitative and quantitative data obtained from different WPs identified several potential factors (see Table 13) that may affect on the risk of its occurrence. Further empirical evidence needs to be collected to explore the relative influence of these factors and to ascertain whether interventions or policies targeting some of these factors can effectively reduce or prevent publication and related bias.

The strengths and limitations of individual pieces of work from this project have been discussed in detail in each of the WPs. Overall, although our findings have shed new light on our understanding of publication and related bias, the project and its findings have some important limitations:

1. We focus only on quantitative HSDR investigating interventions and associations.

2. Few studies have systematically examined publication and related bias in HSDR, limiting the conclusion that we can draw from review of the literature.

3. The numbers of HSDR systematic reviews that we evaluated in WP 2 and HSDR cohorts that we followed in WP 4 are still relatively small.

4. Although issues with the application of statistical methods to detect publication and related bias in HSDR are highlighted in WPs 2 and 3, novel methods need to be developed to overcome the issues identified.

5. Owing to lack of comprehensive registration of study protocols, we have not been able to collect direct evidence on publication and related bias concerning quality improvement projects, and to examine outcome reporting bias in study cohorts that we followed.

6. The sample size for our qualitative work in WP 5 is relatively small, and within the scope of the current project we have not explored some of the wider issues related to use of evidence from HSDR in making service decisions and transparency and public accountability of this process.

These limitations need to be addressed in future research.

Publication and related bias in other cognate fields

This project investigated publication and related bias in HSDR. A major challenge for the project was to draw a boundary between HSDR and other fields of research, given its multidisciplinary nature. In our systematic review presented in Chapter 3, we have focused on research in areas that fall under the definition of HSDR adopted in this project. However, we are aware that literature on publication and related bias has emerged in other scientific disciplines. Although it was not possible for us to systematically review this diffuse literature within the scope of this project, we provide below a brief description of key papers that may have some relevance to HSDR.

In our literature search for the systematic review described in Chapter 3, and through citation check, we identified several studies on publication and related bias in cognate fields. We describe these studies briefly in this section.

Points for consideration by stakeholders in relation to publication and related bias in HSDR

In this project we collected prima facie evidence of publication and related bias in quantitative HSDR using a variety of approaches that generated both direct and indirect evidence, as described in the previous chapters. Overall, our findings suggest that publication and related bias can and does occur in quantitative HSDR. However, there is substantial uncertainty with regard to the nature and extent of its occurrence and its impact on decision-making concerning health service delivery.

We presented our findings in a meeting convened at the end of the project, in which we invited a group of key stakeholders of HSDR (see Acknowledgements) to help us make sense of the evidence gathered in this project and formulate potential recommendations for future research practice. The attendees included senior researchers in HSDR and evidence synthesis, journal editors, funders and a PPI representative. Diverse views and perspectives were presented in the meeting, echoing the findings from our interviews. Taken in the round, the emerging consensus seemed to be that a nuanced interpretation, with due attention paid to the epistemological and methodological diversity in HSDR and the changing landscape of research publication, is needed rather than rushing to replicate measures (such as compulsory study registration in an attempt to reduce occurrence of this bias) that have been adopted in clinical research or dismissing the importance of this issue due to current lack of concrete evidence. Here, we highlight pertinent issues that different stakeholders might wish to consider when commissioning, undertaking, reviewing, publishing or using HSDR.

Recommendations for future research

To the best of our knowledge, this project is the first to systematically investigate publication and related bias across different subdisciplines of HSDR. Overall, our findings highlight a paucity of evidence, with many gaps and weaknesses in the evidence base informing this subject. We present below our recommendations for future research that may help in bridging these gaps.

Conclusion

This project collected prima facie evidence on publication and related bias in quantitative HSDR. Overall, our findings suggested that this bias can and does occur in this field, although literature on this topic is scant. The epistemological and methodological diversity of HSDR, the changing landscape of research publication and the boundary delineated by project scope all need to be taken into account when interpreting our findings.

The occurrence of publication and related bias and its impact are likely to vary between studies with different methodological designs for different purposes, with evaluations motivated by service improvement locally most susceptible to such bias. Variation in the practice of research publication may exist between different subspecialties in HSDR but this requires further exploration. The bias is likely to be most profound in relation to publication of HSDR findings in academic journals with a general medical orientation.

The precise magnitude of publication and related bias is difficult to quantify owing to the lack of comprehensive study registration and the vague boundary between research and activities undertaken for service improvement. Epistemology and methodological approaches that do not focus solely on summative assessments of outcomes or associations and that do not heavily rely on inferences based on statistical significance may to some extent mitigate the occurrence and influence of publication and related bias in HSDR.

Documentation of awareness of publication and related bias and formal assessment of the bias were low in HSDR systematic reviews, particularly in reviews of association studies. Methodological guidelines might have some positive impact on practice. Application of statistical tools, such as funnel plots, and related tests for detecting publication and related bias in HSDR is often problematic owing to the heterogeneity of studies and potential confounding in any associations observed between sample size (or precision of estimate) and effect size. These tools are best used, with great caution, as part of the broader effort in investigating potential modifiers of intervention effects or associations. Use of funnel plot asymmetry as a sole indicator of the presence or absence of publication and related bias should be avoided.

Further research to collect empirical evidence based on methods that allow for direct observation and inference of the occurrence of publication and related bias in various subdisciplines and study designs of HSDR is required. The utility and most efficient ways of including quality improvement projects and other evidence beyond academic literature in systematic reviews of HSDR need to be explored.

Contributions of authors

Abimbola A Ayorinde (https://orcid.org/0000-0002-4915-5092) (Research Fellow, Evidence Synthesis) was the main researcher on the project and maintained day-to-day running of the project, involved in all stages of the work and drafted the report.

Iestyn Williams (https://orcid.org/0000-0002-9462-9488) (Reader, Health Policy Management) was the lead for WP 5 of the project, planned and carried out the interviews and led the focus group discussion, analysed the data and drafted WP 5-related sections.

Russell Mannion (https://orcid.org/0000-0002-0680-8049) (Professor, Health Systems) was a member of the Project Management Group and provided senior advice on health services research.

Fujian Song (https://orcid.org/0000-0002-4039-1531) (Professor, Research Synthesis) was a member of the Project Management Group and provided advice on statistics and publication bias-related issues.

Magdalena Skrybant (https://orcid.org/0000-0001-7119-2482) (PPI and Engagement Lead) was a member of the Project Management Group, provided advice and was involved in all aspects of PPI-related activities, and facilitated the focus group discussion.

Richard J Lilford (https://orcid.org/0000-0002-0634-984X) (Professor, Public Health) was a member of the Study Steering Committee and provided senior advice on methodology.

Yen-Fu Chen (https://orcid.org/0000-0002-9446-2761) (Associate Professor, Evidence Synthesis) was the principal investigator of the project, was involved in all stages of the work and drafted the report.

All authors commented on multiple drafts of report sections, helped to revise the report and approved its submission.

Publications

Williams I, Ayorinde AA, Mannion R, Song F, Skrybant M, Lilford RJ, Chen YF. Stakeholder views on publication bias in health services research. J Health Serv Res Pol 2020;25:162–71. https://doi.org/10.1177/1355819620902185

Ayorinde AA, Williams I, Mannion R, Song F, Skrybant M, Lilford RJ, Chen YF. Assessment of publication bias and outcome reporting bias in systematic reviews of health services and delivery research: a meta-epidemiological study. PLOS ONE 2020;15:e0227580. https://doi.org/10.1371/journal.pone.0227580

Ayorinde AA, Williams I, Mannion R, Song F, Skrybant M, Lilford RJ, Chen YF. Publication and related biases in health services research: a systematic review of empirical evidence. BMC Med Res Methodol 2020;20:137. https://doi.org/10.1186/s12874-020-01010-1

Data-sharing statement

Most data collected in this project have been presented in this report. All data requests should be submitted to the corresponding author for consideration. Access to available anonymised data may be granted following review.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health and Social Care.

MEDLINE

Date range searched: 1946 to 30 July 2018.

Date searched: 16 March 2017, updated 31 July 2018.

EMBASE

Date range searched: 1947 to 30 July 2018.

Date searched: 16 March 2017, updated 31 July 2018.