Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: The RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) project

Article history

The research reported in this issue of the journal was funded by the HS&DR programme or one of its proceeding programmes as project number 10/1008/07. The contractual start date was in April 2011. The final report began editorial review in July 2013 and was accepted for publication in January 2014. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

none

Copyright statement

© Queen’s Printer and Controller of HMSO 2014. This work was produced by Wong et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

What are realist and meta-narrative reviews?

Realist and meta-narrative reviews are systematic, theory-driven interpretative techniques, which were developed to help make sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy. Interventions have been described as theory incarnate,20 driven by hypotheses, hunches, conjectures and aspirations about individual and social betterment. Strengthening a review process that helps to sift and sort these theories may be an important step in producing better interventions.

Realist reviews seek to unpack the relationships between context, mechanism and outcomes (sometimes abbreviated as CMO), i.e. how particular contexts have triggered (or interfered with) mechanisms to generate the observed outcomes. 14 Its philosophical basis is realism, which assumes the existence of an external reality (a real world) but one that is filtered (i.e. perceived, interpreted and responded to) through human senses, volitions, language and culture. Such human processing initiates a constant process of self-generated change in all social institutions, a vital process that has to be accommodated in evaluating social programmes.

In order to understand how outcomes are generated, the roles of both external reality and human understanding and response need to be incorporated. Realism does this through the concept of mechanisms, whose precise definition is contested but for which a working definition is, ‘. . . underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest’. 21 Different contexts interact with different mechanisms to make particular outcomes more or less likely; hence, a realist review produces recommendations of the general format ‘in situations [X], complex intervention [Y], modified in this way and taking account of these contingencies, may be appropriate’. Realist reviews can be undertaken in parallel with traditional Cochrane reviews (see, for example, the complementary Cochrane and realist reviews of school feeding programmes in disadvantaged children). 22,23 The Cochrane review produced an estimate of effect size, whereas the realist review addressed why and how school feeding programmes worked, explained examples of when they did not work, and produced practical recommendations for policy-makers.

Meta-narrative reviews were originally developed by Greenhalgh et al. 15,24 to try to explain the apparently disparate data encountered in their review of diffusion of innovation in health-care organisations. Core concepts such as diffusion, innovation, adoption and routinisation had been conceptualised and studied very differently by researchers from a wide range of primary disciplines including psychology, sociology, economics, management and even philosophy. While some studies had been framed as the implementation of a complex intervention in a social context (thus lending themselves to a realist analysis), others had not. Preliminary questions needed to be asked, such as ‘What exactly did these researchers mean when they used the terms ‘diffusion’, ‘innovation’ and so on?’; ‘How did they link the different concepts in a theoretical model – either as a context–mechanism–outcome proposition or otherwise?’; and ‘What explicit or implicit assumptions were made by different researchers about the nature of reality?’

These questions prompted the development of meta-narrative review, which sought to illuminate the different paradigmatic approaches to a complex topic area by considering how the same topic had been differently conceptualised, theorised and empirically studied by different groups of researchers. Meta-narrative review is particularly suited to topics where there is dissent about the nature of what is being studied and what is the best empirical approach to studying it. For example, Best et al. ,25 in a review of knowledge translation and exchange, asked how different research teams had conceptualised the terms knowledge, translation and exchange – and what different theoretical models and empirical approaches had been built on these different conceptualisations. Thus, meta-narrative review potentially offers another strategy to assist policy-makers to understand and interpret a conflicting body of research and, therefore, to use it more effectively in their work.

The need for standards in theory-driven systematic reviews

Realist and meta-narrative approaches can capitalise on and help build common ground between social researchers and policy teams. Many researchers are attracted to these approaches because they allow systematic exploration of how and why complex interventions work. Policy-makers are attracted to them because they are potentially able to answer questions relevant to practical decisions (not merely ‘What is the impact of X?’ but ‘If we invest in X, to which particular sectors should we target it, how might implementation be improved and how might we maximise its impact?’). As interest in such approaches is burgeoning, it is our experience that these approaches are sometimes being applied in ways that are not always true to the core principles set out in previous methodological guidance. 4,14,15,26 Some reviews published under the realist banner are not systematic, not theory driven and/or not consistent with realist philosophy. The meta-narrative label has also been misapplied in reviews which have no systematic methodology. For these reasons, we believe that the time has come to develop formal standards and training materials.

There is a philosophical problem here, however. Realist and meta-narrative approaches are interpretive processes (that is, they are based on building plausible evidenced explanations of observed outcomes, presented predominantly in narrative form); hence, they do not easily lend themselves to a formal procedure for quality checking. Indeed, we have argued previously that the core tasks in such reviews are thinking, reflecting and interpreting. 4,15 In these respects realist and meta-narrative reviews face a problem similar to that encountered in assessing qualitative research, namely the extent to which guidelines, standards and checklists can ever capture the essence of quality. Some qualitative researchers are openly dismissive of the technical checklist approach as an assurance of quality in systematic review. 27 While we acknowledge such views, we believe that from a pragmatic perspective, formal quality criteria – with appropriate caveats – are likely to add to, rather than detract from, the overall quality of outputs in this field. Scientific discovery is never the mere mechanical application of set procedures. 28 Accordingly, research protocols should aim to guide rather than dictate.

The online Delphi method

This study used the online Delphi method, and in this section we introduce, explain and justify our use of this method. The essence of the Delphi technique is to engender reflection and discussion among a panel of experts with a view to getting as close as possible to a consensus and documenting both the agreements reached and the nature and extent of residual disagreement. 29 It was used, for example, to set the original care standards which formed the basis of the Quality and Outcomes Framework for UK general practitioners. 30 Factors which have been shown to influence quality in the Delphi process include:

1. composition (expertise, diversity) of the expert panel

2. selection of background papers and evidence to be discussed by that panel (completeness, validity, representativeness)

3. adequacy of opportunities to read and reflect (balance between accommodating experts’ busy schedules and keeping to study milestones)

4. qualitative analysis of responses (depth of reflection and scholarship, articulation of key issues)

5. quantitative analysis of responses (appropriateness and accuracy of statistical analysis, clarity of presentation when this is fed back)

6. how dissent and ambiguity are treated (e.g. avoidance of groupthink, openness to dissenting voices). 29,31,32

Evidence suggests that the online medium is more likely to improve than jeopardise the quality of the consensus development process. Mail-only Delphi panels have been shown to be as reliable as face-to-face panels. 33 Asynchronous online communication has well-established benefits in promoting reflection and knowledge construction. 34 There are over 100 empirical examples of successful online Delphi studies conducted between geographically dispersed participants (for examples see Keeney et al. ,32 Elwyn et al. ,35 Greenhalgh and Wengraf,36 Hart et al. ,37 Holliday and Robotin,38 and Pye and Greenhalgh39). We were unable to find any online Delphi study which identified the communication medium as a significant limitation. On the contrary, many authors described significant advantages of the online approach, especially when dealing with an international sample of experts. One group commented ‘our online review process was less costly, quicker and more flexible with regard to reviewer time commitment, because the process could accommodate their individual schedules’. 38

Critical commentaries on the Delphi process have identified a number of issues which may prove problematic, for example ‘issues surrounding problem identification, researcher skills and data presentation’29 or defining consensus; issues of anonymity; time requirements for data collection, analysis, feedback to participants and obtaining responses on feedback; defining and selecting experts; enhancing response rates and deciding on how many rounds to undertake. 40 These comments suggest that it is the underlying design and rigour of the research process which is key to the quality of the study and not the medium through which this process happens.

Overview of methods

We used a range of methods to meet the objectives we set out above. In this section we provide a brief overview of the methods we used and how they related to each other. The following methods sections outline specific aspects of the methods we used in more detail.

To fulfil objectives 1 and 2 we undertook a narrative review of the literature that was supplemented by collating feedback from presentations and workshops. We synthesised our findings into briefing materials [one for realist synthesis (RS) and another for meta-narrative reviews]. We recruited members to two Delphi panels, which had wide representation from researchers, students, policy-makers, theorists and research sponsors. We used the briefing materials to brief the Delphi panel, so they could help us in fulfilling objective 3. For objective 4, we drew not only on our experience in developing and delivering education materials, but also relevant feedback from the Delphi panel, an e-mail list we set up specifically for this project (www.jiscmail.ac.uk/RAMESES), training workshops and the review teams we supported methodologically. To help us refine our publication standards (objective 5) we captured methodological and other challenges that arose within the realist or meta-narrative review teams we provided methodological support to. To produce the definitive publication standards, quality standards and training materials (objective 6), we synthesised expert input (from the Delphi panel), literature review and real-time problem analysis (e.g. feedback from the e-mail list, training sessions and workshops, and presentations). Throughout this project we did not set specific time points when we would refine the drafts of our project outputs. Instead, we iteratively and contemporaneously fed any data we captured into our draft publication standards, quality standards and training materials, making changes gradually. Only our Delphi panels ran within a specific time frame. The definitive guidance and standards were, therefore, the product of continuous refinements. We addressed objective 7 through academic publications, online resources and delivery of presentations and workshops. Figure 1 provides a pictorial overview of how the different methods we used fed into each other.

FIGURE 1.

Overview of study processes. RAMESES, Realist And Meta-narrative Evidence Syntheses – Evolving Standards.

Details of literature search methods

Prior to the start of this project we had undertaken initial exploratory searches. These were rapid searches that were not intended to be comprehensive: they involved each project team member identifying in their personal files examples of realist and/or meta-narrative reviews. To identify further reviews, we undertook a search of the reference lists of each retrieved review. This two-step process yielded 13 reviews that were later used by our expert librarian to develop and refine our searches (see Appendix 1). We found that the literature in this field was currently small but expanding rapidly, of broad scope, variable quality and inconsistently indexed. Our purpose for identifying published reviews was not to complete a census of realist and meta-narrative studies. We make no claims that the review we undertook was exhaustive, thus we have not and never intended that it should be published as a stand-alone piece of research. The main purpose of this review was to enable us to produce the briefing materials for our two Delphi panels (objective 3), not to produce an exhaustive summary of all research ever published on the topic. As such, the review we undertook would best be considered as being a rapid, accelerated or truncated narrative review. Such an approach will predictably produce limitations and these are discussed in Chapter 5, Limitations.

We wanted our search to allow us to pinpoint real examples (or publications claiming to be examples) that provide rich detail on their usage of those review activities we wish to scrutinise and formalise. To that end, and drawing on a previous study which demonstrated the effectiveness and efficiency of the methods proposed,41 and employing the skills of a specialist librarian, we searched 16 electronic databases from inception (where applicable) to 15 June 2010. The databases searched are listed below (the number of hits found with each database searched may be found in Table 1):

• Academic Search Complete

• Cumulative Index to Nursing and Allied Health Literature (CINAHL)

• The Cochrane Library

• Dissertation Abstracts

• EMBASE

• Education Resources Information Center (ERIC)

• Global Health

• Google

• HealthSTAR

• MEDLINE

• PASCAL [database of INIST (Institut de l’Information Scientifique et Téchnique)]

• PsycINFO

• Scopus

• Sociological Abstracts

• Social Policy and Practice

• Web of Science [Science Citation Index (SCI), Social Science Citation Index (SSCI), Arts and Humanities Citation Index (AHCI)].

We used the following approaches in our searches:

1. A simple truncated text-word search was conducted on all databases using the following words: (Meta-narrative OR metanarrative OR realist) ADJ (review* OR protocol* OR synthesis OR syntheses OR technic OR technics OR technique*) was used where ADJ (adjacency) was a search operator (Ovid Databases); or (metanarrative OR meta-narrative OR realist) AND (review* OR synthesis OR syntheses OR protocol* OR technic OR technics OR technique*), where adjacency was not a search operator. In the last instance, the search was limited to the title field. The strategy was developed based on a collection of 13 published reviews we had identified in our exploratory searches and was piloted and refined to produce the most sensitive search strategy for the topic.

2. Citation chaining, on databases where this feature was available (Scopus and Web of Science at the time of the search) was performed. Seminal citations were followed, with the reasoning that anyone using realist or narrative techniques would be likely to cite these references. 4,14,15

Results were kept separate for each database in RefWorks (version 5; RefWorks-COS, Bethesda, MD, USA) reference management software and were then collated into a separate merged file from which duplicates were removed.

No language or study design filters were applied. To construct our sample for further analysis (in which we intended to study both exemplary reviews and those that had flaws), we included any review that claimed to be a realist review or a meta-narrative review. Documents were excluded if they were not a review (e.g. editorials, opinion pieces, commentaries, methods papers) or did not claim to be a realist or meta-narrative review. We did not undertake any independent screening or an audit of a random subset for quality control purposes. The whole searching process from start to the retrieval of all full-text documents took approximately 1 month.

We conducted a thematic analysis of this literature which was initially oriented to addressing seven key areas:

1. What are the strengths and weaknesses of realist and meta-narrative review from both a theoretical and a practical perspective?

2. How have these approaches actually been used? Are there areas where they appear to be particularly fit (or unfit) for purpose?

3. What, broadly, are the characteristics of high- and low-quality reviews undertaken by realist or meta-narrative methods? What can we learn from the best (and worst) examples so far?

4. What challenges have reviewers themselves identified (e.g. in the introduction or discussion sections of their papers) in applying these approaches? Are there systematic gaps between the theory and the steps actually taken?

5. What is the link between realist and meta-narrative review and the policy-making process? How have published reviews been commissioned or sponsored? How have policy-makers been involved in shaping the review? How have they been involved in disseminating and applying its findings? Are there models of good practice (and of approaches to avoid) for academic–policy linkage in this area?

6. How have front-line staff and service users been involved in realist and meta-narrative reviews? If the answer to this is ‘usually, not much’, how might they have been involved and are there examples of potentially better practice which might be taken forward?

7. How should one choose between realist, meta-narrative and other theory-driven approaches when selecting a review methodology? How might (for example) the review question, purpose and intended audience(s) influence the choice of review method?

The thematic analysis was led by one member of the review team (GWo). He undertook all stages of the review and shared findings with the rest of the project team so that discussion, debate and refinement of his interpretations of the data in the included reviews could take place. Findings were shared by e-mail and, where necessary, face-to-face meetings took place to discuss any interpretations made of the data. In undertaking our thematic analysis, we familiarised ourselves with the included reviews to identify patterns in the data. We used the questions above, which relate to seven key areas, as a starting point in our sensemaking of the data, and as a project team we were aware that the purpose of the review was to produce briefing documents for the Delphi panels. In these panels we wanted to achieve a consensus on quality and reporting standards, and so what we needed from our review of the literature were data to inform us on what might constitute quality in executing and reporting reviews. We accepted that we might need to refine, discard or add additional questions and topic areas to explore in order to better capture our analysis and understanding of the literature as these emerge from our reading of the papers.

Data were extracted to a Microsoft Excel spreadsheet (Microsoft Corporation, Redmond, WA, USA), which we iteratively refined to capture the data needed to produce our briefing materials. This review was undertaken in a short timeframe, such that the time taken from obtaining full-text documents to producing the final draft for circulation of the briefing documents was approximately 10 weeks. The output of this phase was a provisional summary for each review method that addressed the questions above and highlighted for each question the key areas of knowledge, ignorance, ambiguity and uncertainty. This was distributed to the Delphi panel (as our briefing document) as the starting point for their guidance development work.

Details of online Delphi process

We followed an online adaptation of the Delphi method (see Chapter 1, The online Delphi method) which we had developed and used in a previous study to produce guidance on how to critically appraise research on illness narratives. 36 In that study, a key component of a successful Delphi process was recruiting a wide range of experts, policy-makers, practitioners and potential users of the guidance who could approach the problem from different angles and, especially, people who would respond to academic suggestions by asking ‘so what?’ questions.

Placing the academic–policy/practice tension central to this phase of the research, we planned to construct our Delphi panel to include a majority of experienced academics (e.g. those who have published on theory and method in realist and/or meta-narrative review). We also planned to recruit policy-makers, research sponsors and representatives of third-sector organisations. These individuals were recruited by approaching relevant organisations and e-mail lists [e.g. professional networks of systematic reviewers, CHAIN (Contact, Help, Advice and Information Network; http://chain.ulcc.ac.uk) and INVOLVE (www.invo.org.uk/)]. We approached INVOLVE as we were interested in exploring if we could identify a lay person who might have interest in secondary research and/or informing policy/decision-making through reviews. Those interested in participating were provided with an outline of the study and individuals who indicated greatest commitment and potential to balance the sample were selected. We drew on our own experience of developing standards and guidance, as well as on published papers by CONSORT, PRISMA, SQUIRE, AGREE and other teams working on comparable projects. 16,18,19,42

The Delphi panel was conducted entirely via the internet using a combination of e-mail and an online survey tool (www.surveymonkey.com). We began with a brainstorm round (round 1) in which participants were invited to submit personal views, exchange theoretical and empirical papers on the topic and suggest items that might could be included in the publication standards. This was done as a warm-up exercise and panel members were sent our own preliminary summary or briefing document (see Chapter 3, Details of literature search methods). These early contributions, along with our summary, were collated and summarised in a set of provisional items, which were developed into an online survey and sent electronically (via the online survey tool, SurveyMonkey^®; Survey Monkey, Palo Alto, CA, USA) to participants for ranking (round 2). Participants were asked to rank each item twice on a 7-point Likert scale (1 = strongly disagree to 7 = strongly agree), once for relevance (i.e. should an item on this theme/topic be included at all in the guidance?) and once for validity (i.e. to what extent do you agree with this item as currently worded?). Those who agreed that an item was relevant, but disagreed on its wording, were invited to suggest changes to the wording via a free-text comments box. In this second round, participants were again invited to suggest additional topic areas and items.

Each participant’s responses were collated and the numerical rankings entered onto an Excel spreadsheet. The response rate, average, mode, median and interquartile range for each participant’s response to each item were calculated. Items that score low on relevance were omitted from subsequent rounds. Further online discussion was invited on items that score high on relevance but low on validity (indicating that a rephrased version of the item was needed) and on those where there was wide disagreement about relevance or validity. Following analysis and discussion within the project team, a second list of statements was drawn up and circulated for ranking (round 3). We planned that the process of collation of responses, further e-mail discussion, and reranking would be repeated until a maximum consensus is reached (round 4 et seq.). In practice, very few Delphi panels, online or face to face, go beyond three rounds as participants tend to ‘agree to differ’ rather than move towards further consensus. 36

We had planned to report residual non-consensus as such and the nature of the dissent described. Making such dissent explicit tends to expose inherent ambiguities (which may be philosophical or practical) and acknowledges that not everything can be resolved; such findings may be more use to reviewers than a firm statement that implies that all tensions have been fixed.

Preparing teaching and learning resources

A key aim of our project was to produce publicly accessible resources to support training in realist and meta-narrative review. We anticipate that these resources will need to be adapted and perhaps supplemented for different groups of learners, and interactive learning activities added. 43 We developed, and iteratively refined, draft learning objectives, example course materials and teaching and learning support methods. We drew on our previous work on course development, quality assurance and support for interactive and peer-supported learning in health-care professionals for this aspect of our project. 34,43–45

Real-time refinement

The sponsor of this study, the National Institute for Health Research (NIHR) Health Services and Delivery Research (HSDR) programme, supports secondary research calls for rapid, policy-relevant reviews, some, though not all, of which seek to use realist or meta-narrative methods. We were asked to work with a select sample of teams funded under such calls, as well as other teams engaged in relevant ongoing reviews (selected to balance our sample), to share emerging recommendations and gather real-time data on how feasible and appropriate these recommendations are in a range of different reviews. Over the 27-month duration of this study, we used the feedback we gathered to iteratively refine our draft training materials. Training and support offered to these review teams consisted of three overlapping and complementary packages:

1. An all-comers online discussion forum via JISCmail (www.jiscmail.ac.uk/RAMESES) for interested reviewers who were doing or had previously attempted a realist or meta-narrative review. This was run via light-touch facilitation in which we invited discussion on particular topics and periodically summarise themes and conclusions (a technique known in online teaching as weaving). Such a format typically accommodates large numbers of participants since most people tend to lurk most of the time. Such discussion groups tend to generate peer support through their informal, non-compulsory ethos and a strong sense of reciprocity (i.e. people helping one another out because they share an identity and commitment)46 and they are often rich sources of qualitative data. We anticipated that this forum would contribute key themes and ideas to the quality and reporting standards and learning materials throughout the duration of the study.

2. Responsive support to our designated review teams. We anticipated that our input to these teams would depend on their needs, interests and previous experience. In our previous dealings with review teams we have been called upon (for example) to assist them in distinguishing context from mechanism in a particular paper, extracting and formalising programme theories, distinguish middle-range theories from macro or micro theories, develop or adapt data extraction tools, advise on data extraction techniques and train researchers in the use of qualitative software for systematic review.

3. A series of workshops for designated review teams and other reviewers. We planned to run a series of workshops both to provide training to fellow reviewers interested in using realist or meta-narrative reviews, but also to get feedback from them about what challenges they faced either learning about or undertaking such reviews.

Literature search

Sixteen electronic databases were searched from inception to June 2011 and citation tracking was undertaken generating 248 documents. A flow diagram outlining the disposition of documents can be seen in Figure 2. Table 1 shows the number of hits returned for the databases we searched.

FIGURE 2.

Flow diagram outlining the disposition of documents.

One of the project team (GWo) screened the abstracts and titles and included documents which claimed to be realist or meta-narrative reviews. All the documents judged to be possible realist and meta-narrative reviews were circulated to all the project team and through discussion and debate a final set of included documents were retained. We retrieved what we judged to be 35 possible realist reviews and nine meta-narrative reviews. For the possible realist reviews there was no disagreement between the project team as to inclusion (35 out of 35 were included for analysis). Out of the 11 possible meta-narrative reviews, two were judged not to be meta-narrative reviews, leaving nine documents. Tables 2 and 3 show characteristics of the documents (review title, type of document, year published and topic area) that we drew on for realist reviews and meta-narrative reviews respectively. We conducted a thematic analysis guided initially by the seven questions set out above (see Chapter 3, Details of literature search methods) to produce the briefing documents for the realist and meta-narrative Delphi panels (see Appendix 2). All the data we extracted were either entered into an Excel spreadsheet or written up directly into a draft of our briefing documents.

Our briefing documents were based on our thematic analysis which was guided by seven initial key areas (see Chapter 3, Details of literature search methods for a list of the key areas). We needed differing levels of immersion and analysis for each of the items we included in our briefing documents. Some were more straightforward to derive from our initial questions and our reading of the literature. We noted that three out of the initial seven questions [(1) What are the strengths and weaknesses of realist and meta-narrative review from both a theoretical and a practical perspective?; (2) How have these approaches actually been used? Are there areas where they appear to be particularly fit (or unfit) for purpose?; and (7) How should one choose between realist, meta-narrative and other theory-driven approaches when selecting a review methodology? How might (for example) the review question, purpose and intended audience(s) influence the choice of review method?] had overlaps and could be collapsed into one topic area for consideration by our Delphi panels. We judged that questions 1, 2 and 7 were related to matching the research question to the method. We noted that in our included reviews, most researchers had also considered this an important topic to address – often through an explanation of why they had deliberately chosen either a realist or meta-narrative review. We therefore included this issue as items 6 and 5 (for meta-narrative and realist reviews respectively) in our briefing document for our Delphi panel. These items asked the Delphi panel members to clarify what a research question would look like in a meta-narrative or realist review.

When doing realist and meta-narrative reviews ourselves, we had previously noted that such reviews often covered broad topics and needed to be progressively focused. Two of our initial questions related to these issues: (5) What is the link between realist and meta-narrative review and the policy-making process? How have published reviews been commissioned or sponsored? How have policy-makers been involved in shaping the review? How have they been involved in disseminating and applying its findings? Are there models of good practice (and of approaches to avoid) for academic-policy linkage in this area?; and (6) How have front-line staff and service users been involved in realist and meta-narrative reviews? If the answer to this is ‘usually, not much’, how might they have been involved and are there examples of potentially better practice which might be taken forward? We had asked questions 5 and 6 to ascertain if other researchers had noted this as an important process and, if they had, what approaches had they used. Within our included reviews, the breadth of the initial topic areas had been identified as a challenge and a range of different approaches had been used to focus reviews. The issue of the need to focus reviews thus seemed to us an important one to include in our briefing documents (as items 9 and 8 for meta-narrative and realist reviews respectively). Items 9 and 8 in our briefing documents asked the Delphi panel members to consider if it was important for researchers to explain how and why their review was shaped and contained.

Question 3 [What, broadly, are the characteristics of high- and low-quality reviews undertaken by realist or meta-narrative methods? What can we learn from the best (and worst) examples so far?] of our initial questions required the most immersion and analysis. With this question we had wanted to understand what processes a review team had to undertake to produce a high-quality review. As a project team we had our own ideas but wanted to explore if these were reflected in our reading of the included reviews. We first had to decide if we could agree among ourselves on which of the included reviews were high, mixed or low quality. To do this, each review was read in detail (GWo) and the characteristics of each review that determined its quality were extracted into an Excel spreadsheet. The headings on this spreadsheet were: study name; type of document; year submitted; topic area; purpose of review; understood method?; methodological comments; lessons for methods; methods for reporting; and challenges reported by reviewers and notes.

Once completed (one for realist reviews and another for meta-narrative reviews), the spreadsheet and the full-text documents were circulated to the rest of the project team and through e-mail discussion and debate, a consensus was achieved. The next process was then to reread each of the included reviews to determine which review processes were necessary to lead to a high-quality review. Again, this was led by GWo and each review process was added to a draft of the briefing documents. These drafts were circulated to the rest of the project team and a consensus achieved through discussion and debate. The briefing materials were the result of seven rounds of revisions.

The contents of our briefing materials were as follows:

• an explanation of how we would like the Delphi panel members to contribute

• background to the review methods

• methodological issues we identified for each method

• a summary of the published examples

• our preliminary thoughts on what might be included as publication standards items.

The complete briefing document circulated to the Delphi panels for realist reviews and meta-narrative reviews can be found in Appendix 2.

Delphi panel

Developing quality standards, teaching and learning resources using real-time refinement

We used a range of sources, in real-time to help us develop and refine our quality standards and teaching and learning resources. The data we used to help us came from the following sources:

• JISCMail (www.jiscmail.ac.uk/RAMESES). At the start of the project we set up an e-mail list and membership of this list grew rapidly. As of June 2014, the list has 326 members and there are regular discussions on a range of topics relating to realist and meta-narrative reviews.

• Methodological support to review teams. Over the course of this project the project team members have provided differing levels of methodological support to reviewers undertaking realist and meta-narrative reviews. This has ranged from providing answers to questions raised by e-mail or on JISCMail to regular face-to-face meetings. The level of support needed by each team differed considerably depending on each team’s initial level of expertise. Table 8 provides an overview of the projects we provided more in-depth methodological support to and also brief details of the nature of each type of support provided. When providing methodological support we contemporaneously made notes on issues and topics that might be relevant in helping us in this part of the project. An example of the type of records we made of our discussions with a review team we provided methodological support to can be found in Appendix 7.

Changes to protocol

Near the start of this project we published our project protocol. 99 During the course of the project we varied two aspects of our protocol, as described in the sections below.

Limitations

To develop the briefing materials for our Delphi panels we undertook a narrative review. This review has limitations that are likely to have introduced a number of biases and so – potentially at least – limit the inferences that can be made from the included reviews. For example, the search process for the review, despite being developed by an expert librarian, was not exhaustive. All the screening for inclusion and exclusion was undertaken by one screener and no quality checks were undertaken. Both processes may well mean that we are likely to have missed some reviews. Once reviews had been included, data extraction was undertaken by one researcher, and omissions in data extraction are likely to have occurred. However, all the included reviews and the data extraction spreadsheet were circulated to all project team members and so a degree of informal quality checking did occurr. Deciding what should be included in the Delphi panels’ briefing materials was undertaken by the entire project team. We are aware that any item or topic included in the briefing materials was done so as a result of our subjective interpretations, raising questions about reproducibility. However, the briefing documents we produced were not an end product in itself, but the starting point for the Delphi panels to build a consensus. As such, we expected that changes would occur as we ran each round of the Delphi process and, so, were not as concerned that any omissions as a result of the review’s limitations process would have that large an impact on the final publication and quality standards.

We recognise that there is much more to cover in terms of the breadth and depth of the teaching and learning resources we have produced. Because realist and meta-narrative reviews are both relatively new review methods, the wish list we were able to elicit from our fellow reviewers when using these methods is quite long. Given the time and resources allocated for this project we elected to focus on providing depth, rather than breadth on the issues that were most challenging. With time, we hope to use the community of practice we have developed to address more and more methodological challenges.

As experience grows with the use of these methods, it is very likely that the resources we have produced will need to be updated. We welcome and invite methodological development in realist and meta-narrative reviews. We expect that what we have produced should be gradually refined and updated as methodological developments take place with increasing use of realist and meta-narrative reviews. Thus we view the publication and quality standards and teaching and learning resources more as a starting point than definitive resources that must not be altered in any way.

We are aware that both realist and meta-narrative reviews are used for secondary research on a wide range of topics and by reviewers from a broad range of disciplines. The level of expertise of the users of our resources will also vary considerably, from novice to seasoned reviewer. These two aspects mean that some latitude is needed in the use of the resources we have produced. For example, not all the publication standard’s items will always be applicable when reporting all reviews. Or when assessing the quality of a review, there may be justifiable reasons for a review to not meet some quality criteria. We have tried to anticipate the varied uses that realist and meta-narrative review might be put to by providing a degree of flexibility in our standards. For example, in our publication standards, if adaptations are made to the review method (as originally described), then reviewers are invited to provide an explanation for any such adaptations.

Research recommendations and implications for practice

In common with many quality and reporting standards there is a dearth of research to demonstrate that such standards necessarily change practice and improve the quality of research. 18 This will also be true for the standards we have produced and, therefore, research to demonstrate a change in practice and improvement in the quality of realist and meta-narrative reviews is needed.

As experience with realist and meta-narrative reviews grows and more are undertaken, new methodological insights are likely to occur. These need to be captured and analysed to determine if the quality and publication standards we have produced continue to be fit for purpose or need to be updated. Ideally, further funding might enable a project similar to this one, i.e. RAMESES II, to address the updating of the standards, though as much groundwork has already been done a more truncated project may suffice.

Our training materials are focused on what we were able to identify as being the processes that fellow reviewers found the most challenging to execute. There are additional processes that we have not focused on and further work is needed to identify these. With our training materials we have chosen to produce learning materials that teach by using examples from published reviews and through use of learning and reflection activities. They have not been formally evaluated and are likely to benefit from iterative cycles of evaluation and updating based on findings.

Finally, both realist and meta-narrative reviews are relatively new approaches and as with any approach, capacity building is an issue. This project has enabled the project team to support and build capacity with a number of researchers and set up an e-mail mailing list to bring researchers together. A pressing need for the future is to maintain the momentum generated by this project. To this end, the JISCMail e-mail list continues to run but we invite any researchers interested in either method to join us in helping to build capacity.

Contributions of authors

Dr Geoff Wong is a co-investigator in the RAMESES project and is a senior lecturer in primary health care at Queen Mary University of London, London, UK.

Professor Trish Greenhalgh is the principal investigator on the RAMESES project and is Professor of Primary Health Care and Dean for Research impact at Queen Mary University of London, London, UK.

Dr Gill Westhorp is an evaluation expert and has used both the realist evaluation and synthesis methods on a range of projects internationally. She is the Director of Community Matters, a research consultancy in Adelaide, SA, Australia.

Professor Ray Pawson is a co-investigator in the RAMESES project and is Professor of Social Research Methodology at the University of Leeds, Leeds, UK.

Geoff Wong carried out the literature review with the help of Jeanette Buckingham (Librarian, John W. Scott Health Science Library, University of Alberta, AB, Canada). Geoff Wong, Trish Greenhalgh, Gill Westhorp and Ray Pawson analysed the findings from the review and produced the materials for the Delphi panel. They also analysed the results of the Delphi panel and real-time data to produce the publication standards, the methodological guidance and teaching and learning materials. Geoff Wong, Trish Greenhalgh, Gill Westhorp and Ray Pawson conceived of the study and participated in its design. Geoff Wong co-ordinated the study and ran the Delphi panels. All authors read and approved the final manuscript.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.

Publications

Wong G, Greenhalgh T, Westhorp G, Pawson R. RAMESES publication standards: meta-narrative reviews. BMC Med 2013;11:20.

Wong G, Greenhalgh T, Westhorp G, Pawson R. RAMESES publication standards: realist syntheses. BMC Med 2013;11:21.

Jagosh J, Macaulay AC, Salsberg J, Bush PL, Henderson J, Sirett E, et al. Uncovering the benefits of participatory research: Implications of a realist review for health research and practice. Milbank Q 2012;90:311–46.

Noble D, Mathur R, Dent T, Meads C, Greenhalgh T. Risk models and scores for type 2 diabetes: systematic review. BMJ 2011;343:d7163.

Teleconference with Durham Health Services and Delivery Research review team

Advancing Realist Research Conference

© 2011 Greenhalgh et al. ; licensee Biomed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited.