Quality and reporting standards, resources, training materials and information for realist evaluation: the RAMESES II project

Article history

The research reported in this issue of the journal was funded by the HS&DR programme or one of its preceding programmes as project number 14/19/19. The contractual start date was in March 2015. The final report began editorial review in March 2017 and was accepted for publication in July 2017. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Geoff Wong is a member of the National Institute for Health Research Health Technology Assessment programme Primary Care Panel, and is a panel member of the Health and Safety Executive External Peer Review Panel Evaluation Governance Group. During the course of the project Gill Westhorp worked as a consultant and consulting academic undertaking realist evaluations and reviews, and provided some capacity building and some PhD supervision on a commercial basis. These activities were not undertaken under the auspices of this project.

Copyright statement

© Queen’s Printer and Controller of HMSO 2017. This work was produced by Wong et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2017 Queen’s Printer and Controller of HMSO

What is realist evaluation?

Realist evaluation was developed by Pawson and Tilley in the 1990s,10 originally in the field of criminology, to address the question, ‘what works for whom in what circumstances and how?’ in criminal justice interventions. This early work highlighted the following points:

• Social programmes (closely akin to what health service researchers call complex interventions) are an attempt to address an existing social problem (i.e. to create some level of social change).

• Programmes ‘work’ by enabling participants to make different choices (although choice-making is always constrained by such things as participants’ previous experiences, beliefs and attitudes, opportunities and access to resources).

• Making and sustaining different choices may require a change in a participant’s reasoning (e.g. in their values, beliefs, attitudes or the logic they apply to a particular situation) and/or the resources (e.g. information, skills, material resources, support) they have available to them. Programmes provide opportunities and resources. The interaction between what the programme provides and the participant’s ‘reasoning’ is what enables the programme to ‘work’ and is known as a ‘mechanism’.

• Programmes work in different ways for different people (that is, the contexts within programmes can trigger different change mechanisms for different participants).

• The contexts in which programmes operate make a difference to the outcomes they achieve. Programme contexts include features such as social, economic and political structures, organisational context, programme participants, programme staffing, geographical and historical context, and so on. In realist terms, context does not simply denote spatial, geographical or institutional locations. Context refers, among other things, to the sets of ‘social rules, norms values and interrelationships’ that operate within these locations. 10

• Some aspects of the context enable particular mechanisms to be triggered. Other aspects of the context may prevent particular mechanisms from being triggered. That is, there is always an interaction between context and mechanism, and that interaction is what creates the programme’s impacts or outcomes: context + mechanism = outcome.

• Because programmes work differently in different contexts and through different change mechanisms, they cannot simply be replicated from one context to another and automatically achieve the same outcomes. Theory-based understandings about ‘what works for whom, in what contexts, and how’ are, however, transferable.

• Therefore, one of the tasks of evaluation is to learn more about: ‘what works’, in what respects and to what extent, including intended and unintended outcomes; ‘for whom’, that is, for which subgroups of participants; ‘in which contexts’; and ‘what mechanisms are triggered by what programmes in what contexts’.

A realist evaluation approach assumes that programmes are ‘theories incarnate’. That is, whenever a programme is implemented, it rests on a theory about what ‘might cause change’, even though that theory may not be explicit. One of the tasks of a realist evaluation is, therefore, to make the theories underpinning a programme explicit, by developing clear hypotheses about how, and for whom, programmes might ‘work’. The implementation of the programme, and the evaluation of it, then tests those hypotheses. This means collecting data, not just about programme impacts or the processes of programme implementation, but about the specific aspects of context that might impact on programme intended and unintended outcomes, and about the specific mechanisms that might be creating change.

Pawson and Tilley10 also argue that a realist approach has particular implications for the methods required to evaluate a programme. For example, rather than comparing changes for participants who have undertaken a programme with a group of people who have not (as is done in randomised controlled or quasi-experimental designs), a realist evaluation compares context–mechanism–outcome configurations (CMOCs) within programmes. It may ask, for example, whether a programme works more or less well, and/or through different mechanisms, in different localities (and if so, how and why) or for different subgroups of the population. Furthermore, they argue that different stakeholders will have different information and understandings about how programmes are supposed to work and whether or not they in fact do so. Data collection processes (interviews, focus groups, questionnaires and so on) should be constructed to identify and collect the particular information that those stakeholder groups will have, and thereby to confirm, refute or refine theories about how and for whom the programme ‘works’.

Realist evaluation is underpinned by a realist philosophy of science (‘realism’). 11 Philosophically speaking, realism can be thought of as sitting between positivism (‘there is a real external world which we can come to know directly through experiment and observation’) and constructivism (‘given that all we can know has been interpreted through human senses and the human brain, we cannot know for sure what the nature of reality is’). However, it is worth pointing out that this is not to suggest that ‘constructivism’ and ‘positivism’ represent opposite poles on the same continuum. Realism holds that there is a real social world but that our knowledge of it is amassed and interpreted (partially and/or imperfectly) via our senses and brains, and filtered through our language, culture and past experience. In other words, realism sees the human agent as operating in a wider social reality, encountering experiences, opportunities and resources, and interpreting and responding to the world within particular personal, social, historical and cultural frames. For this reason, different people respond differently to the same experiences, opportunities and resources. Hence, a programme (or, in the language of health services research, a complex intervention) aimed at improving health outcomes is likely to have different levels of success with participants in different contexts, and even in the same context at different times.

The need for standards and training materials in realist evaluation

The RAMESES JISCMail listserv [www.jiscmail.ac.uk/RAMESES (an e-mail list for discussing realist approaches)] postings suggest that enthusiasm for realist evaluation and belief in its potential for application in many fields have outstripped the development and application of robust quality standards in the field. Two important prior publications have systematically shown that many so-called ‘realist evaluations’ were not applying the concepts appropriately and were, as a result, producing potentially misleading findings and recommendations. 12,13

Pawson and Manzano-Santaella, in their paper, ‘A realist diagnostic workshop’, used case examples of flawed realist evaluations to highlight three common errors in such studies. 13 First, while it is possible to show associations and correlations in data from many types of evaluation, the focus of a realist evaluation should be to explore and explain why such associations occur. Second, they explain what may constitute valid data for use in realist evaluation. Producing a realist explanation is likely to require a mix of data types to provide explanations and support for the relationships within and between CMOCs. Third, realist explanations require CMOCs to be produced. Pawson and Manzano-Santaella note that some realist evaluations have presented finely detailed lists of contexts, mechanisms and outcomes, but have failed to produce a coherent explanation of how these contexts, mechanisms and outcomes were linked and related, or not related, to each other. Pawson and Manzano-Santaella called for greater emphasis on elucidating programme theory (the theory about what a programme or intervention is expected to do and, in some cases, how it is expected to work) expressed as CMO configurations.

Marchal et al. 12 undertook a review of the realist evaluation literature in health systems research to quantify and analyse the field. They identified 18 realist evaluations and noted a range of challenges that arose for researchers. Absence of prior theoretical and methodological guidance appeared to have led to recurring problems in the realist evaluations they appraised. Marchal et al. 12 noted that ‘[t]he philosophical principles that underlie realist evaluation are variably interpreted and applied to different degrees’. Different researchers had conceptualised concepts used in realist evaluation such as ‘middle-range theory’, ‘mechanism’ and ‘context’ differently. This, they concluded, was often related to fundamental misunderstandings, and the rigour of the evaluation suffered as a result.

These two papers12,13 showed that, although realist evaluation had been embraced by parts of the health research community, it had also proven a challenging task for some who were unfamiliar with the practical application of realism. Both sets of authors called for methodological guidance to allay misunderstandings about the purpose, underlying philosophical assumptions, analytic concepts and methods of realist evaluation.

Objectives

The project had both strategic and operational objectives, and, because it was funded through the health sector, the objectives were framed in relation to health. However, representatives from beyond the health sector were involved to ensure that the products were relevant to any realist evaluation.

Overview of methods

We first provide a brief overview of the range of methods we used to meet the objectives set out above and of how they related to each other. The methods we used in this project closely resemble those we used in another project (the RAMESES project), which developed methodological guidance, reporting standards and training materials for realist and meta-narrative reviews. 14 We have previously published a protocol paper that outlined the methods we intended to use in this project. 15 The following methods sections outline, in more detail, specific aspects of the methods used.

To fulfil operational objectives 1 and 2, we undertook a thematic review of the literature. Findings were supplemented by our content expertise and with feedback collated from presentations and workshops for researchers using or intending to use realist evaluation. We synthesised our findings into briefing materials on realist evaluation for the Delphi panel. We recruited members to the Delphi panel, which had wide representation from researchers, students, policy-makers, evaluators, theorists and research sponsors. We used the briefing materials to inform the Delphi panel in preparation for the task, so they could contribute to developing standards (objective 3). For the advice and consultancy to realist evaluations (objective 4a), we drew on our experience in conducting realist evaluations and developing and delivering education materials, but also on relevant feedback from the Delphi panel, an e-mail list on realist research approaches (www.jiscmail.ac.uk/RAMESES) and the evaluations teams we had supported in the past. To help us refine our reporting standards (objective 5), we captured methodological and other challenges that arose within the realist evaluation projects to which we provided methodological support. All of these sources fed into the reporting standards, quality standards and resources and training materials (objective 7). We did not set specific time points when we would refine the drafts of our project outputs. Instead, we iteratively and contemporaneously fed the data we captured into our draft reporting standards, quality standards and resources and training materials, making changes gradually. Only our Delphi panel ran within a specific time frame. The final guidance and standards were, therefore, the product of continuous refinements. To understand and develop information and resources for patients and other lay participants in realist evaluation (objective 8), we convened a group consisting of patients and the public. We addressed objective 9 through academic publications, online resources and delivery of presentations and workshops. The project was overseen by a Project Advisory Group, which comprised three independent members (see Acknowledgements). This group met with the project team on three occasions (May 2015, November 2015 and May 2016) and provided advice to the project team. Figure 1 provides a pictorial overview of how the different methods we used fed into each other.

FIGURE 1.

Overview of study processes.

Details of literature search methods

With input from an expert librarian, we identified reviews, scholarly commentaries, models of good practice and examples of (alleged) misapplication of realist evaluation. To identify the relevant documents we refined and developed the search used by Marchal et al. 12 for a previous review on a similar topic, and also applied contemporary search methods designed to identify ‘richness’ when exploring complex interventions. 16,17

A search was conducted on 3 March 2015 across 10 databases. Free-text terms were selected to describe realist methods and thesaurus terms were used where available (see Appendix 1). The following databases were searched:

• Cumulative Index to Nursing and Allied Health Literature (CINAHL; via EBSCOhost)

• The Cochrane Library (Wiley Online Library)

• Dissertations & Theses database (ProQuest)

• EMBASE (via OvidSP)

• Education Resources Information Center (ERIC; via EBSCOhost)

• Global Health (via OvidSP)

• MEDLINE and MEDLINE In-Process & Other Non-Indexed Citations (via OvidSP)

• PsycINFO (via OvidSP)

• Scopus, Science Citation Index (SCI), Social Science Citation Index (SSCI) & Conference Proceedings, Citation Index – Science (CPCI-S)

• Web of Science Core Collection (Thomson Reuters Corporation, New York, NY, USA).

A forward citation search was conducted via the Web of Science Core Collection for the following key text: Pawson R, Tilley N. Realistic Evaluation. London: Sage; 1997. 10

No language or study design filters were applied. We included any document that referred to or claimed to be a realist evaluation that used the approach as set out by Pawson and Tilley in their key publication, Realistic Evaluation. 10 Documents were excluded if they were not realist evaluations, published prior to the year 2000, book reviews, letters and comment. We set the cut-off point at 2000, as we assumed that evaluations based on Pawson and Tilley’s work would begin appearing in the literature from this point onwards. All citation screening was undertaken by Geoff Wong. The whole searching process, from start to the retrieval of all full-text documents, took approximately 1 month.

We decided that, because of the narrow purpose of our review and the number of relevant citations retrieved, we would stop analysing data when we had reached thematic saturation. As a strategy to manage the potential number of realist evaluations, we decided to start our analysis and synthesis from the most recent (i.e. from 2015) realist evaluations and work ‘backwards’. The decision on when thematic saturation had been reached was made in discussion with the whole project team. For both practical reasons (e.g. resource constraints) and academic ones (no new data), we stopped including new papers when there was agreement that saturation of themes had been reached. Thematic saturation was reached once the group agreed by consensus that the new realist evaluations identified contained no new themes or only subthemes that related to the three questions listed below in bullet points.

The thematic analysis was led by Geoff Wong, who undertook all stages of the review and shared findings with the rest of the project team so that discussion, debate and refinement of interpretations of the data could take place. Findings were shared by e-mail and, when necessary, face-to-face meetings were conducted to discuss interpretations of the data.

In undertaking our thematic analysis, we familiarised ourselves with the included evaluations to identify patterns in the data. Aware that the purpose of the review was to produce briefing documents for the Delphi panel, we considered the following questions:

• What is considered by experts in realist evaluation to be current best practice (and what is the range and diversity of such practice)?

• What do experts in realist evaluation, and other researchers who have undertaken a realist evaluation, believe counts as high quality and necessary to report?

• What issues do researchers struggle with (based on thematic analysis of postings on the RAMESES JISCMail list archive as well as the published literature)?

In the panels, we wanted to achieve a consensus on quality and reporting standards, and thus what we needed from our review of the literature were data to inform us on what might constitute quality in executing and reporting realist evaluations. We accepted that we might need to refine, discard or add additional questions and topic areas in order to better capture our analysis and understanding of the literature as these emerged from our reading of the evaluations.

Data were extracted to a Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA) spreadsheet that we iteratively refined to capture the data needed to produce our briefing materials. This review was undertaken in a short time frame. The time taken from obtaining full-text documents to producing the final draft for circulation of the briefing documents was approximately 12 weeks. The output of this phase was a provisional summary that addressed the questions above and highlighted, for each question, the key areas of knowledge, ignorance, ambiguity and uncertainty. This was distributed to the Delphi panel (as our briefing document) as the starting point for its work.

Our purpose in identifying published reviews was not to complete a census of realist evaluations. We make no claims that the review we undertook was exhaustive; thus, we never intended that it should be published as a stand-alone piece of research. In other words, the purpose of our review was not to produce definitive summaries in response to the themes above but to prepare a baseline set of briefing materials for the Delphi panel, and to deliberate on and add to them in the next step. As such, the review we undertook would be best considered as being a rapid, accelerated or truncated thematic review. Such an approach will predictably produce limitations, and these are discussed in Chapter 4, Limitations.

Details of online Delphi process

We recruited Delphi panel members purposefully, to ensure that we had representation from evaluators, researchers, funders, journal editors and experts in realist evaluation. Individuals were recruited through relevant organisations and targeted e-mails, and also through personal contacts and recommendations. Those interested in participating were provided with an outline of the study, and individuals who indicated the greatest commitment and potential to balance the sample were selected.

The Delphi panel was run online using SurveyMonkey (SurveyMonkey, Palo Alto, CA, USA). Participants in round 1 were provided with the briefing materials we developed from the literature review and were invited to suggest what might be included in the reporting standards. Responses were analysed and fed into the design of questionnaire items for round 2.

In round 2 of the Delphi Panel, participants were asked to rank each potential item twice on a 7-point Likert scale (1 = strongly disagree to 7 = strongly agree), once for relevance (i.e. ’Should an item on this theme/topic be included at all in the guidance?’) and once for validity (i.e. ’To what extent do you agree with this item as currently worded?’). Those who agreed that an item was relevant, but disagreed on its wording, were invited to suggest changes to the wording via a free-text comments box. In this second round, participants were again invited to suggest additional topic areas and items. We did not prespecify stop-points for establishing when consensus has been achieved. This was because we wanted to have the flexibility to return to the Delphi panel items that we judged might need further input. Although we accept that this may have enabled us to preferentially return some items and not others, we guarded against this by sending all Delphi panel members an end-of-round report detailing all the findings, changes made to the text and items to be returned to the next round. Panel members were invited to contact us should they have any concerns with the items that were not returned for re-rating, such as believing that the item should be returned to the panel, or disagreeing with wording changes.

Participants’ responses were collated and the numerical rankings were entered onto an Excel spreadsheet. The response rate, average, mode, median and interquartile range (IQR) for each item was calculated. Items that scored low on relevance were omitted from subsequent rounds. We invited further online discussion on items that scored high on relevance but low on validity (indicating that a rephrased version of the item was needed) and on those for which there was wide disagreement about relevance or validity. The panel members’ free-text comments were also collated and analysed thematically.

Following analysis and discussion within the project team, we drew up a second list of statements that were circulated for ranking (round 3). Round 3 contained items for which consensus had not yet been reached. For items on which consensus had been reached, we did not return these to rounds 3, 4 or beyond for panel members to re-rate, even if we had made changes to the wording. This was because, when we undertook the RAMESES project, we had received informal feedback from the Delphi panel members indicating that round 2 of the online Delphi process had been very time-consuming. We were advised that to retain a high response rate for subsequent rounds, we should minimise the time commitment we asked of panel members. We planned that the process of collating responses, further e-mail discussion and re-ranking would be repeated until a maximum consensus was reached (rounds 4, 5, and so on). In practice, very few Delphi panels, online or face to face, go beyond three rounds because participants tend to ‘agree to differ’ rather than move towards further consensus. We used e-mail reminders to optimise our response rate from Delphi panel members. We considered consensus to be achieved when the median score was 6 or above.

We planned to report residual non-consensus as such and to report the nature of the dissent described (if any). Making such dissent explicit tends to expose inherent ambiguities, which may be philosophical or practical, and acknowledges that not everything can be resolved; such findings may be more use to those who use realist evaluation than a firm statement that implies that all tensions have been fixed. We used the findings from the Delphi panel to develop the reporting standards and methodological quality standards for realist evaluations.

Developing quality standards

The quality standards were designed to support professional development, assist evaluators to assess the quality of various aspects of the evaluation process and to assist reviewers with meta-evaluation (i.e. assessing the quality of evaluations).

To develop the quality standards, we drew on the following sources of data:

• free-text comments from participants and findings from the Delphi panels

• personal expertise as evaluators, researchers, peer reviewers and trainers in the field

• feedback from participants at workshops and training sessions run by members of the project team

• comments made on RAMESES JISCMail.

The data from the sources above were collated contemporaneously and discussed within the project team. Iterative cycles of discussion and revisions for content and clarity of the drafts were needed to develop the standards. Box 1 provides an illustration of how we drew on the data sources to produce the quality standards.

Developing, delivering and refining resources and training materials for realist evaluation

An important part of our project was to produce publicly accessible resources to support training in realist evaluations. We anticipated that these resources will need to be adapted, and perhaps supplemented, for different groups of learners, and interactive learning activities added. We developed, and iteratively refined, draft learning objectives, example course materials and teaching and learning support methods. We drew on a range of sources to inform the content and format of our training materials as well as our experience as trainers and consultants on realist evaluations.

We sought out examples of the kinds of requests that are often made by evaluators for support on realist evaluation, for example using the rich archive of postings on the RAMESES JISCMail listserv from both novice and highly experienced practitioners, going back 3 years. We also proactively asked the list members for additional examples, and used our empirical data from the Delphi panel and our literature review to identify relevant examples. Finally, we sought input from UK RDS staff interested in realist evaluation to describe the kind of problems people bring to them, and where they feel that further guidance, support and resources are needed.

We used a thematic approach to classify examples into a list of problems and issues, each with a corresponding training need(s) and resources to address them. These were developed iteratively in regular discussions and meetings of the research team. Our goal was to develop a coherent and comprehensive curriculum for training realist researchers and for ‘training the trainers’.

Support and consultancy to realist evaluations

The support we offered to fellow evaluators and researchers using realist evaluations consisted of two overlapping and complementary levels:

1. Online discussion and support via JISCMail for evaluators and researchers, at any level, interested in or undertaking a realist evaluation. When questions or issues were raised, either one of the project team or another list member would reply. Where necessary, summaries were made of discussions and clarification was provided by members of the project team.

2. Direct requests for support and training. During the course of the study, members of the project team were frequently approached to provide methodological support to realist evaluation projects. The exact content, nature and duration of the support provided was discussed between the relevant team members to ensure that what was provided met the needs of those who requested the support.

Realist evaluation and ‘training the trainers’ workshops

Throughout the 24 months of the project, members of the project team offered training workshops to other evaluators, researchers and patient organisations on an as-requested basis. When asked to provide a workshop, the logistics and content of each workshop were discussed between the relevant project team member and the hosts.

For the ‘training the trainers’ workshops, we engaged with the NIHR’s RDS. We did this by e-mailing each regional service and also asking for expressions of interest via e-mail lists and personal contacts.

Develop, deliver and refine information and resources for patients and other lay participants in realist evaluation

To develop these resources, we convened a panel of lay participants with the help of the Patient and Public Involvement Co-ordinator from the Nuffield Department of Primary Care Health Sciences at the University of Oxford. We sought to invite lay participants who had been involved in research studies and came from a range of backgrounds and ages. During the panel, we sought to understand what lay participants might wish to know if they were to participate in a realist evaluation and provided examples of the potential materials for their consideration.

Literature search

We searched 10 electronic databases from inception (where applicable) to March 2015 and, along with citation tracking, retrieved 4426 documents after removal of duplications. A total of 1498 duplicates were removed, along with a further 737 papers that did not meet our inclusion criteria. A total of 2191 papers were screened by title and abstract for inclusion with 1503 excluded at this stage. Figure 2 shows the disposition of the documents and Table 1 the number of citations returned for each database searched.

FIGURE 2.

Flow diagram outlining the disposition of documents.

One of the project team (GWo) screened the abstracts and titles and included documents that claimed to be realist evaluations. In total, 152 documents were judged to be realist evaluations. Because of the narrow focus of our review of the literature on realist evaluations, as discussed in Chapter 2, Details of literature search methods, we worked ‘backward’ from 2015 to earlier years and sought to stop analysis at the point of thematic saturation. We achieved thematic saturation after analysis of 37 out of the 152 realist evaluations. Out of these realist evaluations, 32 (from years 2015 and 2014 inclusively) evaluated health-related topics, and five (from years 2015 to 2012 inclusively) evaluated non-health-related topics. We made this distinction to ensure that we analysed realist evaluations that covered a range of topic areas, as the approach is used in a broad range of topic areas beyond health research. Hence, Table 2 shows only the characteristics of the documents we analysed (evaluation title, type of document, year submitted for publication and topic area) and drew on to produce our briefing document for the Delphi panel.

Because many evaluation reports are not published and our search strategy focused on published materials, the great majority of documents we analysed were journal articles about evaluations rather than complete evaluation reports. We acknowledge that full evaluation reports may have provided greater detail. However, because journal articles usually require a description of both methods and findings, our focus was methodological and the literature review served only to identify issues to refer to the Delphi panel; therefore, we remain confident that the sample was adequate for the task.

We conducted a thematic analysis guided, initially, by the three questions set out above (see Chapter 2, Details of literature search methods) to produce the briefing documents for the realist evaluation Delphi panel (see Appendix 2). All the data we extracted were either entered into an Excel spreadsheet or written up directly into a draft of our briefing document. Of the three questions set out above, two refer to what experts in realist evaluation and researchers who have undertaken a realist evaluation consider to be best practice and high quality. Much of this information was contained in the documents listed in Table 2, but we also had to supplement our understanding by drawing on more methodological documents. 1,10,12,13

Our first question [what is considered by experts to be current best practice (and what is the range and diversity of such practice)?] related to perceptions of methodological rigour in the execution of realist evaluations. Addressing this question required the most immersion and analysis. With this question, we wanted to understand expert opinions about best practice to produce a high-quality realist evaluation. As a project team, we had our own ideas, but wanted to explore whether or not these were reflected in the included evaluations. We first had to decide whether or not we could agree among ourselves on which of the evaluations we analysed were of high, mixed or low quality. To do this, each evaluation was read in detail (GWo) and selected characteristics were extracted into an Excel spreadsheet. The headings on this spreadsheet were study name, type of document, year submitted, country, topic area, purpose of evaluation, understand realism?, methodological comments, lessons for methods, methods for reporting and challenges reported by reviewers’ notes.

Once completed, the spreadsheet and the full-text documents were circulated to the rest of the project team. Through e-mail discussion and debate, a consensus was achieved on which studies were deemed high, mixed or low quality. The next step in the process was to re-read each of the included evaluations to determine which evaluation practices and processes were necessary to lead to a high-quality evaluation. Later on in the project, to develop reporting standards for realist evaluations, we used these findings to inform what needed to be reported to ensure that sufficient information was available to the reader, so that they were able to make judgments about methodological rigour. This addressed our second question (what do experts and other researchers believe count as high quality and necessary to report?). Again, this was led by Geoff Wong, and each issue that needed addressing was added to a draft of the briefing documents. To further strengthen the inferences we made on issues that needed to be addressed and, hence, included in our briefing materials, we looked back through the archives of the RAMESES JISCMail e-mail listserv to identify if the issues we had included had also been raised by other researchers. We also drew on the methodological issues raised in methods papers on realist evaluations in a similar way. 12,13

The drafts of briefing materials were circulated to the project team and a consensus was achieved through discussion and debate. The briefing materials were the result of four rounds of revisions.

The contents of our briefing materials were as follows:

• terminology

• philosophical basis of realist evaluation

• classification

• title

• rationale for using realist evaluation

• methods

• data collection methods

• programme theory

• findings

• conclusion

• recommendations.

The complete briefing document circulated to the Delphi panels for realist reviews and meta-narrative reviews can be found in Appendix 2.

Delphi panel

We ran the Delphi panels between May 2015 and January 2016. We recruited 35 panel members from 27 organisations across six countries. The panel members comprised evaluators of health services (23), public policy (nine), nursing (six), criminal justice (six), international development (two), contract evaluators (three), policy- and decision-makers (two), funders of evaluations (two) and publishing (two) (note that some individuals had more than one role).

We started round 1 in June 2015 and circulated the briefing materials document to the panel. We sent two chasing e-mails to all panel members, and within 8 weeks all panel members who indicated that they wanted to provide comments had done so. In round 1 of the Delphi panel, 33 members provided suggestions for items that should be included in the reporting standards and/or comments on the nature of the standards themselves. We used the suggestions from the panel members and the briefing document as the basis of the online survey for round 2.

Round 2 started at the end of September 2015 and ran until early November 2015. Panel members were invited to complete our online survey and asked to rate each potential item for relevance and validity. A copy of this survey can be found in Appendix 3. Where needed, up to three reminder e-mails were sent to the panel members. For round 2, the panel was presented with 22 items to rank. The overall response rate across all items for this round was 76%. Once the panel had completed its survey, we analysed their ratings for relevance and validity. Full details of the round 2 results can be found in Table 3. We also produced a post-round briefing document from round 2, which detailed for each item:

• the response rate

• mode

• median

• IQR

• the action we took for each item based on the panel’s ratings

• an anonymised list of all the free-text comments made.

Based on the rankings and free-text comments, our analysis indicated that two items needed to be merged and one item removed. Minor revisions were made to the text of the other items based on the rankings and free-text comments. After discussion within the project team, we judged that only one item (the newly created merged item) needed to be returned to round 3 of the Delphi panel. Prior to the start of round 3, the post-round briefing document from round 2 was circulated to panel members. We did not receive any communication indicating that the panel members disagreed with the actions we undertook in response to their ratings and free-text comments from round 2.

For round 3, we asked the panel to consider again only the single item for which a consensus had not been reached. We produced an online survey for round 3 and, again, asked them to rate the item for relevance and validity. To keep the panel updated, we provided it with our post-round briefing document from round 2 (available on request from authors). Round 3 ran from late November 2015 to early January 2016. A copy of this survey can be found in Appendix 4. Two reminder e-mails were sent to the panel members. Once the panel had completed its survey, we analysed its ratings for relevance and validity (Table 4). The response rate for the single item included in round 3 was 80%. We produced a post-round briefing document from round 3 and circulated this to all our panel members for the sake of completeness (available on request from authors). We did not receive any communication indicating that the panel members disagreed with the actions we undertook in response to their ratings and free-text comments from round 3. Overall, consensus was reached within three rounds on both the content and wording of a 20-item reporting standard.

Using the data we gathered from the three rounds of the Delphi panel, we produced a final set of items to be included in the reporting for realist evaluations. These were published in June 2016 in BMC Medicine, an open-access journal. 55 Within this publication, we have provided an ‘example’ for each standard; that is, an example of good practice drawn from published evaluations. Our reporting standards have also been accepted and listed on the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network, a resource centre for good reporting of health research studies (www.equator-network.org).

Developing quality standards

We developed quality standards for two user groups, which are set out using rubrics:

1. evaluators and peer reviewers of realist evaluations

2. funders or commissioners of realist evaluations.

Developing, delivering and refining resources and training materials for realist evaluation

Two types of educational materials were developed: resource materials (made freely available online) and training materials.

The resource materials focus on topic areas that the literature review, Delphi panel and discussion list had identified as being most challenging and/or required further clarification. To make the materials accessible, we established a rough word limit of around 1000 words per topic, with very clearly defined topics, and written in as plain English as possible. This means that more introductory materials are accessible to those with very limited prior knowledge or experience of realist evaluation. It also means that more advanced readers can search for specific topics without having to wade through the more introductory resources, and that additional materials can easily be added in future.

Each of the resource materials provides references for those who wish to understand a topic area in greater depth. Many provide examples from previously completed evaluations to illustrate key points. Some provide direct links to more detailed articles on the same topic and/or to additional resources. For example, the ‘realist interview’ resource links to a longer journal article and to a list of questions that can be used in realist interviews or as a guide to start developing realist interview questions.

Most of the resource materials were written by one or two individuals within the project team and were then peer reviewed internally by a realist methodological expert. A couple were written by people outside the project team with interests in specific topics. These were each reviewed by at least two team members. The resource materials are open access and can be found on the RAMESES project website [http://ramesesproject.org (accessed 15 September 2017)].

An overview of the topic areas currently covered may be found in Table 7. Additional topics are also planned by members of the project team, to be added at a later date.

Support and consultancy to realist evaluations

We were approached by a wide range of evaluators who asked us for help with their realist evaluation projects. Selection was done on a ‘first-come, first-served’ basis. An overview of the 17 realist evaluation projects we provided methodological support and consultancy to may be found in Table 8.

Realist evaluation and ‘training the trainers’ workshops

We provided training workshops to organisations interested in learning more about realist evaluation on a first-come, first-served basis. When we were contacted, we entered into discussion with the individuals who contacted us and arranged bespoke training to meet their needs. These ranged from short 15-minute presentations to whole-day workshops. Table 9 lists the 29 realist evaluation presentations or workshops we ran nationally and internationally.

In terms of ‘training the trainers’ workshops, we wanted to build capacity within the NIHR RDS. We discussed what the training needs might be initially with colleagues at the RDS London’s East London Team. Their feedback was supplemented with comments we received from our project’s Advisory Group. After the publication of our project’s protocol paper,15 we were contacted by colleagues from the RDS East Midlands and, with their assistance, organised two workshops for regional staff.

Develop, deliver and refine information and resources for patients and other lay participants in realist evaluation

To develop resources for patients and other lay participants in realist evaluation, we first discussed, within the project team, what might be required. We also sought input from our project’s Advisory Group. We then drafted a specimen document outlining what a realist evaluation is and when it might be used, and this also explained what might be expected of a participant when taking part in a realist evaluation. We did not develop any materials for seeking ethics approvals as we established that organisations or institutions had a diverse range of processes, and so a one-size-fits-all set of documents was not likely to be useful. To gain feedback on the documents, we convened a 90-minute face-to-face meeting with six members of the public from diverse backgrounds in September 2016 in Oxford (only five out of the six invited participants attended on the day). This meeting was facilitated by Geoff Wong, who made contemporaneous notes. At this meeting, we introduced ourselves and then proceeded to explain the purpose of the session. The participants then spent time refining and providing feedback on the documents we provided. We also discussed their ideas about best how to present this information. The session finished with a summary of what they had suggested, and also a way of taking their proposals forward. Based on their suggestions and feedback, Geoff Wong drafted new materials and these were sent round to the participants for comments and feedback.

In brief, after some clarification, the participants felt that it probably does not matter to the person who is being recruited into a realist evaluation what exactly a realist evaluation is. In other words, the detail of what a realist evaluation is or is not was unlikely to matter to the potential participant, so much of the detail in the text of the documents we initially provided was not needed. We were advised the text should be short and kept to half of a side of A4- or one side A5-sized paper. The agenda and notes from the session may be found in Appendix 5. The only new material that the participants felt was needed was a ‘generic’ text that could be used in a patient information leaflet when recruiting to a realist evaluation, and this can be found in Box 2.

Changes to the protocol

Near the start of this project we published our project protocol. 15 During the course of the project we varied the following aspects of our protocol. One of the objectives of our project was to produce resources and training materials for lay participants, and those seeking to involve them, in realist evaluations. We have partially addressed this objective, in that some of the resources and training materials we have produced about aspects of realist evaluations are such that they are accessible to those with no to limited prior knowledge or experience of realist evaluations (see Chapter 3, Developing, delivering and refining training materials for realist evaluation). From our discussions within the project team, with other realist evaluators (e.g. in training workshops) and our project’s Advisory Group, we came to the judgement that these materials would be accessible and helpful to lay participants who are more involved in realist evaluations, for example in their capacity as co-applicants or co-investigators in a project, and would help them understand more about realist evaluations.

However, for individuals who will be recruited into a realist evaluation, we had initially intended to develop draft template information sheets and consent forms that could be adapted for ethics and governance activity. On the issue of consent forms, again from discussion within the project team, other realist evaluators and our project’s Advisory Group, we came to the judgement that there was too much diversity between organisations that grant ethics approval for us to be able to produce a generic template. Different organisations had such diverse processes and requirements for seeking ethics approval that we judged it best for those seeking such approvals to consult and adhere to their organisation’s requirements. As such, we did not produce draft consent forms for realist evaluations. We were, however, able to develop a resource and training material entitled ‘Realist evaluation and ethical considerations’ (see Table 7) that will help to guide realist evaluators when developing information sheets and consent forms for recruiting participants into realist evaluations.

We had planned to deliver three 2-day ‘realist evaluation’ workshops and three 2-day ‘training the trainers’ workshops for a range of audiences. When we approached, or were approached by, those interested, we negotiated with them the logistics and content of each workshop. The preference from those interested was overwhelmingly for shorter workshops, so we ended up providing more workshops, but of a shorter duration, than we had planned. We were unable to find a mutually convenient time before the end of the project to organise any further ‘training the trainer’ workshops beyond the two 1-day workshops we provided to RDS East Midlands in June and October 2016.

Limitations

To develop the briefing materials for our Delphi panels, we undertook a literature review. This review has limitations that are likely to have introduced a number of biases and so, potentially, at least, they limit the inferences that can be made from the included evaluations and methodological pieces. For example, the search process for the review, despite being developed by an expert librarian, was not exhaustive. All the screening for inclusion and exclusion was undertaken by one screener and no quality checks were undertaken. Both processes may mean that we are likely to have missed some evaluations. However, given that the intent was to reach theoretical saturation, and that we retrieved many more evaluations than were necessary to achieve it, this is unlikely to have caused a significant problem to the other stages of the project.

An additional challenge we faced during the literature review was that, at the time of the project, there were no quality standards against which to judge the quality of realist evaluations; it was a function of this project to develop them. This was identified as a need in a range of methodological pieces we analysed as part of the review. 12,13 Therefore, we had to use the project team’s collective judgement, informed by our experience in conducting and teaching realist evaluations, and the literature, to judge the quality of the realist evaluations included in the review. This is an important limitation of our review processes.

Once evaluations had been included, data extraction was undertaken by one researcher, and omissions in data extraction are likely to have occurred. However, all the included evaluations and the data extraction spreadsheet were circulated to all project team members, and so a degree of informal quality checking did occur.

Decision-making on what should be included in the Delphi panel’s briefing materials was undertaken by the entire project team. We are aware that any item or topic included in the briefing materials was included as a result of our subjective interpretations, raising questions about reproducibility. However, the briefing documents we produced were not an end product in themselves, but the starting point for the Delphi panel to build a consensus. In addition, we deliberately asked Delphi panel members to enter into a discussion and suggest items for inclusion in the quality and reporting standards. We also provided the panel members with an end-of-round report and invited them to contact us should they have any concerns about the actions we had taken after we analysed their ratings. As such, we expected that changes would occur as we ran each round of the Delphi process, and thus we are confident that any omissions as a result of the review’s limitations processes are unlikely to have a significant impact on the final reporting and quality standards. We accept that the review of the literature could have been more thorough (e.g. all evaluations analysed and more than one reviewer involved), but we made the judgement that the findings of the review contributed only part (albeit an important part) of the data to inform the Delphi Panel’s briefing document. Other sources of data were the project team’s expertise, that of the Delphi panel itself and data from the RAMESES JISCMail list. We felt that, in order for us to ensure that we delivered as much as possible on all the objectives of this project, the review needed to be truncated and our energies spent elsewhere. To provide transparency on what we have done, we have reported, in detail, all stages of the review itself and the rest of the project.

We recognise that there is much more to cover in terms of the breadth and depth of the training materials we have produced. Because realist evaluation is developing as an approach to evaluation, the ‘wish list’ we were able to elicit from our fellow evaluators who have used this approach was quite long. Given the time and resources allocated for this project, we elected to focus on providing sufficient depth in an accessible manner, rather than breadth on the issues that were the most challenging. With time, we hope to use the community of practice we have developed to address more, and more methodological, challenges.

As experience grows with the use of realist evaluation, it is very likely that many of the resources we have produced will need to be updated. We welcome and invite methodological development in realist evaluations. We expect that what we have produced should be gradually refined and updated as methodological developments take place with increasing use of realist evaluation. Thus, we view the reporting and quality standards and resources and training materials more as a starting point than as definitive resources that must not be altered in any way.

We are aware that realist evaluation is used to evaluate a wide range of topics and by evaluators from a broad range of disciplines and affiliations. The level of expertise of the users of our resources will also vary considerably, from novice to seasoned evaluators. These two aspects mean that some latitude is needed in the use of the resources we have produced. For example, not all the items in the reporting standards will be applicable for all evaluations. Or, when assessing the quality of an evaluation, there may be justifiable reasons for an evaluation to not meet some quality criteria. We have tried to anticipate the varied uses that realist evaluation might be put to by providing a degree of flexibility in our standards. For example, in our reporting standards, if adaptations are made to the evaluation design (as originally described), then evaluators are invited to provide an explanation for any such adaptations.

Finally, we were not able to produce detailed generic templates of draft information and consent forms for participant recruitment into realist evaluations. We have explained why this was the case above (see Chapter 3, Develop, deliver and refine information and resources for patients and other lay participants in realist evaluation, and Chapter 4, Changes to the protocol).

Research recommendations and implications for practice

Realist evaluation, despite having been first introduced in 1997, still has a great deal to do in terms of capacity building and methodological development. This is because it is only in the last few years that its popularity has grown as a form of theory-driven evaluation approach to make sense of complex interventions or programmes. This has created a situation in which some evaluators are using the realist evaluation approach for the first time on projects and some struggle with it.

Thus, capacity building is the priority for realist evaluation as an approach. Dedicated training courses, run by experienced realist evaluators, are needed. We anticipate that developing and running such courses will be easier with the key topic areas and consensus standards identified in this study, although even with such resources, some learners may still struggle to engage with the philosophical basis of the realist evaluation approach. Practical ‘how to’ resources and training materials were limited before, but this project has developed 15 of these to help fill this need. Course developers now have a reference point from which to build their training courses, and learners, a yard-stick against which to judge the quality of their work. The resources and training materials we have developed for this project are designed to be accessible to the novice but also to signpost more advanced learners to further resources. As such, they may be used as part of the basic building blocks of a ‘curriculum’ for realist evaluation courses.

As experience with realist evaluation grows and more evaluations are undertaken, new methodological insights are likely to occur. These need to be captured and analysed to determine if the quality and reporting standards we have produced continue to be fit for purpose or need to be updated. At present, no formal process exists to advance this agenda. Ideally, further funding might enable a project similar to this one – that is, RAMESES III – to address the updating of the standards, although, because much groundwork has already been done, a more truncated project may suffice.

At present, those interested in realist evaluation (and review) might initially need to make small and gradual methodological gains, perhaps by embedding an element of methodological development with in their projects. Disseminating what they have learnt from undertaking their realist evaluations (or reviews) will be a key activity. At present, only ad hoc, informal help and support from more experienced realist researchers given to novices, the sharing of tips and templates used, and debate and discussion of contentious issues takes place. Some of this activity is happening on the RAMESES JISCMail list that we set up as part of the RAMESES Project. There is the potential that the RAMESES JISCMail list could be further developed and supported to serve as an avenue for advancing and disseminating methodological lessons in realist evaluation (and review). For example, it may be one way for realist evaluators to address the issue of generic templates of draft information and consent forms for participant recruitment into realist evaluations, an area that we did not fully address in our project. This might be through the sharing of particularly useful examples of these resources between evaluators and researchers.

Realist evaluators might want to consider learning from the example of organisations like the Cochrane Collaboration, in which motivated researchers have collaborated in a more organised way to systematically and gradually undertake methodological development. At present, many who contribute to, and support, the RAMESES JISCMail do so voluntarily, and with the end of this project all inputs to this list will be on a voluntary basis. Building some sort of future structure that is more sustainable is important. A potential benefit of being more organised is that priorities can be established on which methodological issues in realist evaluation (and review) need more attention, and duplication can be avoided. For example, the resources and training materials we developed are focused on what we were able to identify as the main issues that fellow evaluators found the most challenging to understand and/or execute. There are additional issues that we have not focused on or have only been able to address in passing. Further work is needed to develop resources for these, and other issues, as they arise. The resources and training materials we have designed are also intentionally brief. Developing new resources and building on the ones we developed, by drawing on the methodological lessons learnt from undertaking realist evaluations (or reviews), could potentially be a focus of a better-organised body of realist evaluators and researchers.

Finally, there is a dearth of research to demonstrate that quality and reporting standards necessarily change practice and improve the quality of research. 57,58 This will also be true for the standards we have produced and, therefore, research to demonstrate a change in practice and improvement in the quality of realist evaluations is needed at some point. There is also a counter-theory that such standards may constrain innovation in the development and application of realist methods, and testing this theory could form part of any evaluation of the standards.

Contributions of authors

Geoff Wong (Clinical Research Fellow, Realist Research Methodologist) carried out the literature review, analysed the findings from the review, produced the materials for the Delphi panel, analysed the results of the Delphi panel and developed the patient and lay materials.

Gill Westhorp (Professorial Research Fellow, Evaluator and Realist Research Methodologist), Joanne Greenhalgh (Associate Professor and Realist Research Methodologist), Ana Manzano (Lecturer in Health and Social Policy, Social Research Methodologist), Justin Jagosh (Senior Research Fellow and Realist Research Methodologist) and Trisha Greenhalgh (Professor of Primary Care and Social Scientist) analysed the findings from the review, produced the materials for the Delphi panel and analysed the results of the Delphi panel.

Joanne Greenhalgh assisted in the development of the patient and lay materials.

Gill Westhorp, Joanne Greenhalgh, Ana Manzano and Justin Jagosh developed and internally peer reviewed the resources and training materials.

Trisha Greenhalgh conceived the study and all the authors participated in its design.

All the authors provided realist evaluation support and training to various organisations during this study. All authors read and contributed critically to the contents of this report and approved the final manuscript.

Publication

Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med 2016;14:96.

Data sharing statement

All non-personal data from this project can be obtained from the corresponding author.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.