Developing a reporting guideline to improve meta-ethnography in health research: the eMERGe mixed-methods study

Search strategy

We first conducted comprehensive database searches that were followed by expansive searches to identify published and unpublished research in any language. These searches were iterative and evolved as the review progressed because their purpose was to build our knowledge of recommendations and guidance in conducting and reporting meta-ethnography rather than to answer a tightly defined research question. 43

To identify relevant literature, we started with seminal methodological and technical publications known to our expert academic advisors and the project team including Noblit and Hare’s book,25 detailed worked examples of meta-ethnographies and publications relating to qualitative evidence synthesis more generally (e.g. reporting guidelines for qualitative evidence synthesis approaches and reviews of qualitative syntheses including meta-ethnographies). Relevant texts were included from other disciplines that use meta-ethnography, such as education and social work. We performed citation searching, reference list checking (also known as backward and forward ‘chaining’) of the seminal texts and searched key websites (e.g. The Cochrane Library). Comprehensive database searches were also conducted to identify other methodological publications. 43 Details of databases and other sources that were searched, as well as the search terms that were used, can be found in Appendix 1 (Tables 5 and 6, respectively).

Comprehensive database searches to identify methodological publications

Searches were performed in 16 databases, in July and August 2015. The search strategy was designed in MEDLINE (Medical Literature Analysis and Retrieval System Online, or MEDLARS Online) and refined by testing against a set of key papers already known to the team. The search terms were developed and piloted in collaboration with a researcher highly experienced in the conduct of systematic reviews (RT). The terms related to meta-ethnography and qualitative synthesis, and to methodological guidance, were tailored to each bibliographic database. Reviewers also hand-searched reference lists in included texts (those meeting inclusion criteria for the review) for other relevant studies not already identified. A systematic approach was used to record and manage references, which were stored in the bibliographic software EndNote version X7.4 (Thomson Reuters, CA, USA). The list of included publications from database searches and expansive searches was shared with our academic expert advisors who suggested potential additional publications.

Screening and selection of texts to include

We originally intended to independently double-screen all references by title and abstract; however, we reviewed this decision because the highly sensitive search strategy resulted in a very large number of retrieved references from the comprehensive database searches. We reviewed our screening strategy after independent screening had started; we decided not to double-screen references from the database searches published prior to 2006 to enable us to meet our aims and project timelines. The references published pre 2006 that referred to qualitative evidence synthesis had generally been superseded and the majority of relevant references about meta-ethnography were already known to the project team and expert advisors. We were confident that any relevant publications published prior to 2006 would be identified through expansive searches and expert recommendations. However, as a precaution, titles and abstracts of references from 2005 and older (n = 1204) were electronically searched for key terms (‘ethnograph’, ‘Noblit’) to identify any that referred to meta-ethnography; references containing these terms were then screened by title and abstract by one reviewer (EFF).

Overall, titles and abstracts of 6271 references retrieved through database searches were independently double-screened and a further 1204 were screened by one reviewer. All additional references identified from other sources were double-screened. A total of seven reviewers (IU, EFF, Derek Jones, NR, JN, EASD and MM) were involved in screening retrieved publications, using the inclusion and exclusion criteria presented in Table 1.

Disagreements over inclusion/exclusion were resolved through discussion. A third reviewer also screened publications if the first two reviewers could not reach agreement. A final check of the full text of the articles was conducted for inclusion/exclusion before the data extraction was conducted.

Data extraction

Data were extracted in the qualitative analysis software NVivo 10.0 (QSR International, Warrington, UK), using a coders’ guidance document, which was shared by all coders. The guidance was developed by Emma F France and piloted against five key methodological publications and then discussed with the team for refinement. Four reviewers performed the data extraction (IU, EFF, Karen Semple and Joanne Coyle), working from the same guidance. Data were extracted from each included publication by only one reviewer because this was a qualitative review in which the key principles are transparency and consensus, rather than independence and inter-rater reliability. However, the completeness of the data extraction was double-checked by a second reviewer for 13 of the publications to ensure accuracy. In order to maximise the resources and time available, data were extracted from the richest and seminal publications first, as assessed by Emma F France and Isabelle Uny, and then from the other publications until all were coded and analysed. NVivo 10.0 was used to facilitate management of, and data extraction from, the publications. Guidance and recommendations from the 57 methodological texts were coded into the ‘nodes’ or data extraction categories described below, which are primarily based on Noblit and Hare’s25 seven phases of meta-ethnography conduct, with some additional categories for the data (e.g. ‘Definition or nature of meta-ethnography’) that were not specifically about the seven received meta-ethnography phases. The reason for creating these nodes was their fitness to providing an answer for the research question. The nodes at which data were extracted were:

• definition or nature of meta-ethnography and how it differs from other qualitative evidence synthesis approaches

• selection of a qualitative evidence synthesis approach: how to select a suitable qualitative evidence synthesis approach for one’s aim or research question (this new phase was labelled ‘phase 0’ and added by the eMERGe team).

• Phase 1: getting started – deciding the focus of the review (e.g. guidance or recommendation on choosing a topic).

• Phase 2: deciding what is relevant – which encompassed three subcategories –

  1. quality appraisal of studies – recommendations on ways to appraise the qualities of primary studies to be included.

  2. search strategies for meta-ethnography – recommendations on searching for primary texts or studies.

  3. selection of studies – guidance or recommendations on the manner in which studies to be synthesised were selected.

• Phase 3: reading studies – where advice or recommendation is given on how to read the studies and identify and record the concepts and metaphors contained in each study.

• Phase 4: determining how the studies are related – determining how the concepts and metaphors used in each study relate to others and how they can be synthesised. This phase was also divided into three subcategories –

  1. definition of reciprocal translation – when concepts in one study can incorporate those of another; the coding entailed defining this type of translation and identifying advice and recommendations on how this could be undertaken.

  2. definition of refutational translation – where concepts in different studies contradict one another; the coding entailed defining this type of translation and identifying advice and recommendations on how this could be undertaken.

  3. definition of line of argument (LOA) – when the studies identify different aspects of the topic under study that can be drawn together in a new interpretation; the coding entailed defining this type of translation and identifying advice and recommendations on how this could be undertaken.

• Phase 5: translating studies into one another – the way in which metaphors and/or concepts from each study and their inter-relatedness are compared and translated into each other.

• Phase 6: synthesising translations – how to synthesise the translations to make them into a whole that is greater than its parts.

• Phase 7: expressing the synthesis – how the synthesis is presented, the message conveyed and for which audience.

Some other categories were also included in data extraction, which are reported on in this document:

• issues of context in meta-ethnography

• number of reviewers required to undertake a meta-ethnography

• validity, credibility and transferability issues in meta-ethnography.

Data analysis

Publications were read repeatedly and compared using processes of constant comparison. Extracted data were analysed qualitatively mainly by two members of the team (IU and EFF). To support the analysis, memos were written for each category in which each reviewer could record their analysis of the data extracted at the particular node. As the analysis progressed, areas of agreement and uncertainties were noted in the memos and Isabelle Uny and Emma F France drew on each other’s understanding of the data from each node. For complex nodes (e.g. regarding conduct of phases 4, 5 or 6), each reviewer individually identified the key themes and issues in a NVivo memo and then the two coders compared what they had written to check their different interpretations. Following this, one of the coders wrote a detailed analytic memo, to which the other subsequently added more details, or which they could question in light of what they had read. For less complex or less contentious nodes (e.g. regarding conduct of phases 1, 2 or 3), one reviewer conducted the analysis, also using memos that were then checked by the other reviewer for accuracy and transparency. Throughout, each reviewer maintained an analysis journal in NVivo and any analysis decisions made at project meetings or internal meetings were logged in a folder on our shared electronic drive (all meetings were also audio-recorded for easy reference). Once completed, the initial analysis was collated and shared with the wider team and then discussed and revised to add rigour to the process and gain further perspectives on wider interpretation and analysis of the data contained in each node.

The guidance and advice provided in the included publications around each node/category varied in richness and detail. Nonetheless, the full range of practice was documented, regardless of the richness of the text. However, each reviewer also noted whether or not they felt that the texts they extracted data from were ‘rich in details’ (i.e. whether or not they were a detailed account related to meta-ethnography with in-depth explanation and rationales that went beyond description). As the analysis drew to its final stage, we detailed definitions for each of the phases as understood and described in the included publication. We summarised and analysed advice and recommendations given on the conduct of each and every phase, and noted the pitfalls and criticisms in the conduct and reporting of meta-ethnography raised by each author. The findings that emerged from the analysis are, thus, presented below.

Definition or nature of meta-ethnography and how it differs from other qualitative evidence synthesis approaches

The analysis of the methodological texts determined that meta-ethnography is an interpretive method of synthesis rather than simply an aggregative one. It was described by Noblit and Hare25 as: ‘the comparative textual analysis of published field [qualitative] studies’, the aim of which is to create new interpretations.

A meta-ethnography analyses qualitative data in an inductive way to develop concepts, theories and models. Meta-ethnography attempts to preserve the contexts of the studies synthesised and uses a process of translation, which will be described at length in Phase 5: translating studies into one another.

Although it was conceived solely as a method of synthesis by Noblit and Hare,25 other authors have, over time, also used meta-ethnography as a systematic literature review methodology. 6 Moreover, although meta-ethnography was designed to synthesise interpretive qualitative studies, one text44 in this review argued that meta-ethnography could be used to synthesise qualitative and quantitative studies together; however, in order to do so, those authors drew on meta-ethnography to develop a new approach called ‘critical interpretive synthesis’.

There was some discussion within the methodological texts regarding what constitutes the ‘unique’ characteristics of meta-ethnography as a qualitative evidence synthesis method. According to Noblit and Hare,25 the processes that they presented in seven phases were not necessarily unique to meta-ethnography. However, they argued that the underpinning use of Turner’s theory of social explanation45 embedded in the process of translation differentiated meta-ethnography from other qualitative evidence synthesis methods. After a meeting with the eMERGe team in June 2016, Professor George Noblit provided further reflections on the process of translation as follows:

In Noblit and Hare’s text, synthesis is seen as a form of translation of accounts into one another. The nature of such translation is based on S. Turner’s Sociological Explanation as Translation (Cambridge: Cambridge University Press, 1980) in which he examines comparative explanations—the essential form of meta-ethnography. Turner notes that practices, and the concepts used to describe such practices, may vary from those in another society. In doing comparisons, then one may use the concept from one society, or create a new concept, in making the comparisons of the societies. In this, explanation is a form of translation and that ‘an adequate translation would yield us claims that had the same implications in both languages’ (p. 53). Accounts can be substituted for language in this quote, for the purposes of meta-ethnography. Synthesis as translation starts with a puzzle that is of the form where one study says x, what is another study saying? Addressing this puzzle requires formulating an analogy between the studies. As we add studies, we may find that the translation/analogy offered with the initial studies does not hold up.

Professor George W. Noblit, University of North Carolina at Chapel Hill, 2016, personal communication

The analysis of the methodological texts in this review indeed showed that what is seen to distinguish meta-ethnography from other qualitative evidence synthesis methodologies is the translation process.

One of the other key characteristics of meta-ethnography, as seen by some authors, is that it aims to arrive at new interpretations greater than those of individual studies. 6,25

The main uncertainties surrounding the nature of meta-ethnography are around whether or not it could be used to synthesise both qualitative and quantitative studies, and whether or not purely descriptive studies (of which there tend to be many in health research) or deductive studies should be excluded from the synthesis.

‘Phase 0’: selecting a qualitative evidence synthesis approach

This review identified a new stage before ‘phase 1: getting started’, which was labelled ‘phase 0: selecting a qualitative evidence synthesis approach’. It relates to the rationale for choosing meta-ethnography as the qualitative evidence synthesis approach for the topic in question. This review demonstrated that better guidance is needed here to avoid reviewers choosing the wrong method of qualitative evidence synthesis or having to amend a method to suit their needs when a more suitable one might already exist.

Through stages 1 and 2 of this project, it became clear to the eMERGe team that authors of meta-ethnographies often cite Noblit and Hare25 and state that their method is meta-ethnography when, in fact, they are not conducting a meta-ethnography. A number of strategies to avoid this were identified in the review of methodological texts, including:

• investigating other qualitative evidence synthesis approaches before choosing meta-ethnography (e.g. ensuring that an interpretive qualitative evidence synthesis is required and that the type and quantity of studies to be synthesised fit with the method selected)

• ensuring that the synthesis research question and aim drive the choice of qualitative evidence synthesis approach (e.g. whether it aims to generate a model or theories of behaviour or experiences, or aims at conceptual and theoretical development)

• making sure that the qualitative evidence synthesis chosen fits with the epistemological stance of the team of reviewers/the reviewer, their skills and experience of the methods used (meta-ethnography may not be best suited to novices in qualitative research)

• ensuring that the time and resources available fit with the conduct of a meta-ethnography.

Ultimately, the review revealed that meta-ethnography needs a high level of qualitative expertise. It is a time-consuming enterprise and this must also be taken into account in phase 0. Clear guidance is required on the conduct of meta-ethnography (particularly phases 5 and 6) to help researchers choose the most suitable qualitative evidence synthesis approach.

Phase 1: getting started

Noblit and Hare25 describe this phase as:

identifying an intellectual interest that qualitative research might inform . . . In this phase, the investigator is asking, How can I inform my intellectual interest by examining some set of studies?

pp. 26–725

Ideally, a meta-ethnography aims to address a gap in knowledge, for instance by asking whether or not a qualitative evidence synthesis has previously been conducted on a particular topic or by asking whether or not it can offer new explanations of the topic. The methodological review found that authors recommended that an aim and research objectives be defined, at least in broad terms, at the start of undertaking the meta-ethnography, even if they are refined later in the process. An example of how this may be reported can be found in a worked example by Britten et al. 46 about lay experiences of diabetes mellitus and diabetes mellitus care.

Although the issues identified regarding phase 1 were not contentious, there were some uncertainties around the best way to define, or refine, the research question in a meta-ethnography as there is a link between the research question and the selection of studies to be included in the thesis (e.g. the final research question will determine which studies are included). 28,47

Phase 2: deciding what is relevant

With regard to the meta-ethnographic conduct, it bears remembering that Noblit and Hare’s25 book was published at a time when online bibliographic databases were unavailable. They created meta-ethnography as a method of synthesis but did not provide detailed guidance on selecting studies for inclusion in the synthesis. Subsequently, other researchers have applied systematic search and selection procedures to the identification and selection of studies for inclusion in a meta-ethnography.

The analysis of the methodological texts confirmed the view that published meta-ethnographies mostly use comprehensive systematic review style searches traditionally used for quantitative reviews of intervention effectiveness. However, some authors in this review stressed that this was not the only method available for meta-ethnography. For instance, some suggested that exhaustive searching may be suitable for making generalisable claims or to provide a comprehensive picture of research on a topic, whereas non-linear or purposeful searching might be more appropriate in other cases, such as meta-ethnography, where the intention is to generate a theory. Whatever the case may be, authors stressed that the search strategy ought to match the intended purpose of the meta-ethnography. One of the difficulties raised by authors, however, is that qualitative research reports are sometimes challenging to identify through electronic database searches. They therefore urged reviewers to supplement database searches with alternative methods, such as searching grey literature.

Phase 3: reading the studies

The review identified various reading strategies for phase 3, such as reading while recording themes and identifying concepts (including refutational ones) and their context within the framework of the research question. Some read the papers or accounts chronologically and some started with the most conceptually rich, although there was no evidence to indicate how reading papers in different orders may affect the synthesis output. Authors of methodological texts often used tables (or mind maps) to display concepts, sometimes distinguishing concepts of the research participants from those of the primary study authors (referred to by some authors as first- and second-order concepts, respectively). Some also used phase 3 to appraise the quality of the studies. Some authors further specified the importance of reading being carried out by two or more reviewers. This review concludes, however, that one of the key uncertainties in this phase was around how to preserve the meaning of, and relationships between, concepts within and across studies when reading.

Phase 4: determining how the studies relate

Noblit and Hare25 expressed that the studies may relate in three main ways:

1. reciprocally (because they are about similar things)

2. refutationally (because they are about different things)

3. as a LOA (because they offer part of a higher meaning).

Some methodological texts in this review ventured that a well-conducted phase 3 will help to determine how studies relate to one another; however, most authors show how they related the studies in a grid or table (some detailed descriptions of how this has been reported can be found49–51). Some texts analysed in this review suggested that ‘relating studies’ is best done collaboratively by a team who interpret the concepts separately first and then come to a decision together.

We concluded that the main uncertainties about the conduct of phase 4 are:

• whether or not it is possible to relate studies that are profoundly epistemologically different, and what is the best way to preserve the semantics and context of the metaphors or concepts contained in each study through the ‘relating’ process

• how the order in which studies are appraised and synthesised may affect the outcome of the synthesis (e.g. use of index paper)

• whether or not reciprocal findings in studies may tend to be given more weight than refutational ones.

Phase 5: translating studies into one another

As expressed earlier, the process of translation is key to meta-ethnography conduct. It was defined by Noblit and Hare25 as idiomatic rather than literal. From this methodological review, we can conclude that the process of translation is not a linear process but an iterative one, which aims to translate concepts from one study into another study and, thus, arrive at concepts or metaphors that embody more than one study. 52

This review found that there is no single way of conducting the translation in a meta-ethnography and the various methods have not been formally compared in methodological research. However, it is the eMERGe team’s contention that whichever method of translation is used should be made explicit and transparent by the authors. The diagrams in Figure 4 were designed by the eMERGe team to represent three well-defined methods of translation described by some of the authors of the methodological texts included in this review.

FIGURE 4.

Three possible methods for the conduct of phase 5 as interpreted by the eMERGe project.

Reciprocal translation

Reciprocal translation, according to Noblit and Hare25 and to a number of other authors reviewed in this study, takes place when studies are roughly about the same thing and their meaning can be interpreted into one another. The conduct of reciprocal translation was described in detail by Campbell et al. ,24,52 and their approach has been used by other authors in this review. We have summarised the approach of Campbell et al. 24,52 in Figure 5.

FIGURE 5.

Reciprocal translation process as interpreted by the eMERGe project.

A number of authors in the review recommended using tables or grids to represent the reciprocal translation analysis (a particularly detailed example of this can be seen in Malpass et al. 49).

This review found that one of the issues regarding the conduct of reciprocal translation is that it can be done in such a way as to result in a simple recoding or recategorising of themes from the primary studies rather than being interpretive. A meta-ethnography should strive to lead to new interpretations and theories of the topic under study.

Phase 6: synthesising translations

Noblit and Hare25 define this phase as follows:

Synthesis refers to making a whole into something more than the parts alone imply. The translations as a set are one level of meta-ethnographic synthesis. However, when the number of studies is large and the resultant translations numerous, the various translations can be compared with one another to determine if there are types of translations or if some metaphors and/or concepts are able to encompass those of other accounts. In these cases, a second level of synthesis is possible, analyzing types of competing interpretations and translating them into each other.

p. 2925

The manner in which the translation is synthesised depends mainly on the way phase 5 was conducted. Some authors express that, to a certain extent, translation and synthesis happen together, in an iterative manner. There is also no single way in which to carry out the synthesis. Some of the methods used by authors of worked examples of meta-ethnographies are described in two diagrams in Figure 6.

FIGURE 6.

Two possible methods for the conduct of phase 6 as interpreted by the eMERGe project.

The review indicates that there is potential to develop a theory from the synthesis in phase 6, but that very few authors describe whether or not, or how, they did this. One notable exception is Britten and Pope,51 whose worked example produced middle-range theories in the form of hypotheses that could be tested by other researchers. Part of the issue is that theory is understood differently by different authors.

Phase 7: expressing synthesis

Noblit and Hare25 expressed in their book that the meta-ethnography synthesis output must be intelligible to the audience at whom it is aimed. Because of this, it could take the form of a written statement, but also could be conveyed by video or other art forms, although this has been rare. However, Noblit and Hare state:

The intention here is not to pander to the audience. Having our syntheses readily intelligible does not mean reducing the lessons of ethnographic research to an everyday or naive understanding of a culture. The focus on translations is for the purpose of enabling an audience to stretch and see the phenomena in terms of others’ interpretations and perspectives.

p. 2925

Authors in this review noted that the expression of the synthesis has to suit the audience and be clear so that policy makers and intervention planners can make use of it. However, some warned about the difficulties in remaining independent in the expression of the synthesis; Booth57 for instance expressed concerns that pressures from funders to publish new findings may influence the final product.

Issues of context in meta-ethnography

The context in a meta-ethnography concerns not only the characteristics of primary studies (e.g. the socioeconomic status of participants, their location, study setting, research designs, methodological details, and political and historical contexts), but also the context of the reviewers themselves (funding, political climate, respective expertise and world views).

The review found that authors deemed context to be important to meta-ethnography. Indeed, from their initial work on meta-ethnography, which was designed specifically to preserve the contextual aspects of studies to be synthesised, Noblit and Hare25 contended that other aggregative qualitative evidence syntheses, by contrast, were:

context-stripping [and] impeded explanation and thus negated a true interpretive synthesis.

p. 2325

For the authors in this review, taking into account the context of the studies to be synthesised was seen to bring credibility to a meta-ethnography. Authors in this review recommended laying out the context from each primary study in a grid or table for readers to see. Unfortunately, context is often a problem for meta-ethnography as it tends to be poorly reported in primary studies in health-care research. The uncertainty with regard to the issue of context is how to synthesise large numbers of studies with different contexts.

Number of reviewers required to undertake a meta-ethnography

The review showed that authors believed that there are benefits in a meta-ethnography, as with qualitative research in general, being undertaken by more than one reviewer. The reasons given were that it:

• aids the translation process

• leads to richer and more nuanced interpretations, as reviewers have alternative viewpoints and perspectives

• encourages explorations of dissonance

• brings more rigour to the process and increases the credibility of the research process.

Although an optimum number of reviewers for a meta-ethnography cannot be stated here as it has not been the subject of empirical research comparisons, the review certainly expressed that there were weaknesses in undertaking a meta-ethnography with only one reviewer, for example a lack of exploration of alternative interpretations. A review of meta-ethnographies published between 2012 and 201329 showed that the actual number of reviewers in recently published meta-ethnographies varied from one to seven, with the most common number being two or three. The composition and experience of the team of reviewers were seen as important. Findings suggest that the team must fit the aim of the synthesis and represent a range of perspectives, genders and skills (e.g. translators, data retrievers, user representatives and reviewers from different disciplines). Some authors suggested that qualitative evidence synthesis expertise is needed in the team to undertake a meta-ethnography. Other authors addressed the issue of power dynamics within the team of reviewers (e.g. different levels of seniority).

Validity, credibility and transferability issues in meta-ethnography

Within this review, the debate around validity or credibility and around generalisability or transferability in meta-ethnographies revolved around how useful or credible the findings from this type of synthesis are, as well as on whether or not they can be generalised or transferred to other settings.

Criticisms of meta-ethnography

Some of the main criticisms of meta-ethnography identified in the review of methodological papers included (1) the fact that, although there was some good practice in the conduct of meta-ethnography, a lot of those reviews that are labelled as such are actually critical literature reviews rather than interpretive meta-ethnographies; (2) the large number of studies selected for inclusion in some meta-ethnographies; and (3) the fact that some reviewers of meta-ethnography have used aggregative approaches in the attempt to conduct meta-ethnographies, and, in others, it is unclear what process was used to arrive at the final synthesis. Another criticism included that meta-ethnography reviewers sometimes failed to make clear how they selected their studies, whereas others offer incomplete analyses, in which first-, second- and third-order constructs are not always distinguished. Another main critique was that few meta-ethnographies actually conduct any refutational translations and few offer proper LOA syntheses, instead conducting only reciprocal translations.

Aim

To identify and develop good practice principles and standards in the reporting of meta-ethnographies on which to inform the draft reporting standards for consideration by the expert and stakeholder Delphi groups in stage 3.

Research questions

• What good practice principles in meta-ethnography reporting can we identify to inform the draft reporting standards for consideration by the expert and stakeholder Delphi groups in stage 3?

• From these good practice principles, what standards in meta-ethnography conduct and reporting can we develop to inform recommendations and guidance?

To address these questions, stage 2 consisted of two sequential stages:

1. stage 2.1 – documentary and interview analysis of seminal and poorly reported meta-ethnographies

2. stage 2.2 – audit of recent peer-reviewed, health- or social care-related meta-ethnographies to identify if, and how, they meet the standards and to further inform and develop the guidance and reporting criteria.

Methods

We aimed to analyse and review 10–15 poorly reported and 10–15 seminal meta-ethnographies. We asked expert academics from the Project Advisory Group to recommend meta-ethnography journal articles that they judged to be seminal and those that they considered to be relatively poorly reported and to explain why. 43 In addition, published reviews of meta-ethnography quality were searched by the project team (RJR and EFF) to identify low-quality examples.

Journal articles were considered for inclusion if they were:

• a peer-reviewed meta-ethnography journal article

• published following Noblit and Hare’s 1988 book25

• considered by our expert advisors and/or published reviews of meta-ethnographies to be either

  • seminal (i.e. to have influenced or significantly advanced thinking and/or to be of central importance in the field of meta-ethnography), or

  • relatively poorly reported.

Thirteen seminal and three poorly reported meta-ethnographies were suggested by experts. Because few poorly reported meta-ethnographies were identified, we searched three published reviews of qualitative syntheses. 17,18,29 This identified a further 13 papers as relatively poorly reported meta-ethnographies. In total, 29 meta-ethnographies were analysed: 13 considered to be seminal and 16 regarded as relatively poorly reported (see Appendix 4).

Findings

The following similarities and differences between poorly reported and seminal meta-ethnographies were identified.

Introduction

Ultimately, meta-ethnographies are a form of qualitative evidence synthesis that can be used to inform policy and practice. We therefore wanted to include the views of meta-ethnography end-users on the utility of published meta-ethnographies, to identify issues of reporting important to them. This was based on the assumption that the reporting needs and priorities of end-users may differ to those using meta-ethnographies within an academic capacity.

Methods

Findings

A summary of participants’ perceptions are presented below and the differences between how professional end-users and academics approach, judge and use meta-ethnography articles are highlighted.

Discussion of stages 2.1a and 2.1b

Poor reporting can be a barrier to end-users’ trust in, and subsequent use of, meta-ethnography findings to inform policy and practice. We identified aspects of reporting that are important to end-users and that may differ to those using meta-ethnography for academic purposes, in order to ensure that these aspects were considered during the reporting guidance development. The documentary analysis of 29 seminal or poorly reported meta-ethnographies, together with the interviews of potential end-users of meta-ethnographies enabled us to identify good practice principles and contributed towards our development of standards in the reporting of meta-ethnographies.

Both the data analysis of the seminal and poorly reported meta-ethnographies and the interviews with professional end-users suggested that it was difficult to identify clear boundaries between the reporting of meta-ethnography phases. This influenced the development of the guidance and reporting standards by further reinforcing the project team’s understanding that some reporting standards could potentially sit within more than one phase of a meta-ethnography.

Analysis of the seminal and poorly reported meta-ethnographies showed that, on the whole, the seminal meta-ethnographies provided far more detail on their conduct within these phases than the poorly reported meta-ethnographies. The authors of the seminal meta-ethnographies provided more of a step-by-step description of the processes followed by the review team and the reasoning behind them. A variety of processes were adopted during analysis and synthesis, with the commonality being clear, detailed description, typically with illustrative examples from the review being discussed.

Not all of the interview participants’ reporting requirements were met by all of the meta-ethnographies. Participants stressed the need for clear, comprehensive information on a meta-ethnography’s scope, findings and implications for policy and/or practice. This was required to enable end-users to quickly judge the article’s relevance to their needs. For participants, a meta-ethnography’s relevance was not limited to potential matches between the end-user’s population of interest (e.g. a particular patient group) and the samples within the primary studies reviewed. In order to be considered relevant, end-users also wanted credible findings and implications that they could use in practice. Because it is a meta-ethnography’s findings and the implications of these that are of key interest to professional end-users, if the primary focus of the article is methodological development or theoretical debate (as was the case for some of the seminal meta-ethnographies), it is typically of less interest to them. Formats such as executive summaries or policy briefings regarding the meta-ethnography findings could meet some of their unmet needs more effectively than academic journal articles.

The findings of the studies presented in 2.1a and 2.1b are limited by the subjective judgement of papers perceived to be seminal or poorly reported meta-ethnographies by the Project Advisory Group or the authors of the reviews they were drawn from, and the relatively small number of professional end-users (n = 14) interviewed. Nevertheless, their findings provided additional perspectives and insight that guided the development of draft reporting standards.

Audit methods

Nicola Ring and Emma F France led the development of the provisional audit standards but all team members were involved in refining these. Ruth L Turley conducted the searches for the sample meta-ethnographies. Isabelle Uny, Nicola Ring and Emma F France screened potential studies for inclusion in the audit. Emma F France, Nicola Ring, Rachel J Roberts, Jane Noyes, Margaret Maxwell and Ruth L Turley audited sampled meta-ethnographies against the provisional standards. The audit was conducted in April 2016.

Identification of sample of meta-ethnographies for audit

A systematic search was carried out in six electronic databases in October 2015. The search was comprehensive to identify all possible meta-ethnographies that could be included in the audit. The databases searched and the search terms used in stage 2.2 are included in Appendix 6. In order to capture published meta-ethnographies from a broad range of disciplines and journals for the audit, the team added broad multidisciplinary databases (SCOPUS and Web of Science) as a replacement for discipline-specific databases such as ASSIA (Applied Social Sciences Index and Abstracts), PsycARTICLES and PubMed as these had already been searched in stage 1. The search was carried out by a researcher highly experienced in the conduct of systematic reviews (RLT). Those records found in the databases search were then merged with records identified in a search for meta-ethnographies in the Cochrane register of qualitative evidence syntheses. Removing duplicates resulted in a total of 571 items (see Figure 7). These items were then screened by titles and abstracts (IU) to exclude all meta-ethnographies that did not meet the following inclusion/exclusion criteria:

• Inclusion criteria –

  • title, abstract and/or keywords made reference to meta-ethnography or meta-ethnographic techniques or methods of Noblit and Hare25

  • report of a synthesis of primary qualitative research studies

  • had a health- or social care-related focus (e.g. patients’ experiences of a health condition or health service; health professionals’ experiences of delivering care; personal experience of health promotion initiatives or public health issues)

  • published between 1994 and 2015 in English, French or Spanish. 43

• Exclusion criteria –

  • title, abstract and/or keywords made no reference to meta-ethnography or meta-ethnographic techniques or to methods of Noblit and Hare25

  • not a qualitative evidence synthesis, or, if a qualitative evidence synthesis, not conducted using approaches other than meta-ethnography

  • did not have a health or social care focus (e.g. school education)

  • meta-ethnographies reported in languages that could not be translated by the team

  • meta-ethnographies first-authored by members of the Project Advisory Group and worked examples included in stage 1 or stage 2.1. 43

Initial screening reduced the number of possible meta-ethnography studies for inclusion in the audit to 243. From this pool of possible studies, a purposive sample with a wide range of published meta-ethnographies was selected (EFF, IU, NR) using the following inclusion criteria:

• published in a range of different journals (e.g. medical, nursing, midwifery, allied health professional, social care or social science) and at least one meta-ethnography in report rather than journal article format

• conducted by reviewers in different disciplinary backgrounds (e.g. medicine, nursing, midwifery, sociology, psychology, allied health professions, social work), in different countries and from different philosophical traditions

• conducted by single and multiple (team) reviewers

• national and international focus of primary studies (e.g. included studies from different countries)

• included different types of qualitative data

• standalone meta-ethnography study and meta-ethnography conducted alongside a quantitative systematic review

• examples represented a range in number of included studies (e.g. < 10, > 50)

• reviewers reported using ‘normal’ versus ‘adapted’ or ‘modified’ meta-ethnography methods.

Meta-ethnographies for inclusion in the purposive audit sample were screened initially based on title and abstract. The goal of purposive sampling and selection was to ensure that the final audit sample was diverse with a wide range of included meta-ethnographies but it was a time-consuming process. As sampling progressed, some meta-ethnographies were excluded because the sample contained too many on the same health topic or from the same type of journal or country. Selected meta-ethnographies were chosen because they offered a different perspective from other sampled meta-ethnographies, such as being from a different continent or conducted by a single reviewer or from a different discipline. The sampling process highlighted a significant increase in recently published meta-ethnographies, so the decision was taken to exclude older articles (pre 2005) because they did not reflect contemporary developments in the field. An initial purposive sample of 49 meta-ethnographies was sent to the full project team for their review prior to the final selection of 40 eligible studies being made. However, when full texts of these 40 papers were read, 21 of these publications were reported in formats that were not recognisably meta-ethnography (e.g. they combined qualitative and quantitative data or appeared to be literature reviews). Project time scales meant that authors could not be contacted for additional information and clarification so, after team discussion, these publications were removed from the audit sample. This resulted in a final audit sample of 19 (Figure 7).

FIGURE 7.

The PRISMA flow diagram for stage 2.2. Adapted from Moher et al. 32 This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Adapted from Moher et al. 32 2009. This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Development of provisional standards and audit tool

The development of provisional standards was a lengthy and iterative process. Initially, a Microsoft Word 2010 (Microsoft Corporation, Redmond, WA, USA) version 14 template was created based on the seven meta-ethnography phases identified by Noblit and Hare25 and the new phase 0 (choosing meta-ethnography) identified during stage 1. Every item of advice and recommended practice reported in stage 1 and stages 2.1a and 2.1b for these phases was transposed into the template and converted into draft standard(s). A bespoke audit tool was then created around these draft standards (Table 2). The audit standards needed to be measurable. Rather than reporting whether standards were met as either a ‘yes’ or ‘no’, the standards measured whether criteria were met in full, in part or not at all. Some standards (e.g. those relating to comprehensive literature searching methods and quality appraisal) did not apply to every study, so the audit tool was designed to differentiate between standards that were not applicable compared with those that were not met.

TABLE 2 - Excerpt from version 1 of the draft standards and audit reporting tool

N/A, not applicable.

Advice/recommendations	Standard(s)				Evidence source(s)
Phase 1: getting started with meta-ethnography
Reviewers need to understand the topic/subject area well enough to know that there is qualitative data potentially suitable for inclusion in a meta-ethnography	Meta-ethnography reports in their introduction or literature review should: provide information on the availability of qualitative data at the outset of the study that potentially could be synthesised				Stage 1
Audit tool (version 1)
Standard number	Phase 1: getting started with meta-ethnography Meta-ethnography reports should include –	Yes – in full	Yes – in part	No	N/A	Comment
1/1	Information (e.g. in a literature review) on the availability of qualitative data that potentially could be synthesised

Version 1 of the audit tool was then refined (e.g. similarly worded standards appearing in the same meta-ethnography phase were merged to avoid repetition), creating version 2 of the audit tool, which was then piloted. Nicola Ring and Emma F France intended to independently pilot the tool on two meta-ethnographies randomly selected from the audit sample60,61 and then compare/discuss their findings prior to revising the tool pre audit. Applying the audit tool was more complex than anticipated because of the large number of standards (initially there were 138), inconsistencies in reporting and the nature of meta-ethnography, so Emma F France and Nicola Ring jointly applied the audit tool initially to both papers and then to a third paper (Montforte-Royo et al. ,62 also from the audit sample). The pilot enabled Emma F France and Nicola Ring to revise the draft standards (e.g. removing ambiguous language and duplicate standards). Version 3 of the audit reporting tool contained 109 provisional standards, of which 86 applied to all papers. To facilitate collation of results and data analysis, version 3 was converted into Microsoft Excel^® 2010 (Microsoft Corporation, Redmond, WA, USA) version 14 format (version 4); an excerpt is shown in Figure 8, and Tables 8–15 (see Appendix 7) contain details of all standards. For each standard, a result of fully met, partially met, not met or not applicable was recorded (using a ‘1’); these four possible outcomes were for audit use only. Standards that may not apply to every meta-ethnography were shaded grey for easy identification. There were cells/space for auditors to make comments on individual standards as well as overall comments. Use of the audit tool was explained to auditors during a team meeting and written guidance was provided. Auditors were randomly assigned a selection of meta-ethnographies for auditing. Once each paper was audited, results were checked by a second member of the audit team. Any disagreements were referred to Nicola Ring or Emma F France for a final decision. Post audit, the project team met to discuss the audit process and content of provisional standards.

FIGURE 8.

Excerpt of the Microsoft Excel audit tool (version 4).

Audit findings

This was a retrospective audit. Sampled meta-ethnography publications were audited against provisional reporting standards that did not exist when these papers were written. Audit findings simply indicate how these previously published meta-ethnographies were reported compared with the provisional reporting standards. Audit findings should not be taken as comment about the quality or robustness of individual meta-ethnography studies. Meta-ethnography reporting is known to be problematic. 29 It is not intended that individual audited publications are singled out as examples of poor reporting when meta-ethnography reporting generally is suboptimal. Consequently, this report presents an overview of the main audit findings (including reporting strengths and weaknesses) and, where possible, we have avoided referring to individual publications. Findings are, in the main, presented for the 86 standards that applied to each meta-ethnography.

Nineteen publications from 18 studies were included in the audit (Table 3). These meta-ethnographies were published in 16 journals and one report by authors from nine countries. Two papers were single authored. One study appeared as a journal paper and a report. Two studies conducted the meta-ethnography alongside a quantitative systematic review on the same topic. 63,64 The number of included studies in these meta-ethnographies ranged from 4 to 51 (see Table 3).

Seventeen of the 19 audited meta-ethnographies fully or partially met ≥ 50% of the applicable audit standards but there were considerable variations in reporting; for example, the applicable standards met in full for each publication ranged from 43% to 88% (mean 63%) (see Table 3). Mostly, the audit standards were met in part rather than in full: only six publications (32%) were considered to meet fully ≥ 50% of the applicable standards. The publication meeting most standards in full or in part (88%) was, however, 300 pages long (Galdas et al. 63). Overall, the 19 meta-ethnography publications all had some reporting strengths, but there were other elements of their reporting that were less good. So, a publication may have met a relatively small number of standards in full but, overall, be well reported because lots of other standards were met in part. This is best illustrated by the meta-ethnography study that was audited in two publication formats: the journal paper (Wells et al. 201365) met more standards overall (in full and in part) but the research report (Wells et al. 201164) met more standards in full (see Table 3).

The percentage of applicable audit standards met in each meta-ethnography phase varied considerably. Only two meta-ethnography phases had ≥ 70% of their applicable standards met (see Table 3). On average, the three best-reported meta-ethnography phases were phase 7 (expressing the synthesis), phase 6 (synthesising translations) and phase 1 (getting started), with a mean of 76%, 70% and 68% of standards fully or partially met, respectively (Table 4). Overall, the three least well reported phases were phase 0 (selecting synthesis approach), phase 3 (reading the studies) and phase 5 (translating studies), with 50%, 38% and 51% of standards met to some extent, respectively (see Table 4). The number of audit standards met varied between individual publications. For example, in phase 4 the number of applicable standards met by each publication ranged from two to five, with no study meeting all applicable standards.

Phase 0 (selecting meta-ethnography) contained seven standards applicable to every publication. Across the 19 publications, 50% of these standards were fully or partially met, with an average score of 3.4 standards met to some extent. Overall, the best-reported standard related to the type of social explanation that reviewers wanted to produce: 17 audited publications met this standard in full or in part. Although reviewers were good at reporting their review context and specifying why meta-ethnography was selected as the most appropriate qualitative evidence synthesis approach (see Appendix 7, Table 8), in most cases reporting lacked depth. For example, reviewers stating that meta-ethnography was chosen because it had been ‘used with good effect in health research’,73 rather than providing a fuller rationale as to why meta-ethnography was considered to be the most appropriate qualitative evidence synthesis approach (e.g. ‘it emphasises concept’). 63 The poorest-reported standards related to reviewers stating the type of social explanation they expected to produce in line with Turner’s theory,45 their qualitative expertise and their interpretive perspectives, such as their epistemological position.

Phase 1 (getting started) contained five standards applicable to every publication. Overall, this phase was relatively well reported across the 19 publications, with 68% of these standards being fully or partially met and an average score of 3.3 standards met to some extent. Consequently, phase 1 was one of the best-reported phases in the audit. Reviewers were good at reporting the knowledge gap to be filled by meta-ethnography (n = 18) and their review aims (n = 19). However, reporting of review questions or objectives needed to be improved as most publications (n = 12) did not explicitly specify these, which meant that, at times, auditors were unable to determine whether or not these were congruent with meta-ethnography, for instance, whether researchers planned to produce a new theoretical model (which is appropriate to meta-ethnography) or integrate qualitative and quantitative findings (which is more suited to another qualitative evidence synthesis method).

Phase 2 (deciding what is relevant) contained 17 standards applicable to every publication, making this the second largest audit section. More than half (61%) of the phase 2-applicable standards were fully or partially met, with an average score of 9.9 standards met across the 19 publications. This phase included standards relating to how reviewers identified studies for synthesis, such as which databases they searched, the search terms used and the study inclusion criteria, as well as the outcome of the literature searching. The 19 meta-ethnographies were from 18 studies; 10 (56%) of these studies used PRISMA32 (or equivalent) reporting and four (17%) used ENTREQ34 (see Table 3). Overall, meta-ethnography reporting in phase 2 was enhanced through the use of such guidelines, and several standards (numbers 19, 20, 31 and 33) were notable because all meta-ethnographies met these (although not always in full). Areas in phase 2 in which reporting could be improved included identifying which reviewers were involved in the literature searching, stating whether or not reviewers initially worked independently during the search process and by providing more information about search strategies and decisions, such as the years the data search covered. In addition, although all meta-ethnographies provided details on their study inclusion and exclusion criteria, more depth of information was needed.

Overall, phase 3 (reading of studies) was not well reported, with only 38% of the 13 applicable standards being met to some extent (average score of 4.8 standards fully or partially met). There was, for example, lack of clarity about who read the included papers and how data were extracted, including what level of constructs were extracted from the original studies, who extracted data, whether researchers extracted all relevant information from original studies or selected material only and whether or not extracted data were checked for accuracy. A total of 95% of the publications (n = 18) were considered to have partially or fully met the standard pertaining to the reporting of original studies’ contextual information. This high level of reporting was achieved, with one exception, because audited publications provided tabular summaries of included studies (e.g. country of research, number/type of participants and research methods) and a short narrative description. However, across these 18 publications, wider contextual information was not well provided. For instance, the gender and ethnicity of original study participants was poorly reported, although there were some exceptions to this. As Monforte-Royo et al. 62 noted, such information is not always formally described in the original studies. The omission of such details in original studies needs to be explicitly acknowledged in meta-ethnography reports because this contextual information can influence original study findings and, therefore, meta-ethnographic interpretation. Overall, the audited meta-ethnography reports did not explicitly state whether or not reviewers looked for such contextual information in their included studies and this area of reporting needs to be improved in future.

Phase 4 (determining how studies relate) was reasonably well reported, with 60% of the six applicable standards fully or partially met (average score of 3.7 standards met to some extent). Most audited publications met the standards for reporting how reviewers decided that the studies were related (n = 16), whether or not the studies were commensurable in focus (n = 15) and showing how they related them (n = 14) but, half of the time, this reporting lacked depth and these standards were deemed to be only partially met. Generally, reports about how original studies were related focused on their theoretical approaches, concepts/metaphors, types of health conditions and/or countries. Often, included studies were disparate, for example in terms of their health focus, cultural setting(s) and research design, and it was unclear to auditors how these studies related to each other, especially in meta-ethnographies with a large number of included studies (e.g. > 50). Inconsistent reference to, and interchangeable use of, included ‘studies’ and included ‘papers’ in the meta-ethnography reports meant that auditors were sometimes uncertain about whether or not individual papers were related because they were part of the same original study. Overall, how studies within the audited meta-ethnographies were related by temporal context was under-reported. Although the year of publication of original studies was provided, these meta-ethnography reports did not indicate how studies related in a wider temporal context (e.g. whether the original studies were conducted before/after the introduction of an international health policy or clinical guideline that may have influenced their findings and interpretations). Another poorly reported standard related to how multiple perspectives (e.g. academic or sociocultural) were introduced by the reviewers into phase 4; only seven audited publications (37%) fully or partially met this standard.

Phase 5 (translating studies) had the largest number of applicable standards (n = 20). Half (51%) of these applicable standards were fully or partially met (average score of 9.5 standards met to some extent). All publications provided some narrative regarding their phase 5 processes but this was usually too brief to enable auditors to fully understand how this essential meta-ethnography phase was conducted. Alternatively, such information was incomplete with, for example, details provided about the processes of LOA synthesis but not translation. Although all (n = 19) audited publications reported on the number of studies translated (this was the best-reported phase 5 standard), only 14 (74%) explicitly reported whether or not this number was also the same as the number of studies included in the LOA. Generally, there was a lack of information regarding how reviewers took steps, if any, to preserve context and meaning between concepts within and across individual studies during translation and/or whether or not sociocultural factors were considered during LOA synthesis [only eight (42%) and six (32%) publications met these standards to some extent, respectively]. Publications were good at stating what type of translation processes they had conducted (n = 17, 89% met this standard in full or in part), but it was less clear which methods they used to translate included studies [12 publications met this standard (63%) with only five doing so in full]. Reporting of refutational analysis processes, that is, where reviewers look for disconfirming cases, was not well done with most audited publications either meeting these standards in part or not at all. Where publications did report conducting refutational analysis, this usually involved looking at themes or metaphors that contrasted across individual studies, few audited publications reported considering refutational translation from wider contextual perspectives such as gender or ethnicity of original participants. Most publications (n = 18, 94%) presented grids/tables reporting the outcome of their translations and/or LOA synthesis but the depth of such information varied and was sometimes too brief to be really helpful to auditors, even when a supporting narrative was provided.

Phase 6 (synthesising translations) had the lowest number of standards overall (five in total). Only three standards applied to every publication and 70% of the audited publications fully or partially met these standards, with an average score of 2.1 standards being met. The best-reported phase 6 standard asked reviewers to state their new third-order interpretation in text or visually: all publications met this standard, although there was variation in the depth of information provided. Overall, reporting of methods used to develop synthesised translations was inadequate because only 13 audited publications met this standard and in half of these cases, the standard was considered to be only partially met. It was also unclear which reviewers were involved in synthesising translations (only eight publications met this standard). This is an important omission as auditors were uncertain as to which individuals from a team of reviewers had contributed to the synthesis and from what perspective(s) (e.g. discipline and/or epistemology).

Phase 7 (expressing the synthesis) was the best-reported phase overall, with 76% of the 15 applicable standards being met (average score of 10.8 standards fully or partially met). The best-reported standards included those identifying how findings related to potential end-users, the source of provided quotes and the overall limitations of the meta-ethnography (> 80% of the audited publications met these standards to some extent). How review limitations may have affected credibility and trustworthiness of the findings and possible limitations of the new theory or interpretation, such as whether or not it may apply to only certain groups, were fully or partially met in 68% of audited publications (n = 13). Although the audited publications were generally good at highlighting limitations of their review, such reporting focused on the limitations of the original studies rather than reviewers considering whether or not the ways in which they had conducted the meta-ethnography may have limited their findings. In particular, the poorest reporting related to the standard that asked reviewers to state how they encouraged reflexivity during development of their new interpretation [only five publications (26%) met this standard to some extent]. For this standard, auditors were specifically looking for evidence that reviewers had considered, for example, whether or not their new interpretation might be different if the review team consisted of individuals with different epistemological, academic, gender or cultural perspectives or, alternatively, whether or not reviewers had considered if their meta-ethnography findings were limited because of how they selected and related studies, or whether or not they had considered their final interpretation specifically in the context of disconfirming cases in their data. Among the audited publications, Galdas et al. 63 was one of the few that met this standard by, for example, reflecting on the gender of its reviewers. In addition, only six publications (32%) were considered to have fully or partially met the standard that asked reviewers to state what steps they took to keep their interpretation grounded in the original data.

The ENTREQ statement34 was designed to enhance transparency in the reporting of all types of qualitative evidence syntheses. Nine (48%) out of the 19 audited meta-ethnographies were published post ENTREQ, but only four of these explicitly referred to using ENTREQ to guide their meta-ethnography reporting (see Table 3). The applicable standards fully or partially met by these four publications varied from 58% to 88% (see Table 3). The best-reported publication in the audit (the one meeting most applicable standards in full or in part) did use the ENTREQ reporting guidance but this publication was 300 pages long, so its reporting should be more comprehensive than meta-ethnography reports confined by journal word limits. The PRISMA reporting guidance,32 although developed for quantitative systematic reviewing, was frequently used among the meta-ethnographies. Eleven (65%) out of the 17 post-PRISMA audit publications used this framework (or a variation of it) to guide their meta-ethnography reporting. PRISMA and ENTREQ, when used in the audited publications, did enhance reporting overall (specifically in the early stages of meta-ethnography) by, for example, providing clarity in how studies were identified for inclusion. PRISMA and ENTREQ probably also account for the widespread use of quality appraisal of included studies in the audited meta-ethnographies. Quality appraisal is a debated issue in meta-ethnography and is not always conducted, so these audit standards did not apply to every study. Nonetheless, most audited meta-ethnographies (78%; see Table 3) quality appraised their included studies in some way. This finding may also reflect the publication context of these meta-ethnographies as their target journals (or funders) may have expected this practice.

Discussion

Initially, 40 meta-ethnographies were identified for inclusion in the audit based on the reading of their title and abstracts (standard practice in systematic reviewing); however, when the auditors read the papers in full, 21 (52%) were not recognisable as meta-ethnographies and were excluded from the final audit sample. This is a significant finding because, for inclusion in the audit, publications had to refer to meta-ethnography, meta-ethnographic technique or Noblit and Hare25 in their title and/or abstract. Some were not reported in a format that was recognisable as meta-ethnography because, for example, they appeared as literature reviews, aggregated qualitative data or combined qualitative and quantitative studies in the one synthesis. Some of these 21 publications may have been conventional meta-ethnographies, but their reporting did not convey this to the auditors, for example by presenting thematic analysis of constructs reported in the original papers rather than providing a new overarching interpretation, and project time scales were such that original authors could not be contacted for further information. This pre-audit finding reiterates the urgent need for improved meta-ethnography reporting in full text reports/papers and abstracts as well as more judicious use of journal article keywords.

This was not a conventional health-care audit. Usually an audit is applied to discrete processes and outcomes (e.g. did a post-operative patient receive pain relief as per protocol?). Measuring practice against such standards is objective (e.g. did a patient get pain relief at the right time – yes or no?). By comparison, the stage 2.2 audit standards were measuring reporting of a complex and iterative qualitative research methodology within different publications with differing writing styles. We also had six auditors of differing backgrounds (e.g. health professionals, social scientists and information specialists) and with varying interests in meta-ethnography (e.g. researchers, journal editors and academic supervisors). What constitutes good meta-ethnography reporting ultimately depends on the reader’s perspective. Consequently, our audit standards could be met in full or in part. Clearly, whether a standard was considered fully or partially met depended on an auditor’s viewpoint and we did not assess inter-rater reliability. Nonetheless, the audit enabled trends to be identified across the sampled meta-ethnographies. In particular, although meta-ethnography reporting could be considered to be relatively good with all but two phases meeting > 50% of the applicable audit standards, overall, meta-ethnography reporting needed improving as most standards were considered by auditors to be partially rather than fully met.

The small number of purposively sampled meta-ethnography publications (n = 19) is an audit limitation. Although we had planned to have a larger final sample, our actual sample still contained a wide range of meta-ethnography reports, including various topics, journals, countries, academic disciplines and number of included studies. The sample also included single and multiple meta-ethnography reviewers and one study was published in both report and journal paper format. Despite a smaller than planned sample, the audit still generated a wealth of data and allowed the identification of individual standards where reporting needs to be improved. For example, standard 104 in phase 7 (reviewers stating how they encouraged reflexivity in their meta-ethnography) stood out because so few of the audited studies met this standard at all. The audit also enabled a more nuanced understanding of meta-ethnography reporting to develop. For instance, although reporting of the number of studies found for inclusion in a meta-ethnography was good (95% of publications met this standard), there was a need for greater clarity regarding the number of studies actually synthesised, as only 74% of the audited publications provided this information. Another example was a phase 6 standard asking whether or not reviewers of meta-ethnographies that included lots of studies (> 50) stated how they remained grounded with original data to avoid losing conceptual richness during the synthesis of translations. Although only one audited publication had > 50 included studies, the audit process identified that even meta-ethnographies with fewer included studies did not report this aspect well (e.g. reviewers did not state how they preserved the context of studies).

The audit also enabled us to identify ways in which the current meta-ethnography evidence base could be strengthened or refined. In particular, the audit identified that, despite considerable effort during standard development to remove duplication, areas of overlap within and between standards still existed in some meta-ethnography phases. Perhaps the most important example of this relates to phase 6. Synthesising translations is a critical element of meta-ethnography, yet this phase had the fewest number of standards (five overall and only three that applied to all audited studies). During the audit process, it was noted that some phase 5 standards also related to phase 6. For example, standards 72–74 referred to methods of translation and/or synthesis. The standards were systematically based on stage 1 evidence/recommendations for each meta-ethnography phase, so, by highlighting the overlap between these two phases in the theoretical evidence, this finding provides new insight into the lack of clarity in meta-ethnography reporting in practice. This finding suggests that future meta-ethnography reports could be enhanced through clearer differentiation between the specific information required at each of the meta-ethnography phases but especially in the reporting of translating studies (phase 5) and synthesising translations (phase 6).

Use of existing generic reporting frameworks did not help to ensure transparent and robust reporting in the quintessential phases of meta-ethnography (phases 4–6), such as explicit reporting of how reviewers encouraged reflexivity in their review processes and what steps were taken to ensure that meta-ethnography interpretations remained grounded in original data. In particular, use of generic reporting frameworks did not facilitate full reporting of contextual information in these meta-ethnography reports. Earlier stages of the eMERGe study, Project Advisory Group meetings and discussion with Professor George Noblit highlighted the critical importance of context across all meta-ethnography phases. However, the audit identified many ways in which context was poorly reported in the sampled publications across all meta-ethnography phases. For example, in phases 0–1, reviewers did not provide enough information about their review aims and research questions or their purpose for conducting a meta-ethnography. In phases 3–4, there was a lack of wider contextual information regarding studies included in a meta-ethnography, such as participant gender and ethnicity, the temporal context and disciplines/perspectives of the reviewers. There was also ambiguity regarding the context of individual papers, such as reviewers not explicitly stating whether or not any included papers came from the same original study. In phases 5–6, whether or not reviewers considered the contextual differences of included studies as a basis for refutational analysis was generally not stated and there was inadequate reflection on how the internal context of the audited meta-ethnographies influenced the review outcome. For example, how a team consisting of reviewers of a different gender and discipline may have influenced interpretation, or what influence the study selection process may have had of LOA synthesis, was also not reported. Inadequate contextual and reflexive reporting across the meta-ethnography phases was, therefore, an overarching theme arising from the audit findings.

Recruitment

We recruited 77 people to the workshop: 36 academics, 29 lay people/public/patient representatives and 12 other stakeholders. Thirty-one out of the 78 participants participated in the online workshop: 12 academics, 3 other professional stakeholders, 11 lay people and 5 project team members. A number of project participants wished to, but could not, attend the workshop. Nine of these (six academics and three lay people) commented on the workshop materials and detailed notes after the workshop.

Procedure

A 3-hour online workshop was held on 12 May 2016. We interacted with all stakeholders to discuss good and best practice and to further develop the draft standards and items for the reporting guideline and discuss/agree their wording.

Objectives

The objective of the eDelphi was to conduct two identical eDelphi consensus studies that would be run in parallel: one with meta-ethnography methodology expert participants and another with key stakeholders who use synthesised evidence (i.e. professional evidence users and patient and public representatives). These groups were separated as each brings specific expertise and could have potentially different views on the importance of specific items. Such differences may be lost if the samples are merged into the same consensus exercise.

We ran two separate but identical and parallel eDelphi studies: one for meta-ethnography methodology experts and one for other stakeholders. By carrying out two separate eDelphi studies, we ensured that we could differentiate between these two groups and therefore represent both groups’ views, so that items of importance to both groups would be included in the final guidance. If we conducted only one Delphi study, we would have been unable to discern which type of participant made up the majority vote for any item [e.g. an overall majority (dominated by academics) may have voted against including an item, but most other stakeholders might have voted to include the item]. Having two parallel Delphi studies also let participants in each panel compare their own responses with that of their peers when deciding whether or not to revise their previous responses.

We defined consensus as any item in the eDelphi study reaching ≥ 80% agreement on it being either ‘important’ or ‘very important’. Items reaching this level of consensus in either eDelphi study would be included in the final reporting guidance. 81,82

Methods

eDelphi procedure

We used a web-based platform developed at the University of Stirling by Edward AS Duncan and Kevin Swingler specifically for online Delphi studies. It had previously been piloted for acceptability and usability and successfully used in two previous separate studies. 81,82 This web-based platform has efficiency and economic advantages over standard Delphi study methods: its combination of automatic reminders, collation, analysis and feedback functions cannot be found in other generic electronic survey tools and it considerably increases efficiency by reducing the administration and manual analysis that is normally required between Delphi study rounds. Rates of study participation are comparable to paper-based administration methods. 43,82 We sought feedback from lay members of the Project Advisory Group to ensure that the eDelphi process was more accessible to those with disabilities.

The eDelphi study platform includes a recruitment and invitation process. Potential participants’ e-mail addresses were entered onto the platform. They were then sent an e-mail inviting them to participate in the study. The e-mail included information on the eDelphi platform web address and a password and unique identifier to use to log in to the website. Upon logging in, each participant was required to answer some consent questions before beginning the study.

Participants could save their responses during each round, enabling them to complete the questionnaire in more than one sitting. The eDelphi platform enabled data between rounds to be presented to participants visually in the form of a colour histogram or ‘heat map’ (Figure 9), overcoming some of the known limitations of using measures of central tendency88 when feeding back results to participants (e.g. when the median score disguises that consensus is polarised). The histogram for each item presented participants with information on their own response choice in the previous round, the frequency with which each of the four responses was chosen by the whole panel in the previous round (the depth of colour superimposed on the response scale indicates relative frequency) and the choice that they made in the current round. This enabled participants to easily compare their responses to the consensus in the previous round and to then either confirm or update their response. Figure 9 gives an example histogram showing the frequency with which each of the four responses was chosen in a previous round (the darker the shade of green, the greater the number who selected that response; the lighter the shade of green, the fewer the number). The grey circle shows the choice that the current participant made in the previous round and the green circle shows the choice that they made in the current round (in round one, each box is white because no previous selections have been made). Different colour options for the histogram were provided, which participants could select when logged in.

FIGURE 9.

An example of a colour histogram of previous responses used in eDelphi studies.

Results

We invited 71 potential meta-ethnography expert participants to take part in the study. A total of 48 individuals completed round 1 and 28 individuals completed three rounds of the study.

We invited 48 other potential key stakeholder expert participants to take part in the study. A total of 39 individuals completed round one and 23 individuals completed three rounds of the study.

Conclusion

The eDelphi process provided a rigorous method of identifying reporting items that were viewed as important or very important for inclusion in the eMERGe reporting guidance. The rigour of the development of the reporting standards earlier in the project (see Chapters 3 and 4) resulted in almost all of the items in both eDelphi panels reaching consensus for inclusion. This necessitated the study team to consider how items could be meaningfully merged and presented in a usable format for end-users. This process is described in Chapter 6.

Stage 1

We had intended to independently double-screen all the retrieved references by title and abstract. The search output was sensitive but not specific for our purpose, which meant that we retrieved a very large number of references. Therefore, we decided not to independently double-screen references published prior to the year 2006 to enable us to meet our aims and project timelines. The pre-2006 references that referred to qualitative evidence synthesis had been superseded and the majority of relevant papers about meta-ethnography published prior to 2006 were already known to the project team. However, as a precaution, titles and abstracts of references from 2005 and earlier were electronically searched for key terms (e.g. ‘ethnograph’, ‘Noblit’) to identify any referring to meta-ethnography; these references were then screened by title and abstract by one reviewer. We also used expansive searches and approached experts to identify other relevant publications not identified through the database searches.

Stage 2

We made changes to the research questions. The original research question for stage 2 was:

• What good practice principles and standards in meta-ethnography conduct and reporting can we identify from published meta-ethnographies to inform recommendations and guidance?

The revised research questions for stage 2 were:

• What good practice principles can we identify in meta-ethnography conduct and reporting to inform recommendations and guidance?

• From the good practice principles, what standards can we develop in meta-ethnography conduct and reporting to inform recommendations and guidance?

We informed NIHR of the amended research questions and had these approved.

Stage 2.1

We conducted 14 semistructured interviews with professional end-users of evidence syntheses rather than the 10 we had planned in order to include feedback from a wider range of stakeholder organisations.

Analysis of seminal/low-quality meta-ethnographies, including interviews with professionals, and the development of the draft standards, happened in parallel rather than sequentially; this did not affect the quality of the analysis or standard development and allowed us to meet the project schedule.

To analyse seminal/lower-quality meta-ethnographies, we had planned to have three reviewers independently code the same two meta-ethnographies, but this was unnecessary to achieve rigour. Instead, three reviewers shared the coding and discussed and verified the analysis. We focused coding and analysis efforts on the complex analytic synthesis and expressing synthesis phases 4–7, rather than all seven phases of a meta-ethnography to achieve depth of insight within time constraints.

Stage 2.2

We had intended to include a diverse sample of 40 meta-ethnographies in the audit and we selected a purposive sample of 40 meta-ethnographies that met our inclusion criteria for this purpose. However, when the full texts of these papers were read, 21 of the publications were not recognisably meta-ethnographies. The project team discussed how to handle these papers and reached the decision to remove them from the audit. Project time scales meant that we could not conduct further sampling, and so the audit had a final sample of 19 papers.

The aim of this interview

We want to collect your views on the meta-ethnographies you have been provided with in order to identify those areas of meta-ethnography reporting that are important to people or organisations likely to use qualitative evidence to inform policy and practice.

What we want your views on

SCOPUS (inception to date)

• (ABS (meta ethnography) OR TITLE (meta ethnography) OR ABS (metaethnography) OR TITLE (metaethnography))

MEDLINE (1946 to date) and PsycINFO (1806 to date)

• ((meta ethnography) OR (metaethnography)).ti,ab

EBSCOhost Cumulative Index to Nursing and Allied Health Literature (inception to date)

• TI (meta ethnography OR metaethnography) OR AB (meta ethnography OR metaethnography)

International Bibliography of Social Sciences (inception to date)

• ab(metaethnography OR meta ethnography) OR ti(metaethnography OR meta ethnography)

Web of Science Core Collection (inception to date)

Cochrane database of qualitative evidence syntheses

• Terms used: metaethnography, meta ethnography, metaethnographic, meta ethnographic, Noblit.

• Registers are populated by a keyword strategy of Web of Science, Cumulative Index to Nursing and Allied Health Literature Scopus and PubMed, plus Google Scholar alerts and citation searches of key works (which includes Noblit and Hare).

• Inclusion is from 1988 onwards.

Film: What is a qualitative evidence synthesis and what is meta-ethnography?

YouTube video by Dr Emma France, University of Stirling, available at www.youtube.com/watch?v=oPYL3oAwb4Q&feature=youtu.be (accessed 26 March 2018).

Film: meta-ethnography then and now

Video film featuring Professor George Noblit, Professor of Sociology of Education, University of North Carolina available at www.youtube.com/watch?v=w3BB0IschGk (accessed 26 March 2018).

Film: overview of the eMERGe project and development of the reporting guidance

Video film featuring Dr Emma F France and Lynne Gilmour, University of Stirling, available at www.youtube.com/watch?v=A9HzPnYm0RA (accessed 26 March 2018).

Film: eMERGe reporting guidance – the wider context and its possible use

Video film featuring Professor Jane Noyes, Bangor University and Lynne Gilmour, University of Stirling, available at www.youtube.com/watch?v=1CvXm526AbY (accessed 26 March 2018).

Film: eMERGe reporting guidance – their format, content and use

Video film featuring Dr Nicola Ring, Edinburgh Napier University and Lynne Gilmour, University of Stirling, available at www.youtube.com/watch?v=SenAIq8ck0s (accessed 26 March 2018).

Webinar recording of Introducing the New Meta-Ethnography Reporting Guidance – What it is and how to use it

Recording of a webinar led by Dr Nicola Ring, Edinburgh Napier University available at www.youtube.com/watch?v=58zv3PTttok&t (accessed 26 March 2018).