Notes
Article history
The research reported in this issue of the journal was funded by the HS&DR programme or one of its preceding programmes as project number 13/05/11. The contractual start date was in December 2014. The final report began editorial review in June 2015 and was accepted for publication in August 2015. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.
Declared competing interests of authors
none
Permissions
Copyright statement
© Queen’s Printer and Controller of HMSO 2016. This work was produced by Rodgers et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
Chapter 1 Background
Case study is commonly understood to be a method of research that engages in the close, detailed examination of a single example or phenomenon, and is an approach commonly used to understand activity and behaviour within a real-life context. When conducted well, organisational case studies can provide insights into organisational changes in health care that are not easily achieved through other study designs. They can be used to identify facilitators and barriers to the delivery of services and to help understand the influence of context, and high-quality organisational case studies have been used to examine ways of working in acute care,1 primary care,2 mental health services,3 residential care4 and across the NHS more broadly. 5–7
Yin8 describes a case study to be the preferred research method when (1) the main research questions are ‘how?’ or ‘why?’; (2) a researcher has little or no control over behavioural events; and (3) the focus of the study is a contemporary (as opposed to historical) phenomenon. However, there is no set methodology for a case study and the term is often used loosely, but typically combines qualitative and quantitative data collection with a strong observational component. Case study research can be conducted from both relativist and positivist perspectives, and can be used to generate new theories, validate existing theories or address both of these matters. 9 An individual case can be studied alone to understand something about the case itself and its contexts, or compared with other cases for illustrative, explanatory or evaluative purposes. 10
The case study has been proposed as an appropriate method for describing, explaining, predicting or controlling processes associated with phenomena at the individual, group or organisational level. 11 The majority of the National Institute for Health Research (NIHR)’s Health Services and Delivery Research (HSDR) programme-funded case studies are specifically concerned with description or explanation at the organisational level.
In the past, many proposals for organisational case studies submitted to the HSDR programme have been poorly articulated and methodologically weak and were therefore unlikely to deliver robust research findings. Specific areas of concern raised by the HSDR programme included:
-
Absence of clear research questions that the case study method is intended to answer.
-
Vagueness about sampling frame/strategy. Proposals where it is not clear how organisations or sites were selected or what was the basis for sampling.
-
Insufficient theoretical basis. Many studies lack an organising theoretical framework; this can affect all stages, from sampling of sites through to analysis and how findings can add to the body of knowledge.
-
Lack of clarity about the unit of analysis. Some weaker proposals will not identify the unit of interest, whereas good case studies may include data streams around the individual, team, organisation and wider system, and will be explicit about the overall study design and interest.
-
Lack of any clear plans for analysis. Some proposals make no attempt to look actively for data that challenge emerging theories, findings or knowledge of systematic comparative case analyses. Many such studies are purely descriptive, without any explanatory power.
-
Lack of clarity about how data from a range of sources will be integrated.
-
Proposals increasingly claim to use realist evaluation methods for case study work, but make no attempt to establish a programme theory, identify candidate mechanisms or describe other features of realist evaluation.
Consequently, the HSDR programme expressed an interest in identifying the characteristics of good-quality case study research, and in devising quality and publication standards, with particular application to the NHS. More specifically, they described the need for a rapid evidence review alongside a Delphi or expert consensus-building exercise to identify elements of good practice and standards for reporting and publication.
Although some authors have proposed practical methodological guidelines for case study research methods,8,9 these have not been universally adopted. The broad diversity of approaches used within organisational case studies – and the contrasting paradigms that underpin these approaches – mean that any attempt to develop ‘definitive’ methodological guidance in this area is likely to be both highly contentious and resource-intensive. However, the ability of research funders, peer reviewers and other research users to establish methodological quality is at least partly contingent on the clarity of explanation of the methods proposed or utilised. Indeed, several of the concerns raised by the HSDR programme above specifically refer to vagueness or lack of clarity around the reporting of proposed research methods.
Reporting standards already exist for a range of study designs, including randomised trials,12 observational studies,13 systematic reviews,14 clinical case reports,15 qualitative research,16 realist syntheses,17 meta-narrative reviews,18 diagnostic/prognostic studies,19 quality improvement studies20 and economic evaluations. 21 However, to date, no such standards have been reported for organisational case studies.
By encouraging authors to consider how their methods are presented, the availability of an appropriate set of reporting standards for organisational case studies also has the potential to improve research conduct in general. A suitable first step towards better conduct of organisational case studies would be to establish agreement about what needs to be reported among the diverse group of researchers who undertake this kind of research. Should further guidance be needed about appropriate methods, the reporting standards can be used as a foundation on which to build.
The aim of this project has been to identify the characteristics of good-quality organisational case study research and devise reporting standards, with particular application to the NHS. Although a range of opinions and experiences has been sought, the project has not been concerned with case studies outside the remit of the work funded by the HSDR programme. Therefore, it is not intended the reporting standards should be applied to case studies of individuals or to those conducted in other research fields.
In the first instance, we would anticipate that these standards should be used to improve the standard of submissions to the HSDR programme. There may be further potential for dissemination of the standards to the wider world of organisational case study researchers.
Chapter 2 Methods
Research aim
The aim of the project was to develop reporting standards for organisational case study research, with particular application to the UK National Health Service.
Scope
We developed the reporting standards in three stages, as shown in Figure 1:
-
a rapid review of the existing literature to identify content for the standards
-
a Delphi consensus process to develop and refine the final set of standards
-
application of the high-consensus Delphi items to two samples of organisational case studies to assess their feasibility as reporting standards.
FIGURE 1.
Outline of the research process.
Rapid review
A rapid review was used to generate items to populate a provisional framework for organisational case studies. Systematic review methodology was used to identify articles and extract and synthesise data. Because of the rapid nature of the review, the process was less exhaustive and contained less detail than would have been achievable from a full systematic review.
Literature searching
The aim of the search strategy was to identify material about organisational case study methods. It was anticipated that the literature on this topic would be found in textbooks, book chapters, journal articles and research methods guidance; therefore, the search strategy consisted of searches of library catalogues, key author searches, focused searching of health and social science databases and some targeted website searching.
Library catalogue searches
The following library catalogues specialising in health management literature were searched to locate books on case study methods:
-
Health Services Management Centre ONLINE (via the University of Birmingham; www.birmingham.ac.uk/facilities/hsmc-library/library-resources/index.aspx)
-
Health Management Online (via NHS Scotland; www.shelcat.org/nhml)
-
The King’s Fund Library Database (http://kingsfund.koha-ptfs.eu/).
Key author searches
Five authors, detailed in Box 1, featured prominently in the initial literature searches: David Byrne, Bent Flyvbjerg, Roger Gomm, Charles Ragin and Robert K Yin. Searches were carried out via Google (Google Inc., Mountain View, CA, USA) on each author to locate any lists of their publications. Publication lists were found for each author on either their institution website or, where this was not available, through searches of Google Books and Google Scholar.
-
Byrne D. Complex realist and configurational approaches to cases: a radical synthesis. In Byrne D, Ragin C, editors. The SAGE Handbook of Case-based Methods. London: SAGE Publications; 2009. pp. 101–13. 22
-
Byrne D. Introduction: case-based methods: why we need them; what they are; how to do them. In Byrne D, Ragin C, editors. The SAGE Handbook of Case-based Methods. London: SAGE Publications; 2009. pp. 1–10. 23
-
Byrne DS. Complexity, configuration and cases. Theor Cult Soc 2005;22:95–111. 24
-
Flyvbjerg B, Landman T, Schram S. Important next steps in phronetic social science. In Flyvbjerg B, Landman T, Schram S, editors. Real Social Science: Applied Phronesis. Cambridge: Cambridge University Press; 2012. pp. 285–97. 25
-
Flyvbjerg B, Landman T, Schram S. Real Social Science: Applied Phronesis. Cambridge: Cambridge University Press; 2012. 26
-
Flyvbjerg B. Case study. In Denzin NK, Lincoln YS, editors. The SAGE Handbook of Qualitative Research. 4th edn. Thousand Oaks, CA: SAGE Publications; 2011. pp. 301–16. 27
-
Gomm R. Key Concepts in Social Research Methods. London: Palgrave Macmillan; 2009. 28
-
Gomm R, Davies C. Using Evidence in Health and Social Care. London: SAGE Publications; 2000. 29
-
Gomm R, Hammersley M, Foster P. Case Study Method – Key Issues, Key Texts. London: SAGE Publications; 2000. 30
-
Ragin CC. Reflections on casing and case-oriented research. In Byrne D, Ragin CC, editors. The SAGE Handbook of Case-based Methods. London: SAGE Publications; 2009. pp. 522–34. 31
-
Ragin CC, Schneider G. Case-oriented theory building and theory testing. In Williams M, Vogt P, editors. The SAGE Handbook of Methodological Innovations. London: SAGE Publications; 2010. pp. 150–6632
-
Ragin CC, Schneider G. Comparative political analysis: six case-oriented strategies. In Amenta E, Nash K, Scott A, editors. The New Blackwell Companion to Political Sociology. Chichester: Blackwell; 2012. pp. 78–91. 33
-
Yin RK. Applications of Case Study Research. 3rd edn. London: SAGE Publications; 2012. 34
-
Yin RK. Case study methods. In Cooper H, editor. APA Handbook of Research Methods in Psychology, Vol 2: Research Designs: Quantitative, Qualitative, Neuropsychological, and Biological. Washington, DC: American Psychological Association; 2012. pp. 141–55. 35
-
Yin RK. Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Publications; 2014. 8
Database searches
Initial database test searches revealed difficulties in locating case study methodology literature efficiently without retrieving large numbers of irrelevant results. Therefore, as this was a rapid review, a highly focused search strategy was developed on MEDLINE (via Ovid) to identify papers about organisational case study methods. Focusing of subject headings, use of subheadings and searching in the title-only field were utilised in the strategy.
Searches were restricted to English-language papers. A more limited range of databases than usual for a full systematic review was searched. In particular, no specific databases of conference proceedings, theses or foreign-language studies were searched.
Relevant databases covering literature from health, health management and social science were searched: MEDLINE & MEDLINE In-Process, Applied Social Sciences Index and Abstracts (ASSIA), Health Management Information Consortium, PsycINFO and the Social Science Citation Index. The MEDLINE strategy was adapted for use in each database.
Website searches
The following websites were searched to identify any guidance documents on case study methods:
-
ESRC National Centre for Research Methods (www.ncrm.ac.uk/)
-
ESRC Research Methods Programme (www.ccsr.ac.uk/methods/)
-
The Social Research Association (http://the-sra.org.uk/)
-
Methods@Manchester (www.methods.manchester.ac.uk/).
Citation searching
Citation searching on case study methods texts from key authors had been planned in the protocol. However, test citation searches identified large numbers of results; therefore, given the rapid nature of the review, citation searching was not feasible within the timescale.
Records were managed within an EndNote library (version X6; Thomson Reuters, CA, USA). After deduplication 3465 records in total were identified.
Further details of the full search strategies and results can be found in Appendix 1.
Inclusion and exclusion criteria
We sought to identify three sources of information:
-
methodological texts that reported on the methods used in conducting organisational case study research
-
real-world ‘example’ case studies identified from the searches
-
methodologically sound ‘exemplar’ case studies identified by the NIHR HSDR programme.
Inclusion and exclusion criteria are listed in the following sections.
Methodological texts
Texts were included if they:
-
described the conduct of organisational case studies, where organisational means relating to an organised body of people with a particular purpose, such as a business, government department or charity group
-
contained practical advice on conducting case study research.
Texts were excluded if they were:
-
concerned with case studies of individuals (e.g. describing a single patient)
-
concerned with qualitative, quantitative or mixed methods in general, rather than case studies in particular
-
primarily conceptual or theoretical discussions without practical guidance.
Methodological texts were not restricted by topic area. Thus relevant methodological texts from outside health/social services literature, such as business and education, were eligible for inclusion.
We focused on practical rather than conceptual texts to identify potential items for reporting standards, but were mindful that organisational case studies can have different underlying epistemological assumptions (e.g. positivist vs. relativist), and that some paradigms lend themselves more easily to practical advice than others.
Example case studies
These were included if they:
-
reported an organisational case study (as defined above)
-
were undertaken in a UK NHS or social services settings.
The purpose of including the example case studies was to identify any additional items for the Delphi consensus process (see Delphi consensus process) that had not already been identified from the methods literature. We therefore prioritised those organisational case studies with particular relevance to a UK NHS and social services settings.
Exemplar case studies
The funders of this review provided examples of what they considered to be methodologically strong case study research projects funded by the NIHR HSDR programme. These were also examined to identify further items to inform the Delphi consensus process.
Selection of relevant evidence
Methodological texts
An initial examination of the EndNote library identified a very large number of irrelevant records referring to research methods more broadly, therefore we ran a search for ‘case stud*’ in the title or abstract in order to restrict the results to relevant methodological texts. Two reviewers (MR/ST) then independently screened titles and abstracts, with disagreements resolved through discussion. Full-text copies were obtained for potentially relevant records and again screened independently by the same two reviewers. Disagreements were resolved by discussion with a third reviewer (AE).
EPPI-Reviewer version 4 (Evidence for Policy and Practice Information and Co-ordinating Centre, Social Science Research Unit, Institute of Education, University of London, UK) text-mining software was used by one reviewer (ST) to screen the remaining titles and abstracts to establish whether or not any relevant texts could have been missed by our restricted search. The text-mining process ‘learned’ what were relevant texts as the reviewer progressed through screening and brought these to the top of the list, enabling faster retrieval of full texts for assessment and potential incorporation into the review. All titles and abstracts were screened, with a decreasing number of texts being selected as the process continued. Full-text copies were obtained for potentially relevant records and screened for inclusion by one reviewer (ST). A second reviewer (MR or AE) examined excluded records. Disagreements were resolved by discussion.
Example case studies
To identify example case studies we further restricted the results gathered from the EndNote library by searching for the terms ‘organisational’ or ‘organizational’ in either the title or abstract within the ‘case stud*’ subset of results.
One reviewer (ST) screened the titles and abstracts, obtained full-text copies of potentially relevant records and selected these for inclusion. The selection was checked by a second reviewer (MR) and disagreements were resolved through discussion with a third reviewer (AE).
Exemplar case studies
Twelve case study projects were identified as exemplar case studies by the HSDR programme staff who commissioned the review. For each project, we downloaded the relevant commissioning brief and, where available, the protocol, final report and journal articles from the HSDR website.
Data extraction
Data extraction forms were created to capture the key methodological components described in individual studies. The forms provided a standard framework while accommodating different approaches; the authors’ own wording was used wherever possible.
The methodological texts were extracted first, beginning with the two most commonly cited case study methods texts. 8,36 The remaining methodological texts were then extracted in reverse chronological order. For the subsequent data extraction, we tried to restrict extraction to additional non-duplicate items; truly identical items identified from two or more sources (i.e. duplicates) were only extracted once, though if two or more items were considered to be similar (but non-identical), these were retained.
Data from the example and exemplar case studies were then extracted in a similar way. For included case studies, we focused on identifying the reporting methods, rather than critically appraising the underlying methodology; we aimed to develop a generic reporting structure that could be applied to a range of different types of organisational case study.
Data extraction was conducted by one reviewer (MR or ST) and checked by a second (MR, ST or AE). Disagreements were resolved through discussion.
Synthesis
To generate items for the Delphi consensus process, the individual data extraction forms from the methodological texts were combined into one overall document. A comprehensive and iterative process of refinement was then undertaken, combining and grouping similar components and further removing duplicates. A similar process was undertaken to add in any additional components from the individual example and exemplar case studies.
An initial framework was created by broadly grouping items by research stage, as follows:
-
planning and study design
-
data collection
-
data analysis
-
reporting.
This provisional framework expanded and evolved as the items were extracted, synthesised and revised.
Delphi consensus process
The content of the framework was refined and developed through a modified Delphi consensus process. The Delphi technique is a structured and iterative method for collecting anonymous individual opinions from a panel with relevant expertise in the topic where consensus is required. 37 The basic principle is for the panel to receive successive questionnaires, each one containing the anonymous responses to the previous round, and for them to modify their responses until consensus is reached.
The Delphi consensus process was employed in order to obtain consensus from experts on the minimum set of reporting criteria that could form the basis of standards for the reporting of future organisational case studies.
Design
The reporting standards were developed over two rounds:
-
In the first round, participants were presented with all the unique items identified from the rapid review. They were asked to rate each item as being ‘essential’, ‘desirable’ or ‘not necessary’ for the reporting of organisational case studies. Participants were also asked whether or not the provisional framework in which items were presented was appropriate, and were given the opportunity to adapt this alongside the content.
-
In the second round, participants received a restructured list of items incorporating the results of the first round. Within each section, participants were first presented with high-consensus items (i.e. those receiving > 70% ‘essential’ responses in round 1), and given the opportunity to state whether such items should be reported by all organisational case studies, specific types of organisational case study or do not need to be reported. The remaining non-consensus items were ranked according to their positive/negative ratio of ratings from round 1. This ratio was calculated for each item by dividing the sum of ‘essential’ and ‘desirable’ counts by ‘not necessary’ counts. Consequently, a ratio value of 1 would indicate an even balance of positive and negative ratings. Participants were provided with each item and its corresponding ratio and again asked whether the item should be reported for all, some or no organisational case studies.
In both rounds, the high-consensus threshold was set at 70% agreement among respondents for each item. This threshold was chosen as it reflects a greater than 2 : 1 ratio of agreement to dissent, representing much stronger consensus than would a simple majority agreement threshold of 50% or greater.
Each round was open for 3 weeks, with a reminder sent to non-responders at the end of the first week.
Participants
Experts and parties interested in the conduct of organisational case study research (methodologists, research funders, journal editors, interested policymakers and practitioners) were invited to participate. Individuals were identified through the rapid review, personal contacts, and by contacting the following organisations: Health Services Research Network; the Social Research Association; the UK Evaluation Society; and the National Centre for Research Methods.
All contacts were assured confidentiality, with the aim of encouraging participation and openness, and all were invited to each round of the survey, including previous-round non-responders (unless they chose the option to withdraw from further contact).
In order to assess representation of different stakeholder groups and identify any important differences in their responses, professional characteristics were requested in each questionnaire. These included designation, topic area of interest, research method of interest and proportion of work relating to methodology.
Instrumentation
Questionnaires were administered electronically using on-line survey software Qualtrics (Qualtrics, Provo, UT, USA) and all questionnaires were piloted before distribution.
Analysis
All responses were collected in Qualtrics for initial tabulation and analysis. Subsequent analyses and outputs were produced in Microsoft Excel® (Microsoft Corporation, 2010, Redmond, WA, USA). Where a respondent did not reply to a question, this value was recorded as missing. There was no imputation of missing values.
Ethical approval
Ethics approval for the consensus process was obtained from the University of York Health Sciences Research Governance Committee. Invitees were promised anonymity and submission of completed questionnaires was taken as implied consent.
Translating high-consensus Delphi items into reporting standards for organisational case studies
During the process of gathering the data from the real-world example case studies and the exemplar organisational case studies provided by HSDR, we became interested in how these might match up to the reporting standards. Although this step had not been part of the original protocol, we decided to add an additional step in the development process. One reviewer (ST) applied the list of high-consensus items, as far as was possible in retrospect, to all identified example and exemplar case study publications. These were subjective decisions made by one reviewer and are not intended to be a criticism of the quality of reporting in these publications. Rather our aims were (1) to determine the relevance of these items to the reporting of real-world organisational case studies and (2) to better understand how the results of the Delphi consultation might best be implemented as reporting standards. The results of this application are discussed in Translating high-consensus Delphi items into reporting standards for organisational case studies.
Chapter 3 Results
Rapid review
The searches identified 3465 potentially relevant references. After deduplication there were 2456 references, which were manually screened, with 2348 excluded based on title and abstract. Of the 1009 references screened with the assistance of text mining, 974 were excluded. Thirty-five records were identified from the screening of titles and abstracts during text mining, but no additional texts were included after reading of full texts. Following screening of full texts we included: 25 methodological texts,8–10,34–36,38–56 eight example case studies (17 publications)57–73 and 12 exemplar case studies (16 publications) provided by the HSDR programme. 1–7,74–82 The study by Raine et al. 77,78 was described in publications as ‘a prospective observational study’, but contained many elements of a case study and was identified as an exemplar of organisational case study research. See Figure 2 for details.
FIGURE 2.
A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)14 flow chart.
Rapid review: methodological texts
Twenty-five methodological texts were included in the rapid review. 8–10,34–36,38–56 Dates of publication covered 20 years, ranging from 1994 to 2014. One text47 was received too late to include in the Delphi consensus process, but it included no new items to add to the final list.
A number of key authors in the field of case study methodology had been identified in the early stages of our review (see Chapter 2, Rapid review, Literature searching, Key author searches). Other key authors were also identified. A complete list of these authors and their publications is provided in Table 1. After reading the texts, we selected those that gave practical advice on conducting research.
| Author (year) | Title and publication details |
|---|---|
| Crowe et al. (2011)38 | The case study approach. BMC Med Res Methodol 2011;11:100 |
| Darke et al. (1998)39 | Successfully completing case study research: combining rigour, relevance and pragmatism. Inf Syst J 1998;8:273–89 |
| Fitzgerald and Dopson (2009)40 | Comparative case study designs: their utility and development in organizational research. In Buchanan DA, Brynam A, editors. The SAGE Handbook of Organizational Research Methods. London: SAGE Publications; 2009. pp. 465–83 |
| Gagnon (2010)9 | The Case Study as Research Method: A Practical Handbook. Québec City, QC: Presses de l’Université du Québec; 2010 |
| Gibbert and Ruigrok (2010)41 | The ‘what’ and ‘how’ of case study rigor: three strategies based on published work. Organ Res Methods 2010;13:710–37 |
| Gibbert et al. (2008)42 | What passes as a rigorous case study? Strategic Manage J 2008;29:1465–74 |
| Gilgun (1994)43 | A case for case-studies in social-work research. Soc Work 1994;39:371–80 |
| Gillham (2000)44 | Case Study Research Methods (Continuum Research Methods). London: Bloomsbury Academic; 2000 |
| Greene and David (1984)45 | A research design for generalising from multiple case studies. Eval Program Plann 1984;7:73–85 |
| Hays (2004)46 | Case study research. In deMarrais K, Lapan SD, editors. Foundations for Research: Methods of Inquiry in Education and the Social Sciences. Mahwah, NJ: Lawrence Erlbaum Associates Publishers; 2004. pp. 217–34 |
| Hutchinson (1990)47 | The case study approach. In Moody LE, editor. Advancing Nursing Science through Research (Vol. 2). Newbury Park, CA: SAGE Publications; 1990. pp. 177–213 |
| Huws and Dahlmann (2007)48 | Quality Standards for Case Studies in the European Foundation. Dublin: European Foundation for the Improvement of Living and Working Conditions; 2007 |
| Kaarbo and Beasley (1999)49 | A practical guide to the comparative case study method in political psychology. Polit Psychol 1999;20:369–91 |
| Meyer (2001)50 | A case in case study methodology. Field Methods 2001;13:329–52 |
| Moore et al. (2012)51 | Case study research. In Laplan SD, Quartaroli MT, Riemer FJ, editors. Qualitative Research: An Introduction to Methods and Designs. San Francisco, CA: Jossey-Bass; 2012. pp. 243–70 |
| Stake (2005)36 | Qualitative case studies. In Denzin NK, Lincoln YS, editors. The SAGE Handbook of Qualitative Research. 3rd edn. Thousand Oaks, CA: SAGE Publications; 2005. pp. 443–66 |
| Stake (1995)10 | The Art of Case Study Research. Thousand Oaks, CA: SAGE Publications; 1995 |
| Stake (1994)52 | Case studies. In Denzin NK, Lincoln YS, editors. Handbook of Qualitative Research. Thousand Oaks, CA: SAGE Publications; 1994. pp. 236–47 |
| Thomas (2011)53 | How to do your Case Study: A Guide for Students and Researchers. London: SAGE Publications; 2011 |
| Yin (2014)8 | Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Publications; 2014 |
| Yin (2012)34 | Applications of Case Study Research. 3rd edn. London: SAGE Publications; 2012 |
| Yin (2012)35 | Case study methods. In Cooper H, editor. APA Handbook of Research Methods in Psychology, Vol 2: Research Designs: Quantitative, Qualitative, Neuropsychological, and Biological. Washington, DC: American Psychological Association; 2012. pp. 141–55 |
| Yin (2006)56 | Case study methods. In Green GL, Camilli G, Elmore PB, editors. Handbook of Complementary Methods in Education Research. Mahwah, NJ: Lawrence Erlbaum Associates; 2006. pp. 111–122 |
| Yin (1999)55 | Enhancing the quality of case studies in health services research. Health Serv Res 1999;34:1209–24 |
| Yin (1998)54 | The abridged version of case study research. Design and method. In Bickman L, Rog DJ, editors. Handbook of Applied Social Research Methods. Thousand Oaks, CA: SAGE Publications; 1998. pp. 229–59 |
The most commonly cited publications were by the authors Yin8,34,35,54–56 and Stake. 36 Therefore, the items for the Delphi consensus process were initially drawn from six publications by Yin8,34,35,54–56 and three by Stake. 10,36,52 The remaining texts were read in reverse chronological order to identify any additional items, with a decreasing number of new items found as we progressed back in time. See Appendix 2 for the complete list of items together with authors.
The language and paradigmatic assumptions related to each extracted item are likely to reflect the position of the original academic author. For example, the application of concepts such as validity and reliability to case studies derives directly from the publications of Yin. 8,34,35,54–56
Across all the included texts authors gave various definitions of case study research, made different paradigmatic assumptions, and recommended different methods. Rather than taking a particular position, we aimed to capture all these variations for inclusion in the first phase of the Delphi consensus process.
Rapid review: example case studies
Eight example case studies (with 17 associated publications) were included. 57–73 All the studies were conducted in England, with most relating to the NHS and one evaluating prison mental health in-reach services. 73 Dates of publication ranged from 2004 to 2011. The methods, as reported by the authors, covered a variety of approaches (see Table 2 for details).
| Author (year) | Type | Summary of case study | Where conducted | Author-reported case study methods |
|---|---|---|---|---|
| Attree et al. (2008)57,58 | J | The study explored patient safety in an English pre-registration degree nursing curriculum, based on the Nursing and Midwifery Council 2002 curriculum guidelines | NHS health-care trusts, north of England | Multiple organisational case studies |
| aField et al. (2005)59,65,68,69 | R, J | Phase 2: five in-depth organisational case studies were conducted with adult hospice bereavement support services in England | NHS hospices, England | In-depth multiple organisational case studies |
| Hutchinson and Purcell (2010)61 | J | Drawing on case study research in seven NHS trusts, the study considers the role and management of ward managers and paramedic supervisors, focussing on their human resource management responsibilities | NHS, south of England | A multiple case approach |
| Kyratsis et al. (2010)62,63 | R, J | To understand the impact of differing organisational capacity and contextual circumstances on technology selection, as well as the subsequent procurement and implementation of the technologies in 12 English NHS trusts | NHS trusts, England | A qualitative, multisite, comparative case study |
| National Nursing Research Unit (2009)64 | R, J | Phase 2: issues of local implementation of ‘The Productive Ward’ programme in five NHS acute trusts | NHS acute hospitals, England | A mix of qualitative research methods |
| The Offender Health Research Network (2009)70,73 | R, J | Evaluating prison mental health in-reach services using case study sites across the North West, North East and Yorkshire, South West, South East and London regions | Prison and young offender institutions, England | Qualitative analysis and a multiple case study approach |
| aPayne et al. (2007)65 | J | Case study research methods in end-of-life care: reflections on three studies | NHS hospices, England; NHS hospitals, England; NHS UK | Reflections on methods used in three case studies, including Payne et al.a |
| aPayne et al. (2004)60,65–67 | R, J | Phase 3: six in-depth organisational case studies of community hospitals in the South East and South West of England to identify how palliative care for elderly people is delivered in practice from the perspectives of service users and service providers | NHS community hospitals, England | In-depth multiple organisational case studies |
| aRolls and Payne (2004)65,72 | J | A multiple case study design: the context and processes of childhood bereavement services, the experiences of families who use them and the complexity of the contextual conditions that surround UK childhood bereavement services | NHS UK | In-depth multiple organisational case study approach as part of a larger qualitative study |
Some of the case studies were parts of wider projects that included other methods of evaluation. In such cases we focused only on the methods used for the organisational case studies.
The level of reporting of organisational case study methods within individual publications varied. After assessing all the publications for each included organisational case study, no new items were found to add to the Delphi consensus process.
Exemplar case studies
Twelve case studies (16 publications) funded by the NIHR HSDR programme were identified by the funder as being methodologically strong. 1–7,74–82 The methods, as reported by the authors, covered a variety of approaches (see Table 3 for details).
| Chief investigator | Type | Project ID, title and link to HSDR project page | Where conducted | Author-reported case study methods |
|---|---|---|---|---|
| Checkland K2 | R | 08/1808/240: management practice in Primary Care Organisations: the roles and behaviours of middle managers and GPs www.nets.nihr.ac.uk/projects/hsdr/081808240 |
Primary care trusts England | Qualitative case study methods |
| Closs J1 | P | 11/2000/05: the detection and management of pain in patients with dementia in acute care settings: development of a decision tool www.nets.nihr.ac.uk/projects/hsdr/11200005 |
NHS England | Multiple case studies with embedded units of analysis |
| Drennan VM74 | R | 09/1801/1066: investigating the contribution of physician assistants to primary care in England: a mixed-methods study www.nets.nihr.ac.uk/projects/hsdr/0918011066 |
NHS England | Mixed methods (qualitative and quantitative) |
| Gillard S3 | R | 10/1008/15: new ways of working in mental health services: a qualitative, comparative case study assessing and informing the emergence of new peer worker roles in mental health services in England www.nets.nihr.ac.uk/projects/hsdr/10100815 |
NHS England | Comparative case study design |
| Goodman C4 | P | 11/1021/02: optimal NHS service delivery to care homes: a realist evaluation of the features and mechanisms that support effective working for the continuing care of older people in residential settings www.nets.nihr.ac.uk/projects/hsdr/11102102 |
NHS England | Realist evaluation |
| Martin GP5–7 | R, J | 09/1001/40: the medium-term sustainability of organisational change in the National Health Service: a comparative case study of clinically led organisational innovations www.nets.nihr.ac.uk/projects/hsdr/09100140 |
NHS England | Qualitative comparative case study |
| McCourt C75 | R | 10/1008/35: an ethnographic organisational study of alongside midwifery units: a follow-on study from the Birthplace in England programme www.nets.nihr.ac.uk/projects/hsdr/10100835 |
NHS England | Ethnographic study |
| McDonald R76 | P | 08/1809/250: evaluation of the advancing quality pay for performance programme in the NHS North West www.nets.nihr.ac.uk/projects/hsdr/081809250 |
NHS England | Mixed methods (qualitative and quantitative) |
| Raine R77,78 | R, J | 09/2001/04: improving the effectiveness of multidisciplinary team meetings for patients with chronic diseases: A prospective cohort study www.nets.nihr.ac.uk/projects/hsdr/09200104 |
NHS England | Mixed methods (qualitative and quantitative) |
| Randell R79 | P | 12/5005/04: a realist process evaluation of robotic surgery: integration into routine practice and impacts on communication, collaboration and decision making www.nets.nihr.ac.uk/projects/hsdr/12500504 |
NHS Trusts, England | Realist evaluation |
| Rycroft-Malone J80 | P | 12/64/187: accessibility and implementation in UK services of an effective depression relapse prevention programme: Mindfulness based cognitive therapy www.nets.nihr.ac.uk/projects/hsdr/1264187 |
NHS UK | In-depth case studies using exploratory and interpretive methods |
| Waring J81,82 | R, J | 10/1007/01: knowledge sharing across the boundaries between care processes, services and organisations: the contributions to safe hospital discharge and reduced emergency readmission www.nets.nihr.ac.uk/projects/hsdr/10100701 |
NHS England | Ethnographic study |
The list of exemplar case studies contains a number of completed and ongoing projects. Publications included protocols, final reports and journal articles. Most case studies were conducted in England, with one being conducted in all four countries of the UK. 80 All were conducted in the NHS.
After a thorough reading of the publications relating to case studies, no new items were identified to add to the Delphi consensus process.
Chapter 4 Delphi consensus process
Items identified from the literature
After deduplication, a total of 103 unique items were identified for inclusion in the Delphi consensus process. See Appendix 2 for the full list of items. During the extraction process, the classification of items evolved and expanded from four to the six following categories:
-
describing the design
-
background, context and theory
-
describing the data collection
-
describing the data analysis
-
interpreting the results
-
sharing the results and conclusions.
These categories were used to structure the questionnaire, though respondents were given the opportunity to suggest additions or changes to this classification (see Appendix 3).
Round 1 results
Response rate and participants
Of 36 experts invited to take part in the Delphi consensus process, 19 (53%) responded to the first round invitation. All respondents completed the entire questionnaire.
Following the distribution of questionnaires, the funder of this project was contacted by a learned society for social science researchers, which expressed concerns about perceived assumptions underlying the project. The three main concerns raised were (1) the difficulty in mandating standards of conduct for the wide variety of case study approaches, (2) the existence of a quality control system already operating through peer review of the HSDR programme-funded project reports and (3) the risk of moving towards excessive standardisation. Four of the experts invited to participate in the Delphi consensus process co-signed the letter, three of whom also went on to complete both rounds of the survey. The comments from these authors, as well as the concerns raised in the letter, were used to inform and refine the structure of the second round questionnaire, and are discussed further in Round 2 results.
The characteristics of respondents to both rounds of the process are given in Table 4, and their research interests are described in Box 2. The majority of respondents in round 1 were researchers (80%), with substantial experience of authoring or otherwise contributing to organisational case study research (see Table 4). Two respondents classified themselves as research methodologists, two others classified themselves as having an editorial or related publishing role, and one respondent was a research funder. Several respondents expressed research interests related to health and/or social care, and others an interest in different approaches to organisational case study research (e.g. ethnography, qualitative case studies, comparative and theory-related cases).
| Characteristic | Number of respondents | |
|---|---|---|
| Round 1 | Round 2 | |
| Professional role | ||
| Researcher | 15 | 1 |
| Research methodologist | 2 | 1 |
| Journal editor/board member/publishing | 1 | 1 |
| Other (‘researcher and journal editor’) | 1 | 1 |
| How many organisational case studies have you authored? | ||
| 0 | 2 | 1 |
| 1–5 | 6 | 5 |
| 6–10 | 5 | 2 |
| > 10 | 6 | 7 |
| How many organisational case studies have you been involved with other than as an author? (e.g. peer review, commissioning, advisory role) | ||
| 0 | 0 | 0 |
| 1–5 | 6 | 3 |
| 6–10 | 3 | 3 |
| > 10 | 10 | 8 |
| What proportion of your work relates to research methodology? | ||
| 0 | 0 | 0 |
| 1–40% | 13 | 10 |
| 41–60% | 4 | 2 |
| > 60% | 2 | 3 |
-
Health and social care.
-
Evaluation of health IT.
-
Qualitative case studies.
-
Development of different models of service delivery and interface between primary and secondary care.
-
Organisation of care in hospital wards.
-
Use of information technology in health-care settings.
-
Comparative cases; theory-related cases.
-
Evidence implementation; quality improvement
-
Public services broadly – health, children’s services, urban regeneration, disability services.
-
Health care.
-
Change; implementation of evidence; maternity care; user and professional experiences; ethnography.
-
Health services.
-
Ethnography.
-
Health-care organisation.
-
Research funder judging quality of organisational case study research. Also editor for NIHR Journals Library reviewing quality.
-
Research into health policy.
-
Hospitals.
-
Quality improvement and change management.
-
Organisational change.
-
Realist evaluation, qualitative methods.
-
Acute hospital settings.
-
Relationships between organisational structures and policy outcomes.
Rating of items
Respondents were asked to rate absolutely necessary items for reporting case studies as ‘essential’, to rate useful but non-essential items as ‘desirable’ and rate any unnecessary, unclear, redundant, or meaningless items as ‘not necessary’. None of the 103 items was definitively excluded by consensus (i.e. the proportion of ‘not necessary’ ratings was below 70% for every item).
Table 5 shows the 10 items that met the predefined minimum 70% ‘essential’ consensus level. The level of consensus for these items ranged from 74% to 95%, with the highest consensus for ‘describe how the data were collected’ and ‘describe the sources of evidence used’. None of the items classified under the headings of ‘background, context and theory’ or ‘sharing the results and conclusions’ met the 70% ‘essential’ consensus threshold.
| Item | Rating, n (%) | ||
|---|---|---|---|
| Essential | Desirable | Not necessary | |
| Describing the design | |||
| Define the research as a case study | 14 (73.7) | 5 (26.3) | 0 |
| State the broad aims of the study | 16 (84.2) | 3 (15.8) | 0 |
| State the research question(s)/hypotheses | 15 (78.9) | 3 (15.8) | 1 (5.3) |
| Data collection | |||
| Describe how data were collected | 18 (94.7) | 1 (5.3) | 0 |
| Describe the sources of evidence used, for example documentation; archival records; interviews; direct observations; participant-observation; physical artefacts | 18 (94.7) | 1 (5.3) | 0 |
| Describe any ethical considerations and obtainment of relevant/approvals, access and permissions | 15 (78.9) | 4 (21.1) | 0 |
| Data analysis | |||
| Describe the analysis methods | 17 (89.5) | 2 (10.5) | 0 |
| Interpretation | |||
| Describe any inherent shortcomings in the design and analysis and how these might have influenced the findings | 15 (78.9) | 3 (15.8) | 1 (5.3) |
| Consider the appropriateness of methods used for the question and subject matter and why it was that qualitative methods were appropriate | 15 (78.9) | 4 (21.1) | 0 |
| Discuss the data analysis (was it conducted in a systematic way and was it successful in incorporating all observations and dealing with variation?) | 14 (73.7) | 4 (21.1) | 1 |
Among items failing to meet the 70% ‘essential’ threshold, values ranged from 0% to 68%. In order to better inform respondents and to facilitate the rating process in round 2, these 93 non-consensus items were ranked according to their positive/negative ratio of ratings from round 1. This ratio was calculated for each item by dividing the sum of ‘essential’ and ‘desirable’ counts by ‘not necessary’ counts. Consequently, a ratio of 1 would indicate an even balance of positive and negative ratings. Where ‘not necessary’ counts were 0, a value of 0.5 was used to allow the calculation. Ratio values for the non-consensus subsequently items ranged from 0.58 to 38.
Round 1 comments
A number of themes emerged from the comments given by respondents to round 1 (see Appendix 5 for all comments).
Several comments raised concerns about the phrasing of items. These fell into two categories: the inability to label items because they were unclear, inappropriate or poorly worded; and the impression that some items were overly focused on quantitative research and/or were informed by a rigid and predominantly positivist paradigm.
Several other comments explicitly noted that the appropriateness of certain items would be context specific, and so a single rating could not be universally applied across different approaches.
Other comments objected to the very notion of producing standards for the kind of contextualised and creative interpretative processes that are often employed in qualitative research.
None of the respondents suggested any changes to the six item categories.
Round 2 results
Development of the questionnaire
The responses from the first round were used to refine and develop both the introductory information and the restructuring of the items in the next questionnaire, which was distributed in the second round of the Delphi consensus process.
Introduction to round 2
The round 2 questionnaire was prefaced with an introduction that directly addressed the main concerns raised by respondents to the first round.
First, it was clarified that while the funders originally proposed ‘a common quality and reporting standard for organisational case study research’, the research team had anticipated that generic standards for the conduct of organisational case studies would not be possible, and so chose from the start to focus on quality of reporting rather than scientific quality more broadly (i.e. to identify any aspects of case study reporting that could facilitate the reading and judgement processes used by peer reviewers and other audiences). However, in light of the letter received by the society of social science researchers and associated comments from round 1, respondents to round 2 were given the opportunity to explicitly state whether they considered such reporting standards to be feasible or desirable.
It was also clarified that the items presented in the Delphi consensus process were not created by the research team but were derived from the published academic literature, using the authors’ own wording wherever possible. Thus, the language used and the paradigmatic assumptions related to each item likely reflect the position of the original academic author. For example, the contentious application of terms such as ‘validity’ and ‘reliability’ to case study research came directly from the published work of Yin. 8,34,35,54–56
The introduction to the exercise also emphasised the research team’s impartiality regarding the final content of the reporting standards, along with the respondents’ prerogative to exclude any items that they considered inappropriate, confusing, poorly worded or meaningless.
Item presentation
Items were again grouped into six categories: (1) describing the design; (2) background, context and theory; (3) describing the data collection; (4) describing the data analysis; (5) interpreting the results; and (6) sharing the results and conclusions.
Within each section, respondents were first asked to agree or disagree with the inclusion of the high-consensus items from round 1 (> 70% ‘essential’) in generic reporting standards. They were then asked to either upgrade or discard the remaining lower-consensus items, which were presented in decreasing order of the positive-/negative-rating ratio. For all items, respondents had the opportunity to distinguish between items that should be reported for organisational case studies in general, those that should be reported for a particular approach and those that did not need to be reported.
Response rate and participants
Fifteen respondents completed the entire round 2 questionnaire; 14 of these respondents (93%) had taken part in the first round and one respondent only contributed to the second round.
Although a slightly greater proportion of respondents thought the establishment of standards for reporting organisational case studies was desirable than did not, several were uncertain (see Table 6 for response rates and all related comments). Others suggested that the usefulness of any standards would depend upon how they were applied (e.g. as ‘a reference point for aspiration’ vs. a means of enforcing inappropriate standardisation) and where they are applied (e.g. health service research vs. sociology; impact on post-structuralist approaches).
| Question | No opinion | Yes | No | Do not know | Other | Comments |
|---|---|---|---|---|---|---|
| Did you take part in round 1 of this Delphi exercise? | 93.3% | 6.7% | ||||
| Do you think that a publication standard for reporting organisational case studies is desirable? | 0% | 40% | 26.7% | 13.3% | 20% | It depends on the audience or community. Advanced ethnographic case studies targeted at anthropology, cultural studies, sociology or policy studies are arguably distinct from Health Services Research or trial research communities. Also, how do post-structuralist or even narrative case accounts fit with the idea of standards? Standards might constrain creativity and imagination!All depends how it is used. It is one thing to have a standard that acts as a reference point or aspiration; it is another if this is used inappropriately to enforce standards that are not universally suitable for all research that might be subjected to itYes but . . . recognise heterogeneity of case study research |
| Do you think that a meaningful publication standard for reporting organisational case studies is possible? | 6.7% | 33.3% | 26.7% | 26.7% | 6.7% | For some types of studies and not others, I suspect |
Respondents were similarly divided about whether or not meaningful reporting standards would be feasible for organisational case studies. Again, the issue of standards being possible for some studies but not others was mentioned.
As might be expected, given the very high proportion of overlap between rounds, respondents had a similar level of case study experience and range of research interests as in round 1 (see Table 4).
Appendix 6 contains all the free-text comments provided in round 2.
Rating of items
Items considered relevant to all organisational case studies
Thirteen items were rated as ‘should be reported for all organisational case studies’ by at least 70% of respondents. These included all 10 high-consensus items from the first round, plus three further items: ‘identify the specific case(s) and justify the selection’, ‘ensure that the assertions are sound, neither over- nor under-interpreting the data’ and ‘state any caveats about the study’ (Table 7).
| Item | Number of respondents (percentage of total) | ||
|---|---|---|---|
| Should be reported for all | Should be reported for specific types | Does not need to be reported | |
| Describing the design | |||
| Define the research as a case study | 13 (86.7) | 0 | 2 (13.3) |
| State the broad aims of the study | 15 (100) | 0 | 0 |
| State the research question(s)/hypotheses | 13 (86.7) | 2 (13.3)a,b | 0 |
| Identify the specific case(s) and justify the selection [e.g. key case (good example, classic or exemplary case); outlier case (showing something interesting because it is different from the norm); local knowledge case (example chosen on the basis of personal experience or local availability)] | 11 (73.3) | 1 (6.7) | 3 (20) |
| Describing the data collection | |||
| Describe how data were collected | 15 (100) | 0 | 0 |
| Describe the sources of evidence used (e.g. documentation; archival records; interviews; direct observations; participant observation; physical artefacts) | 15 (100) | 0 | 0 |
| Describe any ethical considerations and obtainment of relevant approvals, access and permissions | 13 (86.7) | 1 (6.7)c | 1 (6.7) |
| Describing the data analysis | |||
| Describe the analysis methods | 15 (100) | 0 | 0 |
| Interpreting the results | |||
| Describe any inherent shortcomings in the design and analysis and how these might have influenced the findings | 13 (86.7) | 0 | 2 (13.3) |
| Consider the appropriateness of methods used for the question and subject matter and why it was that qualitative methods were appropriate | 13 (86.7) | 0 | 2 (13.3) |
| Discuss the data analysis (was it conducted in a systematic way and was it successful in incorporating all observations and dealing with variation) | 13 (86.7) | 0 | 2 (13.3) |
| Ensure that the assertions are sound, neither over- nor under-interpreting the data | 11 (73.3) | 0 | 4 (26.7) |
| State any caveats about the study | 11 (73.3) | 0 | 4 (26.7) |
In round 2, the degree of consensus ranged from 73% to 100%, with four items (‘state the broad aims of the study’, ‘describe how the data were collected’, ‘describe the sources of evidence used’, and ‘describe the analysis methods’) achieving 100% consensus. For all 13 items, the degree of consensus was greater than in round 1.
As in round 1, none of the items classified under the headings of ‘background, context and theory’ or ‘sharing the results and conclusions’ met the 70% consensus threshold.
Items considered unnecessary
In the second round, 36 items (35%) were classified as ‘does not need to be reported’ by at least 70% of respondents (see Appendix 7). The degree of consensus ranged from 73% to 93%. This emphasises the much higher level of consensus among respondents relative to that seen in the first round.
Items considered relevant to specific types of case study
Seventy-two items (70%) were considered by at least one respondent to be appropriate in certain contexts but not others. Methodological approaches identified by respondents included ‘quantitative’, ‘qualitative’, ‘positivist’, ‘realist evaluation’, ‘explanatory case studies’ and ‘participatory/action research’. Other types of case study identified included ‘NHS based’, ‘policy-sponsored research’ and ‘charity-funded evaluations’. Respondents very rarely expanded on these labels.
However, there was no consensus that any item should be considered relevant to a particular type of case study (where method-specific items were identified, agreement ranged from 0% to 33%).
Items with no overall consensus
Fifty-two items failed to meet the 70% consensus threshold for either inclusion or rejection (see Appendix 8).
Combining counts of ‘should be reported for all organisational case studies’ with counts of ‘should be reported for the following type of organisational case study . . .’ would result in just three additional items achieving a 70% ‘overall positive’ consensus [‘state whether an inductive or deductive approach to the analysis has been taken’, ‘discuss the sampling (or case selection) and explanation of sampling strategy’ and ‘describe the data collection tool(s) (e.g. questionnaire or observational protocol, including a description of any piloting or field testing of the tool)’ (Table 8)].
| Item | Should be reported for all | Should be reported for specific types | Does not need to be reported | Comments |
|---|---|---|---|---|
| State whether an inductive (e.g. grounded) or deductive (hypothesis testing/theoretical framework) approach to the analysis has been taken | 66.7% | 6.7% | 26.7% | It should be obvious |
| Discuss the sampling (or case selection) and explanation of sampling strategy | 66.7% | 6.7%a | 26.7% | Studies of heterogeneous populations of organisationsa |
| Describe the data collection tool(s) (e.g. questionnaire or observational protocol, including a description of any piloting or field testing of the tool) | 60% | 13.3%b,c | 26.7% | ’One that you want to publish in a positivist journalbWhen new or idiosyncratic data collection methods were usedc |
Chapter 5 Translating high-consensus Delphi items into reporting standards for organisational case studies
As an additional step in the development process, we applied the list of high-consensus items to all example and exemplar case study publications identified earlier in the review process, in order to (1) determine the relevance of these items to the reporting of real-world organisational case studies and (2) better understand how the results of the Delphi consensus process consultation might best be implemented as reporting standards. As stated in Chapter 2, Translating high-consensus Delphi items into reporting standards for organisational case studies, these were subjective assessments applied in retrospect by one reviewer and were used as practical examples to help with the development of the reporting standards and are not intended to be taken as a critical appraisal of the publications.
Example case studies
The high-consensus reporting items were applied to all 17 publications of the eight example organisational case studies (Table 9).
| Category | High-consensus items (see key) | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Design | Data collection | DA | Interpreting the results | ||||||||||||
| Case study | First author (publication type) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | |
| 1 | Attree57 | (J) | Y | Y | Y | N | Y | Y | Y | P | P | N | P | Y | Y |
| Cooke58 | (J) | Y | Y | N | P | Y | Y | Y | P | N | N | P | Y | N | |
| 2 | Field59 | (R) | Y | Y | Y | Y | Y | Y | Y | Y | P | Y | P | Y | N |
| Reid69 | (J) | Y | Y | N | P | Y | Y | Y | Y | N | N | P | Y | N | |
| Reid68 | (J) | Y | Y | N | N | Y | Y | Y | Y | N | N | P | Y | N | |
| 3 | Hutchinson61 | (J) | Y | Y | Y | Y | Y | Y | Y | P | N | N | P | Y | N |
| 4 | Kyratsis62 | (R) | Y | Y | Y | Y | Y | Y | N | Y | N | N | P | Y | Y |
| Kyratsis63 | (J) | Y | Y | Y | Y | Y | Y | N | Y | Y | Y | Y | Y | Y | |
| 5 | National Nursing Research Unit64 | (R) | N | Y | P | Y | Y | Y | N/A | P | P | N | U | U | N |
| Robert71 | (J) | N | Y | N | Y | Y | Y | N | Y | N | N | Y | Y | Y | |
| 6 | Offender Health Research Network73 | (R) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| Ricketts70 | (J) | Y | Y | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | |
| 7 | Payne65 (methods paper) | (J) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| Payne67 | (R) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | |
| Payne66 | (J) | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y | Y | Y | |
| Hawker60 | (J) | Y | N | N | Y | Y | Y | Y | Y | P | N | P | Y | P | |
| 8 | Rolls72 | (J) | Y | Y | N | Y | Y | Y | P | Y | N | Y | U | U | N |
Five of the eight case studies published reports. 59,62,64,67,73 One report from the National Nursing Research Unit appeared to be aimed at end users and contained little methodological detail. 64 All the case studies had at least one journal publication. One journal article65 provided some methodological detail for three of the included case studies and their publications. 59,60,66–69,72
Two linked publications exploring patient safety in the English pre-registration degree nursing curriculum met some of the criteria for describing the design,57,58 though one did not state the research questions/hypotheses (item 3)58 and neither fully identified the specific cases and justified selection (item 4). Both fully described data collection (items 5–7), but only partially described the analysis methods (item 8). Both poorly reported items related to interpreting the results (items 9–13).
A similar pattern of reporting was found in four linked publications evaluating adult hospice bereavement support services, which included a report,59 two journal articles68,69 and a stand-alone paper which reflected on the methods employed in this and other case studies. 65 The journal articles make reference to the full report, but the number of items reported was similar across all the publications. However, the journal articles do not state the research questions/hypotheses (item 3) and either did not report or only partially reported how the specific cases were identified and the selection was justified. Across these publications, there was generally poor reporting on the items relating to interpreting the results (items 9–13). The overarching methodological paper by Payne et al. was published after the three case studies were published and so was not referenced in the report or journal articles. 65
A case study to examine managing ward managers for roles in human resource management generated only one publication;61 this satisfied all the items for describing the design (items 1–4), collecting data (items 5–7) and describing the analysis (item 8). However, items for interpreting the results were poorly reported (items 9–13).
A study to understand the impact of differing organisational capacity and contextual circumstances on technology selection, procurement and implementation included a report62 and a journal article. 63 The items on describing the design (items 1–4) were all reported. Two questions related to describing the data collection were reported (items 5 and 6), but the item relating to ethical considerations (item 7) was not reported for either publication. The journal article63 reported all items for interpreting the analysis, but the report did not. 62
A study looking at issues of local implementation of ‘The Productive Ward’ programme included a report and a journal article. 64,71 The report appeared to be aimed at end users. Three items were reported or partially reported for design (items 2 to 4), two for data collection (items 5 and 6), one partially reported for data analysis (item 6), and one partially reported for interpretation of results (item 9). 64 The journal article reported more of the high-consensus Delphi items. Two items were reported in design (items 2–4), two in data collection (items 5 and 6), one in data analysis (item 9) and three for interpreting the results (items 11–13). 71
One report73 of a case study evaluating prison mental health in-reach services reported all high-consensus Delphi items. The associated journal article70 reported 12 of the 13 criteria, but did not report research questions/hypotheses (item 3).
A case study used to identify how palliative care for elderly people is delivered was published as a report67 and two journal articles,60,66 as well as being linked to the methods paper mentioned earlier. 65 The report reported all high-consensus Delphi items. 67 Items describing the design (items 1–4) were well covered in one article,66 but the other article60 reported neither the broad aims of the study (item 2) nor the research questions/hypotheses (item 3). Reporting of the data analysis and interpreting the results was generally well covered in one article66 but less so in the other publication. 60 Both the report and journal articles were published before the Payne et al. methodological paper65 and therefore did not explicitly reference it.
A case study examining the context and processes of childhood bereavement services72 was also linked to the above-mentioned methods paper. 65 This journal article did not state the research questions or hypotheses (item 3) and only partially reported on ethical considerations (item 7). Items relating to interpreting the data were also poorly reported. The journal article was published before the Payne et al. methodological paper65 and therefore did not explicitly reference it.
In summary, two publications reported less than 50% of the items,58,64 eight reported between 50% and 70% of items,57,60–62,68,69,71,72 four reported over 70% of items,59,63,70,66 and three reported all the high-consensus items. 65,67,73 Across all publications, the items describing the design (items 1–4), data collection (items 5–7) and the analysis (item 8) were largely reported. There was variation in reporting on interpretation of results and several studies either did not report or only partially reported their methods.
There was no clear pattern in the number of items being reported between journal articles and reports. This was a relatively small sample of publications aimed at different audiences, so it would not be appropriate to draw conclusions on levels of reporting within different types of publications.
These publications covered a range of case study methodology and were aimed at different audiences (e.g. end users); therefore, the lack of reporting should not be taken to mean a lack of quality in the methods used, nor as implied criticism of the original authors.
Exemplar case studies
Of the 12 exemplars, only seven had published reports. 2,3,7,74,75,77,81 Of these, two had a single additional journal article and one had two related journal articles. 5,6,78,81 The 13 high-consensus Delphi items of reporting standards were applied to each of the 11 publications (Table 10).
| Category | High-consensus items (see key) | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Design | Data collection | DA | Interpreting the results | ||||||||||||
| Case Study | First author (publication type) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | |
| 1 | Checkland2 | (R) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 2 | Drennan74 | (R) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 3 | Gillard3 | (R) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 4 | Martin7 | (R) | Y | Y | Y | Y | Y | Y | N | Y | P | P | Y | Y | N |
| Martin6 | (J) | Y | Y | N | Y | Y | Y | Y | Y | N | N | Y | Y | N | |
| 5 | Currie5 | (J) | Y | Y | N | Y | Y | Y | N | Y | P | P | Y | Y | P |
| 6 | McCourt75 | (R) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| 7 | Raine77 | (R) | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| Raine78 | (J) | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | |
| 8 | Waring82 | (R) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| Waring81 | (J) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | |
Six out of 11 of the publications reported all 13 of the high-consensus Delphi items. 2,3,74,75,81,82
Three publications for one case study did not fully meet all the items. 5–7 One report, which included data on follow-up to previous case study sites, did not explicitly state that the authors had ethical approvals, access or permissions (item 7). 7 The authors appeared to partially describe shortcomings in the design and analysis and how these might have influenced the findings (item 9). They only partially considered the appropriateness of methods used (item 10) and did not state any caveats about the study (item 13).
The two journal articles associated with the report also did not meet all the items. 5,6 One article, which was partly linked with the main HSDR programme-funded project detailed in the report, did not state the research question/hypotheses (item 3), nor did it describe any shortcomings in the design (item 9), consider the appropriateness of methods used (item 10) or state any caveats about the study (item 13). 6 The other article, published prior to the report, also did not state the research question/hypotheses (item 3). 5 There was no reporting of ethical considerations (item 7). There was only partial reporting of any inherent shortcomings in the design and analysis and how these might influence findings (item 9), consideration of the appropriateness of methods used (item 10) and caveats about the study (item 13).
A report77 and a linked journal article78 reported 12 out of 13 of items, but did not define the research as a case study (item 1). The authors of this study stated it was a prospective observational study but, because it contained many elements of an organisational case study, it was recommended by the HSDR programme for this project.
In summary, one exemplar publication5 reported 61% of the reporting standards, two6,7 reported 69% and six2,3,74,75,81,82 reported all of the reporting standards. The journal articles largely reported the same criteria as the corresponding reports. As with the example case studies above, these publications covered a range of case study methodology and were aimed at different audiences; therefore, the lack of reporting should not be taken as an indication of a lack of quality in the methods used in the studies themselves, nor as implied criticism of the original authors.
As a whole, the exemplar case studies (which had been considered methodologically strong by the HSDR programme team) more consistently reported the high-consensus Delphi items than did the example case studies drawn from the review searches. Of 11 exemplar publications, six (55%) reported all 13 items, compared with just 3 out of 17 (18%) of the example organisational case study publications.
Generic consensus-based reporting standards for organisational case studies
The exemplar organisational case studies identified by the HSDR programme as being of high quality were far more consistent with the high-consensus Delphi items than were a group of example case studies identified purely on the basis of topic relevance. If the latter group of studies are representative of the wider field of organisational case study research, then there is clearly scope to use the identified items to improve the improve the consistency and rigour of reporting in this area.
Though the high-quality case studies used different methodological approaches, they were consistent with one another on the high-consensus Delphi reporting items. This suggests that, although these items can detect consistency and rigour of reporting, they are also sufficiently generic to be applied to a variety of organisational case study methods.
The fact that journal articles sometimes satisfied more items than longer reports for the same case study suggests that the length of a publication is not necessarily related to how clearly the research methods are reported. This may be a deliberate choice. For example, authors may choose to exclude certain items from a report aimed at practitioners or policy-makers, yet include those same items in an academic journal article aimed at other researchers.
Similarly, there may be legitimate methodological reasons for a particular item not being reported. For example, a researcher conducting a purely exploratory case study might not consider it appropriate to state an initial research question or hypothesis (item 3 on the reporting standards); in this case it would be perfectly legitimate to briefly outline the justification for not doing so in the report.
However, it is not always obvious whether the absence of certain information is deliberate or an oversight; any reporting standards for organisational case studies should be aware of this distinction. Therefore, unlike reporting standards such as PRISMA, which mandate the inclusion of every item in a report,14 the reporting standards proposed in Table 11 require the author to refer to a place where the reporting item was reported or where justification for the absence of the item can be found. This approach intends to balance the research freedoms of the knowledgeable researcher with the information needs of the end user.
| Reporting item | Page number on which item was reported | Page number of justification for not reporting |
|---|---|---|
| Describing the design | ||
| 1. Define the research as a case study | ||
| 2. State the broad aims of the study | ||
| 3. State the research question(s)/hypotheses | ||
| 4. Identify the specific case(s) and justify the selection | ||
| Describing the data collection | ||
| 5. Describe how data were collected | ||
| 6. Describe the sources of evidence used | ||
| 7. Describe any ethical considerations and obtainment of relevant approvals, access and permissions | ||
| Describing the data analysis | ||
| 8. Describe the analysis methods | ||
| Interpreting the results | ||
| 9. Describe any inherent shortcomings in the design and analysis and how these might have influenced the findings | ||
| 10. Consider the appropriateness of methods used for the question and subject matter and why it was that qualitative methods were appropriate | ||
| 11. Discuss the data analysis | ||
| 12. Ensure that the assertions are sound, neither over- nor under-interpreting the data | ||
| 13. State any caveats about the study | ||
Chapter 6 Discussion
The final consensus-based reporting standards consist of 13 unique items grouped into four sections (see Table 11):
-
describing the design
-
describing the data collection
-
describing the data analysis
-
interpreting the results.
These standards aim to improve the consistency, rigour and reporting of organisational case study research, thereby making it more accessible and useful to different audiences. These audiences include research sponsors who need to make decisions about whether or not to fund proposed case studies; ethics and research advisory groups who require clarity about the specific planned methods; peer-reviewers who need to be able to evaluate the robustness of a completed case study; and readers and policy-makers who need to understand how the findings of an organisational case study might be interpreted and implemented.
Though several items in the reporting standards refer to the conduct of case study research, the standards are not intended to be a guide on how to undertake an organisational case study. There are multiple texts that address methodology in this area at great length, many of which informed the initial stage of this project. 8–10,34–36,38–56 Any checklist mandating specific case study methods would be far more lengthy than the proposed reporting standards, would be difficult to implement universally across different research contexts and paradigms and would likely encounter resistance from some sections of the research community.
Nevertheless, it can be seen that the majority of specific concerns raised by the HSDR programme when commissioning this work (e.g. absence of clear research questions, lack of clarity about how cases were selected, lack of clear analysis plans, absence of information about data sources) are directly addressed in the final set of reporting standards. Perhaps surprisingly, none of the items that might have addressed a concern about case studies having an ‘insufficient theoretical basis’ reached sufficiently high consensus to be included in the final set of standards. We might have expected a greater proportion of respondents to consider the theoretical or methodological underpinnings of the research as vital to understanding how researchers interpret their results. But in fact, none of the items classified as ‘background, context and theory’ met the high-consensus threshold. It could be that (a) the group of experts consulted truly did not consider the items presented in this category to be sufficiently important, (b) the experts believed that this concern would be addressed by another item (e.g. ‘ensure that the assertions are sound, neither over- nor under-interpreting the data’) or (c) this is an artefact of the Delphi process.
As there was high consensus among experts that the 13 items included in the reporting standards should be reported for all organisational case studies, these items tend towards the generic rather than the specific. In fact, rather than being exclusive to organisational case studies, several of the items reflect good practice for the reporting of research more generally, and are similar to items on reporting standards for other research designs. 12–21 This raised a concern that the reporting standards may be so generic as to be of little value for the intended aim of improving the consistency and rigour of reporting of organisational case study research. However, applying the standards to both exemplar and example case studies suggested that (a) there are published organisational case studies that do not meet these standards, (b) such studies could be reported in a manner that meets the standards without impacting on their underlying methodology and (c) exemplar organisational case studies identified by HSDR as being of high quality are generally consistent with these standards, whereas the variation in standards is much greater among the example organisational case studies that we reviewed (see Chapter 5). Consequently, we believe that even this short list of relatively generic reporting items does have the potential to improve the standard of reporting among organisational case studies conducted in the NHS.
Several experts expressed concerns about the risk of using a checklist, particularly if this were used in an attempt to standardise organisational case study research methods. Given the generic nature of many of the items included in the reporting standards, we believe that the risk of it being used to constrain research methodology is minimal. In addition, the final reporting standards are structured in such a way as to emphasise their primary intention as a means of supporting authors to report their research proposals and manuscripts, rather than as a tool for standardisation of methods. Consequently, for each item, the reporting standards provide the research author with the opportunity to acknowledge where the item has been reported, or to explain why the item has (legitimately) not been reported. In either case, the authors will have met the requirements of the reporting standards by providing the end user with important information about the design and purpose of the study. The standards are not prescriptive: authors are allowed to exercise judgement about how much information they choose to provide. As seen in Chapter 5, these reporting standards can be met in even relatively brief publications.
Several reporting items included in the Delphi consensus process were considered ‘essential’ by a majority of respondents but, failing to meet the pre-defined consensus threshold, they have not been included in the standards presented here. However, there may be an opportunity to expand and/or refine the current reporting standards further, possibly after it has been applied by authors of newly conducted organisational case studies.
Strengths of this project
Unlike a traditional Delphi consensus process, in which items are generated by respondents and then refined in subsequent rounds by the same respondents, we expedited the process by deriving an initial pool of items from a rapid review of the methodological literature relevant to organisational case studies. These items were then rated in two rounds by a Delphi panel of experts, all of whom had direct involvement with case study research. This approach aimed to ensure that both the generation and refinement stages were informed by expert knowledge within the short time frame available for the project. The research team made concerted efforts to avoid influencing the content or outputs of the review and consultation processes, and the processes themselves have been reported as clearly as possible to maximise transparency and avoid bias.
Alongside the review of methodological literature, we also examined two groups of case studies: high-quality exemplar studies identified as such by the NIHR’s HSDR programme and a group of topic-relevant example studies obtained from the wider literature. These ensured that the project was informed by real-world research practices and also provided an opportunity to check the validity of high-consensus Delphi items for inclusion in the reporting standards. Evidence showing a discrepancy between exemplar and example case studies in terms of performance against these items implies that there is scope for greater rigour and consistency of reporting in this area. While the best organisational case studies (such as the exemplars) would not be much improved by these reporting standards, many other studies clearly would be improved.
Some Delphi respondents expressed concerns that the proposed reporting standards might constrain methodological freedom, particularly for researchers using qualitative or interpretive methods. Ultimately, only the most generic items met the minimum consensus threshold. Some of the items are not even specific to the reporting of case studies, but to good reporting practice for empirical research more broadly. Consequently, we believe that the reporting standards are sufficiently broad to encompass the various different approaches and paradigms that fall under the umbrella of organisational case study research. Evidence from the exemplars suggested that a well-reported case study is likely to be consistent with the proposed reporting standards, regardless of the specific research methods or underlying epistemological paradigm. Should an item from the reporting standards be truly inappropriate to the specific case study method, authors are given the opportunity to make this clear (with appropriate methodological justification). This approach aims to balance the needs of readers concerned about methodological quality with the methodological freedom of authors by placing a strong emphasis on transparency.
Consensus among the items included in the reporting standards was generally high, with 100% of respondents in round 2 agreeing that several items ‘should be reported for every organisational case study’. If the Delphi respondents are representative of knowledgeable case study researchers more broadly, we would expect these standards to be acceptable to the wider research community.
Weaknesses of this project
As stated earlier, the research team attempted wherever possible to avoid introducing bias or personal preferences into the review and consultation processes. For example, we intentionally avoided excluding or substantially rewording items identified in the methods literature wherever possible, leaving any decisions about the value of these items to the Delphi panel. However, we were aware that this meant that some of the items could have been worded more clearly or precisely. Although we were keen not to risk changing the original authors’ meaning during the research process, any future piloting of the reporting standards could provide an opportunity to refine the exact wording of the included items.
The total number of respondents was relatively small, though this is frequently the case in Delphi research studies. As the primary aim is to identify the level of consensus among experts, rather than generalising to a larger population, obtaining a sufficient degree of expertise and representative panel can be considered more important than obtaining a large ‘sample’. Data collected in this Delphi suggested that respondents had an appropriate level of expertise and held a range of views regarding the development of reporting standards, from the enthusiastic to the sceptical.
A minority of Delphi respondents suggested that the material presented in the Delphi had a strongly positivist focus. This impression may have been a consequence of the data extraction process, in which discrete practical items related to reporting were extracted from the methods literature. This meant that authors like Yin (who writes in a predominantly didactic–pragmatic style) were more strongly represented among the initial pool of items than were authors that focused on higher-level abstractions or theoretical issues (e.g. Hammersley, Gomm, Flyvbjerg). The perception of ‘positivist’ bias related mainly to items derived from Yin, but it should be noted that these items typically failed to meet the required consensus threshold, and so were not included in the final reporting standards. Although the reporting standards might not reflect the terminology used in fields such as anthropology or geography, they are likely to be appropriate to the types of organisational case study most frequently funded by the HSDR programme. An alternative approach to the identification of items might have been to include an earlier item generation round in the Delphi consensus process, but time and resource constraints prevented this from being possible.
Some reporting items considered necessary by a strict majority of respondents (i.e. > 50% agreement) were not included in the final reporting standards. However, a minimum 70% threshold was intentionally chosen as this gives a greater than 2 : 1 ratio of agreement to dissent, which more accurately reflects high consensus among the Delphi panel, particularly with a sample size such that an individual respondent score constitutes almost 7% of the total. Ultimately, most of items in the reporting standards far exceeded the 70% threshold. Just three of the subthreshold items achieved an ‘overall positive’ consensus by combining ‘report for specific case studies’ ratings with ‘report for all case studies’ ratings (‘state whether an inductive or deductive approach to the analysis has been taken’, ‘discuss the sampling (or case selection) and explanation of sampling strategy’ and ‘describe the data collection tool(s)’). If the reporting standards were to be expanded beyond the 13 current items, based on the available data these would be the most likely candidates for addition.
There is always a risk that easily applied checklists can be improperly used as a substitute for proper methodological understanding by less experienced researchers. The reporting standards presented here are intended to improve the transparency of reporting of organisational case studies and have been presented in a format to ensure that they are used for this purpose. Although awareness of these standards may improve the conduct of organisational case studies, they currently outline the minimum requirements for reporting and should not be considered a simple checklist for establishing methodological quality. Meeting all 13 standards will not guarantee that an organisational case study has been well conducted, but it should provide readers with a sufficient understanding of how the case study was undertaken.
Chapter 7 Conclusions
Key conclusions from the project
The reporting standards presented here are intended primarily as a tool for authors of organisational case studies. They briefly outline broad requirements for rigorous and consistent reporting, without constraining methodological freedom. They are not intended to be used as a critical appraisal tool but, by improving reporting quality, hopefully they will prove useful for research funders, peer reviewers, journal editors and readers. Currently, it appears that not all organisational case studies report the items listed in the reporting standards but, if implemented properly in future, these standards should facilitate peer review of organisational case studies and give greater confidence to readers.
In general, the proposed standards simply require authors to acknowledge key stages of the research process. Applying the standards is unlikely to be onerous, nor to result in a change of methods or a great deal of extended detail in study reports.
Implications for research
Although the proposed reporting standards are based on a high level of consensus and have face validity, their true value cannot be fully established until they have been applied in practice.
In the first instance, we propose that these reporting standards be included as part of the submission requirements for all organisational case studies seeking public funding. Although the full set of standards can only be met once the study has results to report, it might also be useful to make items 1 to 8 of the standards available to authors at the proposal stage. It could be emphasised that clear reporting will be of benefit to reviewers and readers (and ultimately to the authors themselves).
Though these reporting standards do not mandate specific methods, if a reporting item is not reported for legitimate methodological reasons, the onus is on the author to outline their rationale for the reader.
Final report manuscripts should be accompanied by a version of the reporting standard pro forma completed by the study author(s), and both documents should be made available to peer reviewers. Funding boards may want to collect feedback from users (including commissioners, authors and peer reviewers) about the implementation of these standards. They may be more straightforward to implement for some forms of organisational case studies than others (e.g. stand-alone organisational case studies vs. case studies embedded within a larger study design). As mentioned previously, we anticipate that some authors may feel that reporting standards are not relevant or necessary for organisational case studies and that others may consider the standards proposed here to be too generic. However, consultation with research authors will be necessary to build engagement with the concept of reporting standards for organisational case studies among various audiences, and to collect evidence that could be used to evaluate and/or further refine the existing standards.
Implications for practice
We will submit the proposed reporting standards to the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network for consideration. This is an international initiative that seeks to improve the reliability and value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines. Its library for health research reporting (www.equator-network.org/library/) contains 247 reporting guidelines, of which 60 relate to observational research methods. Currently, none of these are explicitly concerned with the reporting or organisational case studies.
Acknowledgements
Contributions of authors
Mark Rodgers and Sian Thomas carried out study selection, data extraction, critical appraisal and write up of the report.
All searching was conducted by Melissa Harden, who also wrote the search sections of the report and commented on the draft report.
Gillian Parker and Andrew Street provided expertise and advice, contributed to the development of the protocol and commented on drafts of the report.
Alison Eastwood carried out study selection, data extraction, critical appraisal and write up of the report.
Advice
James Thomas, of the University College London Institute of Education, provided expertise, advice and assistance with text mining. The review authors are grateful to the Delphi panel of experts whose contribution made this work possible.
Data archiving
All available data can be obtained from the corresponding author. All data will be shared in a way that safeguards the confidentiality and anonymity of respondents.
Data sharing statement
All available data can be obtained from the corresponding author. All data will be shared in a way that safeguards the confidentiality and anonymity of respondents.
Disclaimers
This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.
References
- Dowding D. The Detection and Management of Pain in Patients with Dementia in Acute Care Settings (Protocol). Southampton: NIHR Health Services and Delivery Research programme; 2012.
- Checkland K, Snow S, McDermott I, Harrison S, Coleman A. Management Practice in Primary Care Organisations: The Roles and Behaviours of Middle Managers and GPs. Southampton: NIHR Service Delivery and Organisation programme; 2011.
- Gillard S, Edwards C, Gibson S, Holley J, Owen K. New ways of working in mental health services: a qualitative, comparative case study assessing and informing the emergence of new peer worker roles in mental health services in England. Health Serv Deliv Res 2014;2.
- Goodman C. Optimal NHS Service Delivery to Care Homes: A Realist Evaluation of the Features or Mechanisms that Support Effective Working for the Continuing Care of Older People in Residential Settings. Southampton: NIHR Health Services and Delivery Research; 2013.
- Currie G, Lockett A, Finn R, Martin G, Waring J. Institutional work to maintain professional power: recreating the model of medical professionalism. Organ Stud 2012;33:937-62. http://dx.doi.org/10.1177/0170840612445116.
- Martin GP, Weaver S, Currie G, Finn R, McDonald R. Innovation sustainability in challenging health-care contexts: embedding clinically led change in routine practice. Health Serv Manage Res 2012;25:190-9. http://dx.doi.org/10.1177/0951484812474246.
- Martin GP, Weaver S, Currie G, Finn R, McDonald R. The Medium-Term Sustainability of Organisational Change in the National Health Service: A Comparative Case Study of Clinically Led Organisational Innovations. Southampton: NIHR Service Delivery and Organisation programme; 2013.
- Yin RK. Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Publications; 2014.
- Gagnon YC. The Case Study as Research Method: A Practical Handbook. Québec City, QC: Presses de l’Université du Québec; 2010.
- Stake RE. The Art of Case Study Research. Thousand Oaks, CA: Sage Publications; 1995.
- Woodside AG, Wilson EJ. Case study research methods for theory building. J Bus Ind Mark 2003;18:493-508. http://dx.doi.org/10.1108/08858620310492374.
- Schulz K, Altman D, Moher D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Ann Intern Med 2010;152:726-32. http://dx.doi.org/10.7326/0003-4819-152-11-201006010-00232.
- von Elm E, Altman D, Egger M, Pocock S, Gotzsche P, Vandenbroucke J. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 2007;147:573-7. http://dx.doi.org/10.7326/0003-4819-147-8-200710160-00010.
- Moher D, Liberati A, Tetzlaff J, Altman D. The PRISMA Group . Reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLOS Med 2009;6. http://dx.doi.org/10.1371/journal.pmed.1000097.
- Gagnier J, Kienle G, Altman D, Moher D, Sox H, Riley D, et al. The CARE Guidelines: Consensus-based clinical case reporting guideline development. BMJ Case Rep 2013;7. http://dx.doi.org/10.1111/head.12246.
- O’Brien B, Harris I, Beckman T, Reed D, Cook D. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med 2014;89:1245-51. http://dx.doi.org/10.1097/ACM.0000000000000388.
- Wong G, Greenhalgh T, Buckingham J, Pawson R. RAMESES publication standards: realist synthesis. BMC Med 2013;11. http://dx.doi.org/10.1111/jan.12095.
- Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: meta-narrative reviews. BMC Med 2013;11. http://dx.doi.org/10.1111/jan.12092.
- Bossuyt P, Reitsma J, Bruns D, Gatsonis C, Glasziou P, Irwig L, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. BMJ 2003;326:41-4. http://dx.doi.org/10.1136/bmj.326.7379.41.
- Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S. Publication guidelines for quality improvement in health care: evolution of the SQUIRE project. Qual Saf Health Care 2008;17:i3-9. http://dx.doi.org/10.1136/qshc.2008.029066.
- Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Eur J Health Econ 2013;14:367-72. http://dx.doi.org/10.1007/s10198-013-0471-6.
- Byrne D, Byrne D, Ragin C. The SAGE Handbook of Case-Based Methods. London: SAGE Publications; 2009.
- Byrne D, Byrne D, Ragin C. The SAGE Handbook of Case-Based Methods. London: SAGE Publications; 2009.
- Byrne DS. Complexity, configuration and cases. Theor Cult Soc 2005;22:95-111. http://dx.doi.org/10.1177/0263276405057194.
- Flyvbjerg B, Landman T, Schram S, Flyvbjerg B, Landman T, Schram S. Real Social Science: Applied Phronesis. Cambridge: Cambridge University Press; 2012.
- Flyvbjerg B, Landman T, Schram S. Real Social Science: Applied Phronesis. Cambridge: Cambridge University Press; 2012.
- Flyvbjerg B, Denzin NK, Lincoln YS. The SAGE Handbook of Qualitative Research. Thousand Oaks, CA: SAGE Publications; 2011.
- Gomm R. Key Concepts in Social Research Methods. London: Palgrave Macmillan; 2009.
- Gomm R, Davies C. Using Evidence in Health and Social Care. London: SAGE Publications; 2000.
- Gomm R, Hammersley M, Foster P. Case Study Method – Key Issues, Key Texts. London: SAGE Publications; 2000.
- Ragin CC, Byrne D, Ragin CC. The SAGE Handbook of Case-based Methods. London: SAGE Publications; 2009.
- Ragin CC, Schneider G, Williams M, Vogt P. The SAGE Handbook of Methodological Innovations. London: SAGE Publications; 2010.
- Ragin CC, Schneider G, Amenta E, Nash K, Scott A. The New Blackwell Companion to Political Sociology. Chichester: Blackwell; 2012.
- Yin RK. Applications of Case Study Research. London: SAGE Publications; 2012.
- Yin RK, Cooper H. APA Handbook of Research Methods in Psychology, Vol 2: Research Designs: Quantitative, Qualitative, Neuropsychological, and Biological. Washington, DC: American Psychological Association; 2012.
- Stake RE, Denzin NK, Lincoln YS. The SAGE Handbook of Qualitative Research. Thousand Oaks, CA: SAGE Publications; 2005.
- Custer R, Scarcella J, Stewart R. The modified Delphi technique – a rotational modification. J Voc Tech Educ 1999;15.
- Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol 2011;11. http://dx.doi.org/10.1186/1471-2288-11-100.
- Darke P, Shanks G, Broadbent M. Successfully completing case study research: combining rigour, relevance and pragmatism. Inf Syst J 1998;8:273-89. http://dx.doi.org/10.1046/j.1365-2575.1998.00040.x.
- Fitzgerald L, Dopson S, Buchanan DA, Brynam A. The SAGE Handbook of Organizational Research Methods. London: SAGE Publications; 2009.
- Gibbert M, Ruigrok W. The ‘what’ and ‘how’ of case study rigor: three strategies based on published work. Organ Res Methods 2010;13:710-37. http://dx.doi.org/10.1177/1094428109351319.
- Gibbert M, Ruigrok W, Wicki B. What passes as a rigorous case study?. Strategic Manage J 2008;29:1465-74. http://dx.doi.org/10.1002/smj.722.
- Gilgun JF. A case for case-studies in social-work research. Soc Work 1994;39:371-80.
- Gillham B. Case Study Research Method (Continuum Research Methods). London: Bloomsbury Academic; 2000.
- Greene D, David JL. A research design for generalizing from multiple case studies. Eval Program Plann 1984;7:73-85. http://dx.doi.org/10.1016/0149-7189(84)90027-2.
- Hays PA, deMarrais K, Lapan SD. Foundations for Research: Methods of Inquiry in Education and the Social Sciences. Mahwah, NJ: Lawrence Erlbaum Associates Publishers; 2004.
- Hutchinson SA, Moody LE. Advancing Nursing Science Through Research (Vol 2). Newbury Park, CA: SAGE Publications; 1990.
- Huws U, Dahlmann S. Quality Standards for Case Studies in the European Foundation. Dublin: European Foundation for the Improvement of Living and Working Conditions; 2007.
- Kaarbo J, Beasley RK. A practical guide to the comparative case study method in political psychology. Polit Psychol 1999;20:369-91. http://dx.doi.org/10.1111/0162-895X.00149.
- Meyer CB. A case in case study methodology. Field Methods 2001;13:329-52. http://dx.doi.org/10.1177/1525822X0101300402.
- Moore TS, Lapan SD, Quartaroli MT, Laplan SD, Quartaroli MT, Riemer FJ. Qualitative Research: An Introduction to Methods and Designs. San Francisco, CA: Jossey-Bass; 2012.
- Stake RE, Denzin NK, Lincoln YS. Handbook of Qualitative Research. Thousand Oaks, CA: SAGE Publications; 1994.
- Thomas G. How to do your Case Study: A Guide for Students and Researchers. London: SAGE Publications; 2011.
- Yin RK, Bickman L, Rog DJ. Handbook of Applied Social Research Methods. Thousand Oaks, CA: SAGE Publications; 1998.
- Yin RK. Enhancing the quality of case studies in health services research. Health Serv Res 1999;34:1209-24.
- Yin RK, Green GL, Camilli G, Elmore PB. Handbook of Complementary Methods in Education Research. Mahwah, NJ: Lawrence Erlbaum Associates; 2006.
- Attree M, Cooke H, Wakefield A. Patient safety in an English pre-registration nursing curriculum. Nurse Educ Pract 2008;8:239-48. http://dx.doi.org/10.1016/j.nepr.2007.09.003.
- Cooke H. The surveillance of nursing standards: an organisational case study. Int J Nurs Stud 2006;43:975-84. http://dx.doi.org/10.1016/j.ijnurstu.2005.11.003.
- Field D, Reid D, Payne S, Relf M. Adult bereavement support in five hospices in England. Sheffield: University of Sheffield; 2005.
- Hawker S, Kerr C, Payne S, Seamark D, Davis C, Roberts H, et al. End-of-life care in community hospitals: the perceptions of bereaved family members. Palliat Med 2006;20:541-7. http://dx.doi.org/10.1191/0269216306pm1170oa.
- Hutchinson S, Purcell J. Managing ward managers for roles in HRM in the NHS: overworked and under-resourced. Hum Resour Manage J 2010;20:357-74. http://dx.doi.org/10.1111/j.1748-8583.2010.00141.x.
- Kyratsis Y, Ahmad R, Holmes A. Understanding the Process of Innovation Adoption in 12 NHS Trusts – Technology Selection, Procurement and Implementation to Help Reduce HCAIs. London: Centre for Infection Prevention and Management, Imperial College London; 2010.
- Kyratsis Y, Ahmad R, Holmes A. Technology adoption and implementation in organisations: comparative case studies of 12 English NHS Trusts. BMJ Open 2012;2. http://dx.doi.org/10.1136/bmjopen-2012-000872.
- The Productive Ward: Releasing Time to Care. Learning and Impact Review. Coventry: NHS Institute for Innovation and Improvement; 2009.
- Payne S, Field D, Rolls L, Hawker S, Kerr C. Case study research methods in end-of-life care: reflections on three studies. J Adv Nurs 2007;58:236-45. http://dx.doi.org/10.1111/j.1365-2648.2007.04215.x.
- Payne S, Hawker S, Kerr C, Seamark D, Roberts H, Jarrett N, et al. Experiences of end-of-life care in community hospitals. Health Soc Care Community 2007;15:494-501. http://dx.doi.org/10.1111/j.1365-2524.2007.00714.x.
- Payne S, Kerr C, Hawker S, Seamark D, Davis C, Roberts H, et al. The Provision of Palliative Care for Elderly People in Community Hospitals: Best Hope or Last Hope?. Sheffield: Palliative and End-of-Life Care Research Group, University of Sheffield; 2004.
- Reid D, Field D, Payne S, Relf M. Adult bereavement in five English hospices: types of support. Int J Palliat Nurs 2006;12:430-7. http://dx.doi.org/10.12968/ijpn.2006.12.9.21871.
- Reid D, Field D, Payne S, Relf M. Adult bereavement in five English hospices: participants, organisations and pre-bereavement support. Int J Palliat Nurs 2006;12:320-7. http://dx.doi.org/10.12968/ijpn.2006.12.7.21609.
- Ricketts T, Brooker C, Dent-Brown K. Mental health in-reach teams in English prisons: aims, processes and impacts. Int J Prison Health 2007;3:234-47. http://dx.doi.org/10.1080/17449200701682428.
- Robert G, Morrow E, Maben J, Griffiths P, Callard L. The adoption, local implementation and assimilation into routine nursing practice of a national quality improvement programme: the Productive Ward in England. J Clin Nurs 2011;20:1196-207. http://dx.doi.org/10.1111/j.1365-2702.2010.03480.x.
- Rolls L, Payne S. Childhood bereavement services: issues in UK service provision. Mortality 2004;9:300-28. http://dx.doi.org/10.1080/13576270412331329821.
- A National Evaluation of Prison Mental Health In-Reach Services. Manchester: The Offender Health Research Network; 2009.
- Drennan VM, Halter M, Brearley S, Carneiro W, Gabe J, Gage H, et al. Investigating the contribution of physician assistants to primary care in England: a mixed-methods study. Health Serv Deliv Res 2014;2.
- McCourt C, Rayment J, Rance S, Sandall J. An ethnographic organisational study of alongside midwifery units: a follow-on study from the Birthplace in England programme. Health Serv Deliv Res 2014;2.
- McDonald R. Evaluation of the Advancing Quality Pay for Performance Programme in the NHS North West. Southampton: NIHR Service Delivery and Organisation programme; 2008.
- Raine R, Wallace I, Nic a’ Bháird C, Xanthopoulou P, Lanceley A, Clarke A, et al. Improving the effectiveness of multidisciplinary team meetings for patients with chronic diseases: a prospective observational study. Health Serv Deliv Res 2014;2.
- Raine R, Xanthopoulou P, Wallace I, Nic A’ Bháird C, Lanceley A, Clarke A, et al. Determinants of treatment plan implementation in multidisciplinary team meetings for patients with chronic diseases: a mixed-methods study. BMJ Qual Saf 2014;23:867-76. http://dx.doi.org/10.1136/bmjqs-2014-002818.
- Randell R, Greenhalgh J, Hindmarsh J, Dowding D, Jayne D, Pearman A, et al. Integration of robotic surgery into routine practice and impacts on communication, collaboration, and decision making: a realist process evaluation protocol. Implement Sci 2014;9. http://dx.doi.org/10.1186/1748-5908-9-52.
- Rycroft-Malone J, Anderson R, Crane RS, Gibson A, Gradinger F, Griffiths HO, et al. Accessibility and implementation in UK services of an effective depression relapse prevention programme – mindfulness-based cognitive therapy (MBCT): ASPIRE study protocol. Implement Sci 2014;9. http://dx.doi.org/10.1186/1748-5908-9-62.
- Waring J, Marshall F, Bishop S. Understanding the occupational and organizational boundaries to safe hospital discharge. J Health Serv Res Policy 2015;20:35-44. http://dx.doi.org/10.1177/1355819614552512.
- Waring J, Marshall F, Bishop S, Sahota O, Walker M, Currie G, et al. An ethnographic study of knowledge sharing across the boundaries between care processes, services and organisations: the contributions to ‘safe’ hospital discharge. Health Serv Deliv Res 2014;2.
Appendix 1 Search strategies
Library catalogue searches
Health Services Management Centre ONLINE (University of Birmingham)
URL: https://cssfs8.bham.ac.uk/HeritageScripts/Hapi.dll/search1.
Searched on: 14 July 2014.
Records retrieved: 15.
Search strategy
case AND (method OR methods or methodology)
Health Management Online (NHS Scotland)
URL: www.shelcat.org/nhml.
Searched on: 14 July 2015.
Records retrieved: 47.
Search strategy
Words= case AND W-subjects= method or methods or methodology and W-type= BK not JA
The King’s Fund library database
URL: http://kingsfund.koha-ptfs.eu/.
Searched on: 14 July 2014.
Records retrieved: 17.
Advanced search
Subject: case studies AND subject: research methods – 10 records.
Subject: case studies AND subject: methodology – three records.
Subject: case studies AND subject: methods – 16 records.
All 3 search strings were limited to books.
Key author searches
Web searches via Google were undertaken on 15 July 2014 to find lists of publications for the five key authors identified. Publications lists were found either by searching each author’s institutional website or, where that was not possible, by using Google Books and Google Scholar. All publications were scanned to identify those relating to case study methods.
David Byrne – five publications.
Bent Flyvbjerg – 23 publications.
Roger Gomm – 11 publications.
Charles Ragin – 18 publications.
Robert K Yin – 18 publications.
Database searches
MEDLINE In-Process & Other Non-Indexed Citations and MEDLINE via OvidSP
URL: http://ovidsp.ovid.com/.
Dates database searched: 1946 to 19 July 2014.
Searched on: 22 July 2014.
Records retrieved: 581.
Search strategy
-
*Organizational Case Studies/ (191)
-
Organizational Case Studies/mt, st [Methods, Standards] (29)
-
(organi?ation$ adj5 case adj (study or studies)).ti,ab. (217)
-
1 or 2 or 3 (415)
-
*Research Design/ (23,525)
-
*Methods/ (972)
-
5 or 6 (24,381)
-
(case adj (study or studies)).ti. (24,261)
-
case study research.ti,ab. (149)
-
case-oriented research.ti,ab. (1)
-
8 or 9 or 10 (24,358)
-
7 and 11 (161)
-
case study method$.ti. (37)
-
case-based method$.ti,ab. (19)
-
13 or 14 (56)
-
4 or 12 or 15 (618)
-
exp animals/ not humans/ (3,968,668)
-
16 not 17 (607)
-
limit 18 to english language (581)
Key
-
/ = indexing term (MeSH heading)
-
exp = exploded MeSH heading
-
* = focussed MeSH heading
-
/mt, st [Methods, Standards] = MeSH heading restricted to those with Methods or Standards subheading applied
-
$ = truncation
-
? = wildcard
-
.ti,ab. = terms in either title or abstract fields
-
adj5 = terms within five words of each other (any order)
-
adj = terms next to each other (order specified)
Applied Social Sciences Index and Abstracts (ASSIA) via ProQuest
URL: www.proquest.com/.
Search date: 22 July 2014.
Records retrieved: 627.
Search strategy
(TI,AB(organi?ation* NEAR/5 (“case study” OR “case studies”)) AND la.exact(“English”)) OR (((SU.EXACT(“Research methods”) AND la.exact(“English”)) OR (SU.EXACT(“Methodology”) AND la.exact(“English”)) OR (SU.EXACT(“Research design”) AND la.exact(“English”))) AND ((SU.EXACT.EXPLODE(“Case studies” OR “Single case studies”) AND la.exact(“English”)) OR (TI,AB(“case study” OR “case studies”) AND la.exact(“English”)) OR (TI,AB(“case study research”) AND la.exact(“English”)) OR (TI,AB(“case-oriented research”) AND la.exact(“English”)))) OR (TI,AB(“case study method*”) AND la.exact(“English”)) OR (TI,AB(“case-based method*”) AND la.exact(“English”))
Key
-
SU.EXACT = subject heading
-
TI,AB = terms in the title or abstract fields
-
NEAR/5 = terms within five words of each other (any order)
-
* = truncation
-
“ ” = phrase search
-
la.exact = language limit
Health Management Information Consortium via OvidSP
URL: http://ovidsp.ovid.com/.
Dates database searched: 1979 to May 2014.
Searched on: 22 July 2014.
Records retrieved: 244.
Search strategy
-
(organi?ation$ adj5 case adj (study or studies)).ti,ab. (78)
-
research strategies/ (33)
-
research design/ (198)
-
research methodology/ (287)
-
research methods/ (1251)
-
methods/ (310)
-
method study/ (32)
-
evaluation methods/ (120)
-
2 or 3 or 4 or 5 or 6 or 7 or 8 (2114)
-
case studies/ (2901)
-
(case adj (study or studies)).ti,ab. (5541)
-
case study research.ti,ab. (48)
-
case-oriented research.ti,ab. (0)
-
10 or 11 or 12 or 13 (7490)
-
9 and 14 (120)
-
case study method$.ti,ab. (70)
-
case-based method$.ti,ab. (2)
-
1 or 15 or 16 or 17 (262)
-
limit 18 to english (244)
Key
-
/ = indexing term
-
$ = truncation
-
? = wildcard
-
.ti,ab. = terms in either title or abstract fields
-
adj5 = terms within five words of each other (any order)
-
adj = terms next to each other (order specified)
PsycINFO via OvidSP
URL: http://ovidsp.ovid.com/.
Dates database searched: 1806 to July week 3, 2014.
Searched on: 22 July 2014.
Records retrieved: 856.
Search strategy
-
(organi?ation$ adj3 case adj (study or studies)).ti,ab. (426)
-
*Methodology/ (21,666)
-
case study.md. (0)
-
“2260”.cc. (28,838)
-
2 or 3 or 4 (43,481)
-
(case adj (study or studies)).ti. (19,192)
-
case study research.ti,ab. (768)
-
case-oriented research.ti,ab. (5)
-
6 or 7 or 8 (19,729)
-
5 and 9 (381)
-
case study method$.ti. (79)
-
case-based method$.ti,ab. (37)
-
1 or 10 or 11 or 12 (887)
-
(rat or rats or mouse or mice or hamster or hamsters or animal or animals or dog or dogs or cat or cats or bovine or sheep).ti,ab,sh. (257,776)
-
13 not 14 (883)
-
limit 15 to english language (856)
Key
-
/ = indexing term
-
* = focussed subject heading
-
.md. = terms in the methodology field
-
.cc. = classification code (2260 is code for Research Methods and Experimental Design)
-
.sh. = terms in subject heading field
-
$ = truncation
-
? = wildcard
-
.ti,ab. = terms in either title or abstract fields
-
adj3 = terms within three words of each other (any order)
-
adj = terms next to each other (order specified)
Social Science Citation Index via Web of Science, Thomson Reuters
URL: http://thomsonreuters.com/thomson-reuters-web-of-science/.
Dates database searched: 1956 to 18 July 2014.
Search date: 22 July 2014.
Records retrieved: 1351.
Search strategy
Indexes=SSCI Timespan=All years
-
TS=(organi?ational NEAR/3 (“case study” or “case studies”)) (454)
-
TS=“case study research” (895)
-
TS=“case-oriented research” (7)
-
TS=“case study method*” (969)
-
TS=“case-based method*” (40)
-
(#5 OR #3 OR #2 OR #1) AND LANGUAGE: (English) (1351)
Key
-
TS= topic tag; searches terms in title, abstract, author keywords and keywords plus fields
-
* = truncation
-
“ ” = phrase search
-
NEAR/3 = terms within 3 words of each other (any order)
Appendix 2 Synthesised data extraction of methodological texts
| Section | Items |
|---|---|
| Themes that cross multiple sections | |
| Plan |
|
| Design |
|
| Prepare |
|
| Collect |
|
| Analyse |
|
| Share |
|
Appendix 3 Delphi consensus process round 1 questionnaire
Appendix 4 Delphi consensus process round 2 questionnaire
Appendix 5 Respondent comments from round 1
Do you have any other comments about the design section? (An opportunity to add more items will be given later in this survey)
I’m assuming you’re referring to qualitative case studies, but some of the statements above have a very ‘quantitative’ feel to do them and feel a bit out of place.
Helpful to know the case study method literature that are being used as source references, they do not all agree on the key elements of case study design.
Burdens and risks are the business of ethics committees, so reporting of ethics approvals may act as a proxy for reporting in the paper.
The answer to many of these questions will depend very substantially on the design used, e.g. inductive ethnography is very different from a theoretically based study. Some of the questions asked imply to my mind an overspecification and formalisation of the case study process, e.g. last one – a protocol may evolve rather than being fixed at the start of the study.
I am sceptical of all attempts to reduce good, reflective qualitative research to a set of mandatory steps. I particularly don’t like the insistence on a formal ‘research question’ (as opposed to a topic/area of interest) which can constrain good exploratory case studies with a broader aim of just understanding what’s going on. This is why I am not prepared to tick essential against many of these things, though they may be good in many cases.
My understanding of this Delphi is that it relates to description/presentation of case studies for external audiences. I have answered it accordingly. However, the items under ‘Rate the importance of the following tools and techniques for describing development of the final research question’ did not seem to be about reporting, so I struggled slightly with these. There are also two suggestions in the final section of this page that I did not understand (‘State the deliverables required’ and ‘Specify the need for recommendations’). I tried to leave these unanswered but the web page would not let me, so I have put them down as ‘not necessary’ – but this may be because of my misunderstanding of what they mean.
General comment – you haven’t provided the option of saying something like ‘not appropriate’ rather than not necessary. This pushes the respondent to answer not necessary when they have some issues with the question. The meaning is not the same. AS I couldn’t continue without answering the questions I was not happy with answering, I have ticked desirable for them. I’m really not sure about the validity of a survey where it is not possible to avoid answering a question that you don’t feel is clear or well stated. Q1 – hard to answer as some studies may or may not be defined as case studies, depending on how you frame or think about them. This made the question as presented difficult to answer. Q – Identifying the purpose – boundaries of the case. I think this is essential but found the question difficult to answer as presented as one might define a case in a more systems- based way, so suggesting the context is external was not a helpful way of framing this question, in my view. Heterogeneity of the cases as representative: I couldn’t answer this in the terms set. It should be essential to say something about the type of case and whether it can be considered representative or not – if it is claiming to be so – but as one of your prior questions note, cases may sometimes be selected for quite different reasons than representativeness. State the research questions/hypotheses – yes (I have put essential), but in some studies that are very exploratory, even stating a research question might be considered in appropriate unless constructed broadly enough. In the following question, I have answered desirable but felt this was difficult to respond to it is also poorly framed. Some case studies in their nature would avoid coming to something ‘final’. It depends how you interpret final. Also, I wasn’t sure what the question was really asking. Do you mean that the write up should describe the process of refining the questions as part of the study, in relevant studies? Or beforehand? Or both? State the deliverables required – I didn’t understand this question. Do you mean by funders or external agencies? Or, if relevant, the organisation being studied?
The possible responses are very limiting. The authors seem to have worked out what they think is best and are asking ‘do you agree with us?’
Many of these questions are not intelligible and seem premised on a very positivist world view.
Some kinds of organisational case study would be less dependent on a prior research question/hypothesis – but it is a good discipline to keep checking in on the emerging research question/focus during the course of research. For some of these answers, I wanted to answer ‘it depends’ – if case studies were being used in an evaluative context, then framing around ‘controls’ or comparators may be essential, less so if more exploratory purpose behind the research.
Do you have any other comments about the background, context and theory section? (An opportunity to add more items will be given later in this survey)
Again, a bit confused by some of these statements – how can you know whether ‘exploration was successful’? Also talk of ‘variables’ concerns me – very quantitative language – surely we are searching of understandings and explanations rather than reducing things down to what variables predict what?
some of the words here variables, hypotheses are very strange in this context indeed. a conceptual framework is ordered around concepts and not facts or events, as wrongly implied in the first statement.
The further I get into this the more uncomfortable I feel about the rigidity of the assumptions underlying the questions. It all seems too deterministic, and I am not reassured by a tiny ‘unless using grounded theory’ get-out clause.
Personally I prefer that studies should report a detailed literature review, but I am conscious of the fact that some philosophies discourage a lot of prior literature review, instead doing this work as part of the analysis process following lines of enquiry. This then does raise a reporting question of how and where in a report the relevant literature and theories are discussed.
You’ve assumed the paradigm is one of ‘variables’. I recommend Ramiller and Pentland ‘Management Implications in Information Systems Research: The Untold Story’. Journal of the Association for Information Systems Volume 10, Issue 6, pp. 474–494, June 2009. Also Bent Flyvbjerg ‘Five Misunderstandings About Case-Study Research’: Qualitative Inquiry, vol. 12, no. 2, April 2006, pp. 219–245.
Again I find these questions impossible to answer without more context. There should be a box for a non-response/question unclear. I do not share the stated assumption that reporting standards are easily or meaningfully distilled into a checklist of standards or even desirable. There is a need for paradigm differences and theoretical differences which this questionnaire fails to allow.
Do you have any other comments about the data collection section? (An opportunity to add more items will be given later in this survey)
Do you mean – how the data was analysed? Coding is only one part of the process of qualitative data analysis. . . .
Recruitment and criteria for how the study participants were identified, e.g. stakeholder, practitioner with specialist knowledge and who was excluded for whatever reason, often pragmatic choices have to be made and that needs to be made explicit.
Raw data is likely to be identifiable, so ethically it should only be made available to readers with the consent of participants.
The in depth question is very odd, not sure what it means at all.
But this is all just characteristics of good qualitative research reporting, not specific to case studies.
Again, not all of these seemed to relate to reporting, e.g. ‘Search for data until saturation is reached, that is, the evidence becomes redundant, with no new information’ is a methodological question, not a question of presentation.
See Flyvbjerg’s paper.
So many of these questions are suited to more nuanced answers. Data saturation is not a concept all qualitative researchers deploy for example. I question the inherent assumptions and premise of some of the questions.
Do you have any other comments about the data analysis section? (An opportunity to add more items will be given later in this survey)
Again – very quantitative focused criteria – I think its very important to be able to substantiate any analytical claims made within a case study, but I would not phrase this in terms of ‘internal validity’ – wrong concept to understand it.
Some of the statements above relate, and are therefore more or less important, to the type of case study and the underlying assumptions of the case study approach.
Some of these terms or ideas would be contested by some qualitative researchers as they don’t reflect the essential interpretive and emergent nature of good qualitative research.
Don’t understand what is meant by ‘Describe the criteria used to maintain the overall quality of a case study’.
Researchers may use other relevant concepts than those given here, e.g. as per Guba and Lincoln’s typology. There are other possible approaches apart from inductive or deductive.
Some of these are very obvious. But I’m not sure you’ve covered every element. And I’m not sure this is really a Delphi. It’s more a ‘do you agree with us’ questionnaire.
I think these questions derive from a very positivist understanding and implicit logic model. They are mostly not appropriate or meaningful for those coming from an interpretativist tradition. I think the choice boxes are too narrow and would like to register ‘not appropriate’ rather than ‘not necessary’ for many answers.
Do you have any other comments about the interpretation section? (An opportunity to add more items will be given later in this survey)
Representativeness is a misnomer here – qualitative research does not search for statistical representativeness in the same way that quant research does. You should read Nick Emmel’s book on sampling . . .
Relation to theory may also be key in generating an interpretation.
Suddenly a section which makes sense. It’s all about the credibility and reflexivity of the construction of the story, not following a set of process rules.
There is always an issue of concern over how much raw data to include given that the data tend to be very detailed and ‘bulky’. Also, inclusion of larger amounts can be very tricky in such studies when trying to maintain confidentiality so the balance can be very challenging. Respondents may sometimes be in a position where simply disguising name and role and clearly identifying details may not be sufficient as the role is quite specific. There are also debates about the role of the researcher and responsibility to analyse the data with care rather than resort to presenting large amounts of raw data in the hope that the data will speak for themselves. This requires a lot of elements, many of which have been referred to in the questions here.
One of the above questions implied that the case study is exclusively qualitative. Most good case studies contain some quant data.
Again I am unconvinced at trying to produce standards or black and white answers to such highly contextualised and creative interpretative processes.
Do you have any other comments about the sharing the results section? (An opportunity to add more items will be given later in this survey)
Unsure how many of these statements are case study specific, many would be true for any research report.
These questions are difficult to answer as although I believe sharing is very important there are differing views as to how to do it. Sometimes case studies reveal uncomfortable truths. We all look in the mirror sometimes and feel disappointed or want to see a different image. there may also be considerable differences and conflicts of perspective between different actors and parties in a case. it is essential in my view to feedback in some fashion unless there are very particular barriers to doing this, and to take the responses into full consideration. This can be very informative and revealing in itself, but may not always be straightforward.
This reads as silly ‘Aim for a thoughtful, balanced, and transparent tone of reporting’. Who is going to say ‘aim for a thoughtless, unbalanced and opaque tone’. So why ask this? It might be worth considering whether Van Maanen’s ‘realist’, ‘impressionist’ or ‘confessional’ genres are most appropriate. Experts are likely to disagree and hence you might end up with some data worth analysing.
These closed answers force the respondent into a very narrow set of choices. Most of the reporting categories would need to conditional on the type of report, type of funder and purpose of reporting, all very context-specific.
Publication and ‘push’ to policy and service depends on the quality of research and report! Not a given, although transparency is ultimate aim.
Appendix 6 Complete respondent comments from round 2
Do you think that a publication standard for reporting organisational case studies is desirable?
It depends on the audience or community. Advanced ethnographic case studies targeted at anthropology, cultural studies, sociology or policy studies are arguably distinct from HSR or trial research communities. Also, how do post-structuralist or even narrative case accounts fit with the idea of standards? Standards might constrain creativity and imagination!
All depends how it is used. It is one thing to have a standard that acts as a reference point or aspiration; it is another if this is used inappropriately to enforce standards taht [that] are not universally suitable for all research that might be subjected to it.
Yes but . . . recognise heterogeneity of case study research.
Define the policy relevance
Research focusing upon policy issues.
Ones that are policy relevant.
Ones that focus on policy or policy-related issues.
If it is a policy relevant issue.
Identify the process(es) (e.g. single or multiple/collective, embedded/nested, parallel, sequential, retrospective, cross-sectional, longitudinal)
Not sure what this question (‘processes’) means.
Comparative case studies.
Describe setting/context (physical, economic, historical, cultural, aesthetic) surrounding the case
Studies of atypical organisations (e.g. organisational innovations, pilot schemes, ‘alternative’ models of organisation.
Yes . . . and should be part of sampling frame too.
Identify the broad approach(es) (e.g. testing a theory, building a theory, illustrative, descriptive, interpretive, experimental)
Realist evaluation, qualitative.
Might be many of these things – more important to clarify the purpose of case study research, as above.
Show a prior appreciation of the theoretical issues and setting(s)
Theory-driven ones.
Realist evaluation or qualitative.
Theoretically driven ones.
Those aiming to apply or test a specific theory.
Acknowledge the potential conflicts between the needs and interests of any sponsoring organisations and the requirements of the research objectives. Show judgement to ensure that an appropriate balance between these is maintained
Where this has a bearing on the findings.
If affects decisions about which sites recruited, etc.
State the implications of the resources available to the researcher
Charity-funded evaluations – these tend to have much less resources and often result in ‘quick and dirty’ evaluations.
Where resource constraints may have made the findings biased, incomplete or otherwise misleading.
Describe the boundaries of the case; that is, distinguish the subject of the case study (the ‘phenomenon’) from external data to the case (the ‘context’). Spatial, temporal, and other concrete boundaries should be considered. Abstractions (e.g. the concept of ‘neighbouring’) cannot be considered a case
Contextual ones.
Inter-organisational studies; studies of relationships between organisation and its environment.
Describe the resolution of etic and emic issues in developing the research question. (Etic issues are brought in from the researcher from outside; emic issues emerge from inside the case. As the researcher begins to integrate etic and emic, the research question(s) evolves)
Participatory or action research case studies.
Describe how the final research question(s) was developed and refined from the broad prima facie question(s)
Realist evaluation.
Where the research has theoretical or explanatory ambitions.
May not be relevant if prior research question/focus is broad.
Report ‘Progressive focusing’ (i.e. if early research questions are not helping to thoroughly understand the case, or if new issues become apparent, describe how this changed the research questions)
Realist evaluation or qualitative.
Specify the need for recommendations
If sponsor allows. Sometimes e.g. NIHR you’re not allowed to make rec’s.
Where the purpose is to make recommendations.
Only if policy type research.
Policy focussed evaluations.
I would couch this as drawing out wider implications for the service.
Offer some evidence to the audience that the heterogeneity of the sample of cases is representative of the heterogeneity of the target population
Quantitative, positivist evaluations.
Only where representativeness of this nature is being claimed.
Those which aim to offer generalised or generalisable findings.
When the study depends on case heterogeneity.
Again, being explicit about sampling frame is important even if n = 3 or whatever.
Mention any rival cases that were considered
Where inclusion of the rival cases might prima facie appear likely to affect the results.
Could be useful, not required.
State the deliverables required
Maybe for policy-sponsored research.
Present the case study protocol and describe how it was used
Quantitative or positivist case studies – qualitative researchers know that case study research is iterative and evolving and that its perfectly acceptable for the research to be different from the protocol, quants/positivists have a problem with this!
Does this mean full interview schedules, document analysis, etc.
Describe the likely burden and risks associated with participation for those who [or the site(s) which] comprise the case study
Where the burden and risks may have affected data quality and availability, hence study findings.
Part of access and consent/governance issues.
Describe some early assessments of progress to see if the case should be dropped and another selected
Good idea, if resources allowed.
Outline the conceptual structure (i.e. themes or issues)
Realist evaluation.
Conceptualised ones.
Good to have a sense of theoretical frame or conceptual drivers for selection of cases.
Report the findings of a thorough literature review
If appropriate.
Realist evaluation.
Describe the theory, propositions and related issues developed to guide the case study and to generalise its findings
Positivist ones!
Realist evaluation.
Those which aim to test theories and/or produce generalisable findings.
Where theory is an important aspect of the case study design.
Outline the (logical) connection between the research question(s) and the data collected
Ones that use a logic model. I don’t like these questions at all.
Where the connection is not immediately obvious.
Define the logic linking the data to the propositions (i.e. what kind of analytic techniques were used)
Positivist ones.
Describe any other preparatory research components (e.g. expert interviews, expert workshop)
If it was done, it needs to be reported.
Report whether a pilot case study has been conducted
This is silly. If a pilot was done, it needs to be reported.
Where there are pilots.
Where new methods of data collection and/or analyses are being developed.
Define the criteria for interpreting the findings; that is, explicitly consider rival explanations (theories) at the outset, to guide decisions about which data should be collected, unless using grounded theory
If you’re using ‘criteria’ for interpreting, then you need to report those criteria. Some of us don’t.
Realist evaluation.
For theory-driven studies.
State which of the variables being investigated are hypothesised to be most important for explaining the phenomenon
Quantitative case studies – qual case studies would never use the term ‘variable’.
Explanatory case studies.
Describe any ethical considerations and obtainment of relevant approvals, access and permissions
NHS-based ones.
Describe the data collection tool(s) (e.g. questionnaire or observation protocol, including a description of any piloting or field testing of the tool)
One that you want to publish in a positivist journal.
When new or idiosyncratic data collection methods were used.
Describe the likely impact of the researcher on events and the behaviour of participants at the case study site, and the researcher’s own beliefs, values and prior assumptions
Where integral to service issue and data collection.
Where it is reasonable to have expected such impacts.
Where this is appropriate given the nature of data collected.
Where more participant/action research methods used.
Describe the observation plan and how it was developed
Where observation is used as a key method.
Only where observational techniques were used.
New observational methods; and where the findings depend heavily on observational data.
Where observation is a data collection method and where this is appropriate given the design/theoretical approach of the study.
Describe how the data were coded
Where qualitative data has been collected.
Describe whether the data provided an ‘up-close’ and ‘in-depth’ coverage of the case
Where relevant.
Surely that’s for the reader to determine on the basis of the information you give.
Qualitative.
Search for data until saturation is reached; that is, the evidence becomes redundant, with no new information
Qualitative.
Those with no other criteria for the completeness of data collection (in terms of the study aims and RQs).
Describe data protection measures
Where protocol dictates, e.g. ethics committee says so or legally required.
Those dealing with data whose collection or use raises ethical questions.
Part of ethics/governance – not necessarily separate.
State whether a comprehensive case study database, in which the raw data can be inspected (including notes, documents, tables and narratives) is available to readers
If appropriate.
Project website is advisable.
State that all the evidence was examined
Where incomplete examination would reasonably be expected to make the findings incomplete, biased or otherwise misleading.
Present raw data (including illustrative quotes) where necessary
Where appropriate. You can’t set the rules in stone.
Where relevant to reporting audience.
Qualitative – but don’t expect all raw data to be presented, but quotes to illustrate the points made are essential.
Where appropriate given the data that has been collected.
Describe how promising patterns, insights and concepts were identified
Where appropriate, and perhaps using illustrative elements but not exhaustively.
Qualitative.
Those using inductive methods.
Where appropriate given the design of the study.
Address the concept of internal validity (in explanatory or causal studies) (i.e. establishing a causal relationship, whereby certain conditions are believed to lead to other conditions, as distinguished from spurious relationships)
For explanatory or causal studies.
Describe how triangulation was carried out, especially in confirming and disconfirming major assertions [e.g. data triangulation (validation), investigator triangulation, theory triangulation, methodological triangulation]
Where appropriate.
Where data triangulation is used.
If this is the approach followed.
Those using multiple kinds of data (interview, documents, observation etc.).
This is appropriate for case studies which explicitly say they aimed to triangulate as part of the design.
Outline a chain of evidence that allows the reader to follow the derivation of any evidence from initial research questions to ultimate case study conclusions, via the collected data
In practice, this is hard to do.
Describe the criteria used to maintain the overall quality of a case study
Remove the word ‘criteria’ and replace with ‘approach’.
Not quite sure what this means.
Structure the reporting of the analysis around the research questions
Only those which set out with fixed research questions – many will not.
Useful for presentation.
Omit secondary data that is not essential for understanding and evaluating the case study analysis
Where secondary data analysis was a planned component of the original research design.
Address the concept of construct validity
Quantitative.
Present data in tabular form to summarise and compress data
Where presentationally clearer than text.
Address the concept of reliability
Quantitative.
Only where the methods might seem to raise a prima facie objection that the findings are subjective to the author(s).
Discuss the sampling (or case selection) and explanation of sampling strategy
Studies of heterogeneous populations of organisations.
Use description to provide the reader with a ‘vicarious experience’, or a sense of being there in person, and to enable understanding of the experience from the informants’ perspectives
Depends on reporting format.
Outline the researcher’s perspective and relationship to the case(s). The audience needs to understand researcher’s role and perspective to accept findings
Qualitative.
Where there might be the appearance of a conflict of interest, or the author(s) have a partisan reputation.
Where there are participant researchers.
Provide enough raw data prior to interpretation for readers to consider their own alternative interpretations
Where appropriate.
Where the findings rest on the balance of complex and ambivalent data.
Identify the researcher position. If the researcher has a close relationship or a past history with the case being studied, this information should be made transparent
Only where the researcher has a relationship with the case being studied.
Acknowledge revision of the report taking account of feedback from stakeholders
Where appropriate.
Participatory/action research.
Define the intended audience
This is the key factor that shapes most of the responses.
Report checking ideas and explanations with those in the culture
Where appropriate.
Action research/participatory.
Where the author(s) have heavily interpreted or re-structured the original data.
Where possible have informants/participants review the draft report
Where appropriate.
Study sites should usually have opportunity to review some form of findings.
Be reflective and have feedback workshops with onsite collaborators to ‘road test’ early formulations
Where relevant.
Participatory/action research.
Consult with a range of experts with diverse points of view during after drafting conclusions
Where appropriate.
Participatory/action research.
Include the reactions of data sources (and other prospective readers) to the accounts
Where the author(s) have heavily interpreted or re-structured the original data.
Appendix 7 Items with ≥ 70% consensus ‘Does not need to be reported’ in round 2
| Item | Percentage of total responses | ||
|---|---|---|---|
| Should be reported for all organisational case studies | Should be reported for a specific type of organisational case study | Does not need to be reported | |
| Describe the resolution of etic and emic issues in developing the research question | 20 | 6.7 | 73.3 |
| Describe how the final research question(s) was developed and refined from the broad prima facie question(s) | 6.7 | 20 | 73.3 |
| Present the case study protocol and describe how it was used | 13.3 | 13.3 | 73.3 |
| Describe the likely burden and risks associated with participation for those who [or the site(s) which] comprise the case study | 13.3 | 13.3 | 73.3 |
| Describe any other preparatory research components (e.g. expert interviews, expert workshop) | 20 | 6.7 | 73.3 |
| For purely exploratory studies without any initial propositions, state a purpose and the criteria by which the exploration is judged successful or not | 26.7 | 0 | 73.3 |
| State which of the variables being investigated are hypothesised to be most important for explaining the phenomenon | 13.3 | 13.3 | 73.3 |
| Structure the reporting of the analysis around the research questions | 13.3 | 13.3 | 73.3 |
| Address the concept of reliability (i.e. demonstrating that the operations of a study can be repeated with the same results) | 13.3 | 13.3 | 73.3 |
| Use description to provide the reader with a ‘vicarious experience’, or a sense of being there in person, and to enable understanding of the experience from the informants’ perspectives | 20 | 6.7 | 73.3 |
| Provide enough raw data prior to interpretation for readers to consider their own alternative interpretations | 13.3 | 13.3 | 73.3 |
| Publish the report | 26.7 | 0 | 73.3 |
| Disseminate to scientific (exploratory and explanatory case studies) and policy audiences (exploratory and example case studies) | 26.7 | 0 | 73.3 |
| Acknowledge revision of the report taking account of feedback from stakeholders | 13.3 | 13.3 | 73.3 |
| Mention any rival cases that were considered | 6.7 | 13.3 | 80 |
| State the deliverables required | 13.3 | 6.7 | 80 |
| Describe whether a range of experts were consulted during the final stages of developing the conceptual framework and report the findings of this consultation | 13.3 | 6.7 | 80 |
| Search for data until saturation is reached; that is, the evidence becomes redundant, with no new information | 6.7 | 13.3 | 80 |
| State whether a comprehensive case study database, in which the raw data can be inspected (including notes, documents, tables and narratives) is available to readers | 6.7 | 13.3 | 80 |
| Address the concept of construct validity (i.e. identifying correct operational measures for the concepts being studied) | 13.3 | 6.7 | 80 |
| Be very clear about the research outcomes and how the organisation(s) will benefit from involvement | 13.3 | 6.7 | 80 |
| Define the intended audience | 13.3 | 6.7 | 80 |
| Where possible, have informants/participants review the draft/report | 6.7 | 13.3 | 80 |
| Be reflective and have feedback workshops with onsite collaborators to ‘road test’ early formulations | 6.7 | 13.3 | 80 |
| Consult with a range of experts with diverse points of view during and/or after drafting conclusions | 6.7 | 13.3 | 80 |
| Describe some early assessments of progress to see if the case should be dropped and another selected | 6.7 | 6.7 | 86.7 |
| State that all the evidence was examined | 6.7 | 6.7 | 86.7 |
| Omit secondary data that is not essential for understanding and evaluating the case study analysis | 6.7 | 6.7 | 86.7 |
| Present data in tabular form to summarise and compress data | 6.7 | 6.7 | 86.7 |
| Array and display data in different ways | 13.3 | 0 | 86.7 |
| Explicitly consider the most appropriate overall reporting structure (e.g. linear-analytic; comparative; chronological; theory building; ’suspense’; unsequenced; a chronological or biographical development of the case; a researcher’s view of coming to know the case; description one-by-one of several major components of the case) | 13.3 | 0 | 86.7 |
| Identify the relevant stakeholders | 13.3 | 0 | 86.7 |
| Think about narrative dramaturgically, i.e. in terms of actors, roles and stages | 6.7 | 6.7 | 86.7 |
| Include the reactions of data sources (and other prospective readers) to the accounts | 6.7 | 6.7 | 86.7 |
| Include ‘issue questions’ or ‘issue statements’ when describing the research question. (‘Issues’ identify one or more aspects of the situation or circumstance surrounding the case, in order to frame the inquiry) | 6.7 | 0 | 93.3 |
| List evidence sources in order of importance; give further details about specific items within each source | 6.7 | 0 | 93.3 |
Appendix 8 Items with no overall consensus
| Item | Percentage of total responses | ||
|---|---|---|---|
| Should be reported for all organisational case studies | Should be reported for a specific type of organisational case study | Does not need to be reported | |
| State whether an inductive (e.g. grounded) or deductive (e.g. hypothesis testing/theoretical framework) approach to the analysis has been taken | 66.7 | 6.7 | 26.7 |
| Discuss the sampling (or case selection) and explanation of sampling strategy | 66.7 | 6.7 | 26.7 |
| Identify the purpose of the case study (e.g. exploratory, explanatory, evaluative, intrinsic, instrumental) | 60.0 | 0.0 | 40.0 |
| Describe the data collection tool(s) (e.g. questionnaire or observation protocol, including a description of any piloting or field testing of the tool) | 60.0 | 13.3 | 26.7 |
| Draw attention to any discrepant data/evidence that complicates emerging understanding | 60.0 | 0.0 | 40.0 |
| Identify the process(es) (e.g. single or multiple/collective, embedded/nested, parallel, sequential, retrospective, cross-sectional, longitudinal) | 53.3 | 13.3 | 33.3 |
| Describe setting/context (physical, economic, historical, cultural, aesthetic) surrounding the case | 53.3 | 13.3 | 33.3 |
| Describe how the data were coded | 53.3 | 13.3 | 33.3 |
| Outline a chain of evidence that allows the reader to follow the derivation of any evidence from initial research questions to ultimate case study conclusions, via the collected data | 53.3 | 6.7 | 40.0 |
| Discuss plausible rival explanations for the observed data | 53.3 | 6.7 | 40.0 |
| Display enough evidence for the reader to reach their own conclusions | 53.3 | 0.0 | 46.7 |
| Identify the researcher position. If the researcher has a close relationship or a past history with the case being studied, this information should be made transparent | 53.3 | 6.7 | 40.0 |
| Describe why case study is the appropriate method | 46.7 | 6.7 | 46.7 |
| Define the case broadly [e.g. in a case study of ‘neighbouring’ the case might be defined as either a group of neighbours (people) or as a geographical neighbourhood (place)] | 46.7 | 0.0 | 53.3 |
| Aim for a sense of story to the presentation | 46.7 | 0.0 | 53.3 |
| Acknowledge the potential conflicts between the needs and interests of any sponsoring organisations and the requirements of the research objectives. Show judgement to ensure that an appropriate balance between these is maintained | 40.0 | 13.3 | 46.7 |
| Outline the (logical) connection between the research question(s) and the data collected | 40.0 | 13.3 | 46.7 |
| Define the logic linking the data to the propositions (i.e. what kind of analytic techniques were used) | 40.0 | 6.7 | 53.3 |
| Describe how promising patterns, insights and concepts were identified | 40.0 | 26.7 | 33.3 |
| Address the concept of external validity (i.e. defining the domain to which a study’s findings can be generalised) | 40.0 | 0.0 | 60.0 |
| Describe the analytic approach in detail (e.g. pattern matching, explanation building, time-series analysis, logic models and cross-case synthesis) | 40.0 | 0.0 | 60.0 |
| Discuss the worth and relevance of the research | 40.0 | 0.0 | 60.0 |
| Ensure the account is reflexive (i.e. ‘Sensitivity to the ways in which the researcher and research process have shaped the data collection’ and provision of sufficient information of research process for readers to judge) | 40.0 | 13.3 | 46.7 |
| Aim for a thoughtful, balanced and transparent tone of reporting | 40.0 | 0.0 | 60.0 |
| Ensure the report is easy to read | 40.0 | 0.0 | 60.0 |
| Identify the broad approach(es) (e.g. testing a theory, building a theory, illustrative, descriptive, interpretive, experimental) | 33.3 | 13.3 | 53.3 |
| Report ‘Progressive focusing’, i.e. if early research questions are not helping to thoroughly understand the case, or if new issues become apparent, describe how this changed the research questions | 33.3 | 6.7 | 60.0 |
| Outline the conceptual structure (i.e. themes or issues) | 33.3 | 20.0 | 46.7 |
| Describe the likely impact of the researcher on events and the behaviour of participants at the case study site, and the researcher’s own beliefs, values and prior assumptions | 33.3 | 26.7 | 40.0 |
| Present raw data (including illustrative quotes) where necessary | 33.3 | 26.7 | 40.0 |
| Identify software and describe how it was used | 33.3 | 6.7 | 60.0 |
| Address the concept of internal validity (in explanatory or causal studies) (i.e. establishing a causal relationship, whereby certain conditions are believed to lead to other conditions, as distinguished from spurious relationships) |
33.3 | 13.3 | 53.3 |
| Describe the criteria used to maintain the overall quality of a case study | 33.3 | 13.3 | 53.3 |
| Outline the researcher’s perspective and relationship to the case(s). The audience needs to understand researcher’s role and perspective to accept findings | 33.3 | 20.0 | 46.7 |
| Show a prior appreciation of the theoretical issues and setting(s) | 26.7 | 26.7 | 46.7 |
| Describe the boundaries of the case; that is, distinguish the subject of the case study (the ‘phenomenon’) from external data to the case (the ‘context’). Spatial, temporal and other concrete boundaries should be considered. Abstractions (e.g. the concept of ‘neighbouring’) cannot be considered a case | 26.7 | 13.3 | 60.0 |
| Describe the theory, propositions and related issues developed to guide the case study and to generalise its findings | 26.7 | 33.3 | 40.0 |
| Describe the observation plan and how it was developed | 26.7 | 26.7 | 46.7 |
| Describe data protection measures | 26.7 | 20.0 | 53.3 |
| Define the policy relevance | 20.0 | 26.7 | 53.3 |
| State the implications of the resources available to the researcher | 20.0 | 13.3 | 66.7 |
| Report whether a pilot case study has been conducted | 20.0 | 26.7 | 53.3 |
| Define the criteria for interpreting the findings; that is, explicitly consider rival explanations (theories) at the outset, to guide decisions about which data should be collected, unless using grounded theory | 20.0 | 20.0 | 60.0 |
| Describe how triangulation was carried out, especially in confirming and disconfirming major assertions, for example data triangulation (validation); investigator triangulation; theory triangulation; methodological triangulation | 20.0 | 33.3 | 46.7 |
| In collective case studies, describe analysis of data relating to/the individual component cases . . . | 20.0 | 13.3 | 66.7 |
| Discuss the representativeness of data – incorporate all shades of opinion | 20.0 | 13.3 | 66.7 |
| Specify the need for recommendations | 13.3 | 33.3 | 53.3 |
| Report the findings of a thorough literature review | 13.3 | 20.0 | 66.7 |
| Describe whether the data provided an ‘up-close’ and ‘in-depth’ coverage of the case | 13.3 | 26.7 | 60.0 |
| Report checking ideas and explanations with those in the culture (e.g. organisation) | 13.3 | 20.0 | 66.7 |
| Offer some evidence to the audience that the heterogeneity of the sample of cases is representative of the heterogeneity of the target population | 6.7 | 33.3 | 60.0 |
Glossary
- Case study
- A method of research that engages in the close, detailed examination of a single example or phenomenon.
- Organisational case study
- A case study relating to an organised body of people with a particular purpose, such as a business, government department or charity group.
- Paradigm
- A general set of assumptions, questions and methods that structures a field of inquiry at any given time.
- Positivism
- A philosophical and social scientific doctrine that upholds the primacy of sense experience and empirical evidence as the basis for knowledge and research.
- Post structuralism
- Attributes subjectivity and meaning to systems of differential relations, such as language or power; beyond that, it seeks to explain the generation of those structures, either in terms of historical change or of deeper linguistic and experiential realities.
- Relativism
- A philosophical position that all points of view are equally valid, and that all truth is relative to the individual.
List of abbreviations
- HSDR
- Health Services and Delivery Research
- NIHR
- National Institute for Health Research
- PRISMA
- Preferred Reporting Items for Systematic Reviews and Meta-Analyses