Comparison of case note review methods for evaluating quality and safety in health care

Choice of clinical condition for review

The initial research brief for this study encouraged research teams to consider using a limited range of tracer conditions, mainly concerned with the care for people with chronic conditions. Three conditions were initially proposed for the study – care for people with chronic obstructive pulmonary disease (COPD), heart failure and stroke. Following discussions with the research commissioners, COPD and heart failure were the two conditions chosen for the study.

Choosing the number of case notes for review

In choosing the number of case notes for review we were unable to use prior hypotheses to assist in determining how many case notes would be required for the reliability studies. We considered using van Belle’s method32 of calculating the number of events (e.g. identified opportunities for error as being event = 20 times the number of parameters) from which to assist with this calculation, but subsequently found that it was practically impossible to model the range of opportunities for error presenting in these complex care pathways. In addition, it was decided that the study was more likely to find variations in care rather than identifiable adverse events, and that in health care there could be very large opportunities for error. We therefore took a pragmatic decision to select approximately 50 case notes per condition per hospital. This number also fitted with the custom and practice of the RCP CEEu, in which about 60 case notes per hospital form the basis for review in national clinical audits, and this number of case notes had previously provided sufficient data for studies of inter-rater reliability. 24

For this first phase of the study we therefore sought to obtain sets of 50 case notes from consecutive admissions for each condition in each of eight hospitals. That is, 800 case notes in total.

Selection and recruitment of hospitals and staff

A four-stage process was used to recruit eight study hospitals in England. First, Hospital Episode Statistics33 on 28-day mortality data for COPD and heart failure were accessed through the East Midlands Public Health Observatory. Hospitals were excluded from the selection process if they reported less than 200 inpatient cases per year for either condition, effectively excluding smaller or specialist acute hospitals. There were 136 hospitals in the final data set.

Second, 28-day mortality data for the two study conditions for each hospital was combined, using simple averaging, to create an average 28-day mortality ratio for each hospital. Third, these were then ranked from the lowest mortality to the highest, and the data was split into four quartiles, each of 34 hospitals. And finally, from this ranking, hospitals were randomly selected in each of the lowest- and the highest-mortality quartiles.

Combinations of review methods and proposed numbers/types of staff

In our initial research proposal we sought to create review teams in each hospital, comprising two types of personnel: clinical staff (for example nurses and staff working in clinical audit departments) and doctors in senior stages of their specialist training [medical specialist registrars (SpRs)]. This choice of types of staff was made in order to test some of the assumptions in the literature6 that medically qualified reviewers undertake holistic case note reviews differently from other personnel. Different combinations of reviewer type would review the same records to test inter-rater reliability within types of reviewer and between types of reviewer. Each case note would be reviewed twice by each reviewer: first using holistic methods and then criterion-based methods. We chose this sequence to reduce the bias on the holistic review results that might occur from a reviewer having previously examined the case notes to undertake criterion review.

Subsequent to these initial decisions on numbers of case notes and staff, a more limited set of reviews was agreed with the research commissioning panel, because the costs of undertaking a full set of two reviews of each of 50 case notes across the eight hospitals using four reviewers per condition proved too great for the available study budget. This second proposal still retained the ability to make comparisons between types of reviewer and review methods, albeit with a smaller number of reviewers in total. Table 1 indicates the type of reviewers and frequency of review that were proposed. The number next to the code for clinical audit/nursing staff (CA) and physicians (P) indicates the number of reviewers sought for each condition in each hospital. Each reviewer was expected to evaluate 50 sets of case notes.

For each of the two conditions the framework in Table 1 was used to calculate the number of reviews and reviewers for both the holistic and the criterion-based review methods, so that the total proposed number of case note reviews for the eight hospitals was 3600, using 400 case notes overall for each of the two conditions (800 in all).

Recruitment of hospitals and staff

Recruitment of study participants was a complex and time-consuming process, as participation required the agreement the COPD and heart failure clinical teams at each hospital and was also dependent on the availability of hospital staff to review records for the study. A total of eight hospitals were required for participation in this first part of the study. Because we expected difficulties in recruiting hospitals, mainly a lack of availability of staff within hospitals to review records, a total of 20 randomly selected hospitals (10 in the lower-mortality quartile and 10 in the higher-mortality quartile) were contacted and invited to participate in the study when only eight were actually required to participate in this phase.

From the initial 20 hospitals contacted, the study recruited five hospitals in each quartile, including one reserve hospital per mortality quartile in an attempt to ensure that a minimum number of eight hospitals were available for the analysis. One reserve hospital subsequently dropped out, leaving a total of nine hospitals in the study – four in the lower-mortality quartile and five in the higher-mortality quartile (Figure 2).

FIGURE 2.

Distribution of higher and lower study mortality hospitals. (a) Box plot of COPD/heart failure mortality, showing differences between higher- and lower-mortality groups. (b) Bar chart showing individual hospital-level mortality rates.

Hospitals were invited to participate through contact with one of the specialists in COPD and one in heart failure management. In each selected hospital, consultants specialising in each of the two conditions were approached jointly by the RCP CEEu and the University of Sheffield, with a request that they act as sponsors for the study. Their role was to recruit review staff, within the hospital, who would undertake the data collection for the study. Recruitment of the selected hospitals was completed when two specialists in a hospital agreed to act as sponsor and there were enough staff to undertake the reviews (Figure 3).

FIGURE 3.

Recruitment to the study. HF, heart failure.

The proposed number of reviewers required for each condition at each hospital varied from one to four (see Table 1), and this request sometimes proved difficult to meet in some hospitals. Among the reasons affecting the recruitment of reviewers included whether the hospital had a dedicated audit department and the change-over time of SpR training posts. At the end of the recruitment period, three types of hospital staff (reviewers) were engaged in the study – doctors in specialist training, other staff with a clinical background (of whom many were nurses specialising in the care of one of the two tracer conditions) and non-clinical audit staff. Across the nine participating hospitals, the reviewers comprised 16 doctors, 10 specialist nurses (together with one clinically trained audit person, one pharmacist and one physiotherapist) and nine non-clinical audit staff (i.e. 38 reviewers in total).

Holistic review

The concept of structured implicit review28 has been found to be valuable in North American implicit review studies as a means of reducing the variability previously found in inter-rater reliability studies. 10,11 Structured implicit review attempts to place a framework on data collection by providing headings that can be used in the ‘holistic story’. However, US-based authors working for the RAND Corporation28 chose to use what might be termed a mid-point between criterion-based and textual holistic review, using structured questions that were not as specific as review criteria, but which organised responses to the questions in such a way that might be considered to ask closed questions of the data. See, for example, Box 1.

In this study, the concept of structured holistic review was developed to provide reviewers undertaking holistic reviews with a limited structure, but one that was at the same time not so directive as the structured implicit review framework developed by the RAND teams. In doing so, this allowed for different levels of health-care quality to be identified – from excellent care, to care not provided, to the identification of adverse incidents.

Data was captured under three phases of care and for care overall:

• care during the investigation/assessment phase

• care during the initial management phase

• care during the pre-discharge phase

• quality of care overall.

Using this structured holistic framework, reviewers were asked to provide two forms of assessment of quality and safety of care. First, reviewers provided a written assessment of the quality and safety of care of each patient, using information from the case notes (paper and/or electronic records) of the most recent episode of inpatient care for an exacerbation of the illness.

For the phases of care, reviewers were guided by two prompts:

• Please comment on the care received by the patient during this phase.

• From the records, was there anything in particular worth noting?

Second, reviewers were asked to rate the care received by the patient for each of three phases of care – admission/investigations, initial management and pre-discharge care. Each phase was rated on a six-point scale (1 = unsatisfactory, 6 = very best care), and a definition was provided for each of the points on the scale:

1. Care fell short of current best practice in one or more significant areas resulting in the potential for, or actual, adverse impact on the patient.

2. Care fell short of current best practice in more than one significant area, but is not considered to have the potential for adverse impact on the patient.

3. Care fell short of current best practice in only one significant area, but is not considered to have the potential for adverse impact on the patient.

4. This was satisfactory care, falling short of current best practice in only more than two minor areas.

5. This was good care, which fell short of current best practice in only one or two minor areas.

6. This was excellent care and met current best practice.

7. The format of the questions is set out in Box 2.

Next, in assessing the quality of care overall, reviewers were asked to comment on the care received by the patient overall.

An overall quality-of-care rating was requested for each patient review on a 10-point scale (1 = unsatisfactory, 10 = very best care, using only the two anchor points on the scale) to provide for a global rating of care quality. This was given a wider, more fine grained scale so that reviewers could assimilate their perceptions of care for all of the phases of care to give an ‘in the round’ assessment (Box 3).

Assessing the quality of recording in the case notes

Evaluation of the quality of care through case note review is critically dependent on the quality of recording in the case notes, together with that in associated data sources, such as computerised pathology and radiology results. It might be hypothesised that a poor record could prevent a high-quality retrospective critical review of care. Alternatively, there might be a relationship between poor case notes and poor quality of care. Factors enhancing the use of the record for case note review include the extent to which information is recorded and placed in the case notes, the detail or otherwise of the entry and the legibility of the entry.

It was anticipated that most of the information relevant to the study would be recorded on paper-based case notes but that systems would vary from hospital to hospital, for example in the extent to which the principal case notes provided a holistic record of care or whether medical notes and nursing notes might be held separately.

Reviewers were therefore asked to assess the quality of each record at the end of the holistic review, using a six-point rating scale (1 = inadequate, 6 = excellent):

1. The patient record contains gaps in three or more significant areas.

2. The patient record contains gaps in two significant areas.

3. The patient record contains gaps in one significant area.

4. The patient records are satisfactory and only contain gaps in three or more minor areas.

5. The patient records are good and only contains gaps in one or two minor areas.

6. The patient records are excellent.

Reviewers were asked to complete their assessment in the form shown in Box 4.

Review criteria development for COPD and heart failure

Criterion-based review does not seek judgements of care – it requires the reviewer to only identify and record specific items of care. The purpose of review criteria when used in clinical audit is to gather data on which to make a judgement about the quality of care provided by an institution. However, for the purpose of this study, although the quality of care provided by the hospital was useful information, the prime objective was to investigate the extent to which data collection of a case note review method was reliable and in which type of staff’s hands it was most reliable.

This objective meant that the number of review criteria used for each of the two conditions could be limited to a smaller number rather than, for example, the full set used by national clinical audit projects (for instance the RCP COPD audit comprised about 75 clinical criteria in total). 34

The review criteria were developed using established methods for developing explicit evidence-based review criteria from clinical guidelines. 19,20,22 That is, for each of the two conditions, the first draft of the criteria were developed from the evidence base in the relevant national clinical guideline23,31 and subsequently validated using expert opinion.

Developing data capture tools

In order to facilitate the work of the reviewers and data transfer to the study team, data capture was developed through an electronic format based on Microsoft Access^©. Holistic data capture forms were developed from the format outlined in Holistic review (above), using separate screens for key data, case history data, phases of care and overall care (see Appendix 4, Figure 33, for an example). The database was constructed so that information could be transferred to the study team either by e-mail or by CD, first removing all identifiable data to preserve anonymity for patients and staff. The hospital staff retained access to the full data set to provide for local analysis and audit should they so wish. Criterion-based review data collection fields were created in the same way as those for holistic data.

Because there was considerable variety of local systems and versions of Microsoft Access^© in the study hospitals, copies of Microsoft Access^© were purchased and made available to the reviewers where required. Provision was also made for staff to collect data on a paper form where data-processing facilities were difficult to access. For these records, data entry was undertaken by the research team from anonymised paper records.

Data were collected from consecutive admissions over a period of 6 months before the review process started in each hospital, a time period which varied slightly, but was approximately between January and July 2005.

Reviewer training and case note selection support

The study sought to provide all reviewers with standardised training in case note review, the emphasis here being to train in the data capture methods. Each reviewer was provided with copies of clinical guidelines for COPD and heart failure care as a means of ensuring that all reviewers had an explicit database of the standards of care expected for the two conditions. 23,31 Other than providing the guidelines, in this part of the study there was no intention to try to influence each reviewer’s own implicit standards for quality of care – that is, each reviewer would have their own, internal, standards for the care that they were reviewing.

During a day-long training session, reviewers were provided with an introduction to the two review methods (particularly as most reviewers were not familiar with the holistic method), together with review software training. Quality-of-care variation was discussed using four theoretical scenarios from stroke care that contained aspects of good and poor care (see Appendix 5). Stroke care was chosen for training to avoid biasing the reviewers in their view of quality of care for the two study conditions.

The challenges of finding information in paper-based records and dealing with missing data were also considered along with a discussion about the means of obtaining case notes from the hospital records departments. Particular attention was paid to identification of case notes of admissions for exacerbation of known COPD or heart failure (rather than new cases or admissions for a main condition that was not related to the study), and to selecting case notes from the most recent admission.

During the data collection period a telephone helpline was made available throughout office hours if reviewers had any queries or required advice about the data collection. The study team also contacted each reviewer regularly throughout the study period to track progress with the reviews, and liaised with relevant hospital medical record departments if reviewers had problems obtaining records.

Overall approach

The quantitative analysis was designed to investigate the extent of reliability between individual reviewers and groups of reviewers of the same, and different, professional backgrounds, using measures of internal (intra-rater) consistency, and between-reviewer (inter-rater) reliability for holistic quality-of-care scale scores and criterion-based scores. Correlation and regression analyses were undertaken.

Detailed qualitative analysis of the textual data provided on the phases of care and the overall care was undertaken to explore the relationship between the holistic scale scores for each case and the narrative assessment. This analysis was also used to explore any differences between the results from the different professional groups undertaking the reviews.

Holistic scale score analysis

To assess intra-rater consistency (that is, whether reviewers were internally consistent in their ratings of care) for each individual review, the mean scale score rating was calculated across the three phases of care (admission/investigations, initial management and pre-discharge). The Pearson correlation coefficient was calculated between the mean rating of the three phases (each on a six-point scale) and the overall rating (on a 10-point scale) within each review. The purpose of this analysis was to examine the consistency of the reviewer’s scoring across the phases of care and in the final overall care to discover, for example, whether some reviewers might provide quite low scores for one or more phases of care and then a rather higher score for overall care.

Intraclass correlation coefficients (ICCs) were used as the principal measure of agreement. 35 However, although the kappa statistic is susceptible to prevalence (in this case, of ‘opportunity for error’ rates per set of records),10,11 kappa scores were also computed as measures of agreement for overall scores (see Table 6b below), as this measure of agreement is more commonly used in the literature.

To assess inter-rater reliability between ratings of the same record by different reviewers, raw ratings were converted to ranks to adjust for variation in the range of scores used by different reviewers and ICCs were calculated on these ranks.

Measuring reliability between reviewer pairs

The ICC gives the correlation between any two measurements or ratings for the same subject or patient, using randomly chosen methods or reviewers. ICCs are based on continuous data, unlike kappa statistics, which require the data to be categorical. ICCs were used to assess the reliability between reviews of the same patient records carried out by pairs or groups of reviewers (e.g. two nurses or two doctors) at the same hospital and were calculated first between the holistic quality-of-care ratings allocated by the two reviewers, and, second, between the criterion-based scores.

When undertaking the holistic (implicit) review, each reviewer rated the overall quality of care received by each patient against a 10-point scale. It is possible that different reviewers may have interpreted the rating scale differently (e.g. one reviewer may tend to give higher or lower ratings than another). Therefore, each reviewer’s ratings were converted to a rank. For example, if a reviewer reviewed 50 records then the ratings were ranked from 1 to 50. (In the event of tied ratings, the average rank was used.) The reliability between these ranked ratings for each pair of reviewers was then assessed using ICCs.

For the criterion-based review, care was assessed against a set of condition-specific criteria for either COPD or heart failure care. ICCs were used to assess the inter-rater reliability between the overall number of criteria (as unranked criteria scores) noted by each reviewer as having been met.

Criterion-based review

Data were scored in two ways, first to assess the completeness of the data and to assess the effectiveness of each reviewer type at completing the data collection form, and, second, to calculate a quality-of-care score for each review.

An ‘effectiveness of reviewer’ score for each record review comprised one point allocated for each data field completed by the reviewer (irrespective of whether the criterion was recorded as being met or not being met), and one point subtracted for every data field left blank by the reviewer. These scores were converted to a percentage.

Quality-of-care scores for each record comprised the percentage of the criteria identified by the reviewer as having been met. ICCs were used to estimate inter-rater reliability for overall scores by pairs or triplets of staff reviewing the same records. Because some phases of care generated only a small number of criteria, ICCs were not computed for phases of care.

Intraclass correlation coefficient estimates from the different combinations of reviewers were pooled using a weighting that was inversely proportional to the variance of the estimate. 36

Comparison of holistic scale scores and criterion-based review

Inter-rater reliability results for each of the two review methods were compared. Additionally, an estimate of the within-staff-type consistency across the two review methods was calculated using p-values for differences between the overall holistic quality-of-care ratings and the percentage of criteria recorded as being met.

Comparison of quality scores with hospitals grouped by mortality level

In the original call for proposals for this study it was suggested that the quality-of-care scores might be risk-adjusted by severity of illness of each case. However, we agree with Daley and colleagues37 that risk adjustment remains a controversial and difficult subject. Pitches et al. 38 undertook a systematic review of 36 studies, which included 51 ‘process versus risk-adjusted mortality relationships’, exploring the extent to which variations in risk-adjusted mortality rates were associated with differences in quality of care. They found a positive correlation in only 51% of the relationships, with no correlation in 31% and an unexpected correlation in a further 18%, in what was a very heterogeneous set of studies.

A range of approaches to risk adjustment were considered in the initial phases of the study. Because of the complexity of data capture and the level of workload that could reasonably be asked of the reviewers, individual scoring of the risk for individual patients proved impossible. The chosen approach was therefore to compare aggregate quality-of-care results between the hospitals in the low- and high-mortality-rate groups.

Analysis of holistic textual data

The comments made by the reviewers in their holistic reviews were in two free text areas. The first asked them to comment on the care received during a particular phase of care or for care overall. The second asked if there was anything in particular worth noting from the records about the care (see Box 2, above).

Reviewers provided textual data when commenting on each of the four phases of care and the overall assessment of care. These data were analysed in two ways in order to address the question of whether, as suggested by Weingart and colleagues,6 different staff types were concerned with different elements of care when making their holistic assessments. Data from the past medical history heading was excluded from the analysis because there were few comments, and all were about the case notes rather than about care.

Resource analysis

In the initial proposal to the funding body we set out a proposal for a cost–benefit analysis of the two review methods. Because of resource constraints this did not form part of the final agreement. However, we decided to undertake a limited resource analysis in case choices on reviewer type might essentially be made on cost (where there were limited differences in review results between one or more types of reviewers or review methods).

The resource impact of each reviewer type was explored, based on self-reported data on the time taken to undertake each review and on annual staff cost data taken from Unit Costs of Health and Social Care 2005 (clinical and doctor reviewers) and mid-point administrative and clerical staff costs from Whitley Council pay rates (for non-clinical audit reviewers). 40 The mid-point on the scale was used as the cost for each staff type and only one cost for each staff type was used in the analysis.

Descriptive statistics for the time taken to undertake each review and the cost per review for each staff type were produced in spss. A mean time per review and a mean cost per review for each staff type was calculated. We also included the minimum and maximum range to look more closely at the spread of the data.

Completion rates for scale scores

Data returned by reviewers was checked for completion rates. Tables 3 and 4 show completion rates in excess of 90% for all phases of care, except for the overall phase assessment completion rate for COPD reviews by non-clinical audit staff.

Intra-rater consistency in holistic reviews

Across all three staff types there were statistically significant correlations (r > 0.71, p < 0.001) between the mean scale score ratings that reviewers assigned to the individual phases of care and their rating of the overall quality of care, indicating a fair to good level of intra-rater consistency in rating the quality of care using holistic review scale scores (Table 5). Reviewers appeared to be relatively consistent in the way that they scored quality of care for the phases of care in a case and then gave an overall assessment score for the episode of care.

Inter-rater reliability for holistic review

Holistic review reliability between scale score ratings of the same record by pairs of reviewers was fair within all three staff types, although it varied from one reviewer pair to another and for some pairs was very poor (Table 6a). The overall weighted mean ICC was fair across all three types of reviewers, with no significant differences between staff types. Table 6b displays the same analysis using kappa statistics. The same trend occurs as in Table 6a, that is, the doctor reviewers have a higher level of agreement than the other staff types, although the results for the nurse/clinical group and the non-clinical audit staff are somewhat lower than for the ICC analysis. This is likely to be due to differences in the two methods of analysis.

Comparisons between professional groups

Where reviewers from different staff types used holistic scale score methods to review the same record, inter-rater reliability was assessed within and between staff groups for all phases of care and overall care (Table 7). For the phase of care findings within staff groups there was generally modest to fair agreement within pairs, particularly among the doctors, although even in this group the range was large (see, for example, the initial management results), and, as we have stated in the paragraph above (relating to Table 6a), the level of agreement varied from one reviewer pair to another and for some pairs was very poor. However, where staff from different groups reviewed the same record, agreement between the different professional groups on their assessment of the quality of care was poor to non-existent.

The overall ‘quality-of-care’ score for both holistic and criterion-based methods used across the 684 patient records was similarly rated by the three staff types (between 70% and 79%, where 100% is very best care). Analysis of variance between the holistic overall scale ratings of the three staff types show that the nurse/other clinical group scores were significantly lower than the doctor (p < 0.001) and non-clinical audit groups (p < 0.001). The comparison of the last two groups showed no significant differences (p = 0.352).

Criterion-based reviewer effectiveness

Effectiveness scores relate to the ability of the reviewer to find and access the data in the case record. For each criterion one point was allocated for each data field completed by the reviewer (irrespective of whether the criterion was recorded as being met or recorded as not being met) and one point subtracted for every data field left blank by the reviewer. Effectiveness in capturing criterion-based data was high and similar across all three staff types (Table 8), with mean scores all being around 95% (that is, an average of approximately 1.5 data items missing for each review).

Inter-rater reliability for criterion-based review

Inter-rater reliability between criterion-based scores (that is, the percentage of criteria recorded as being met) for the same record by different reviewers ranged from moderate to good within all staff types, although with the doctors showing a significantly higher level of reliability (Table 9).

Non-clinical audit staff

Several non-clinical audit staff reviewers made no comment in the first box in most instances. In a few instances, those who made no comment did, however, offer a comment in the more general comments box (the second box). Among this reviewer group, the comments in the second box were sometimes about documentation rather than about the care delivered.

A relatively common approach by non-clinical audit staff was to present a list, of things that had been done or requested, or, in some instances, things that had not been done. In some instances the lists were quite exhaustive; however, it was not possible to say whether the reviewer had included them because they had decided they were relevant to judgements about the care the patient received or simply because they looked important. It is difficult therefore to reach firm conclusions about how much selection had been exercised.

This attempt to reach conclusions was made more difficult by the fact that in some instances the comments were clearly lifted verbatim from the notes or, in other instances, a paraphrase was given, which might, perhaps, include some interpretation of what had taken place. A list of items by itself without any information about why they are important or how they related to other aspects of their care means that there is not a sufficient narrative to allow a picture to be formed of what the reviewer thinks about the issue, let alone what their view is of the quality of care was delivered.

Another approach that was observed in the comments from this group was the attention to very specific issues, such as timing of medication issues, lack of follow-up in terms of test results, details of transfers, information given to patient and family, timing of interventions/other aspects of care, including delay length if it occurred. This was sometimes the main approach taken. In other reviews it was combined with the list approach. Two reviewers did, however, pick up issues of adverse incidents or queried the care delivered in a couple of instances, but this was very much the exception rather than the rule.

On the whole, among this staff group, there was relatively little judgement directly expressed by the non-clinical reviewers about the care perceived. There was a limited amount of explicit and implied judgement, but from only relatively few of the reviewers. It was unclear in some instances how much selection had been made about what to include in the comments provided.

Nursing and other clinical staff

As with all staff groups, there was variation in the types of review comments made by nurses, although many reviewers in this group included an element of listing of what had, and had not, been done. In some instances there was only a list of actions and omissions. However, for most cases there was also other information and in many instances implied or explicit judgement for care in general or for specific aspects of care. For example, many reviews gave information about what was done and commented on whether this was appropriate.

As with the non-clinical audit reviewers, some of the nurse reviewers mentioned documentation, but it was quite clear that they more often were concerned with care issues rather than documentation. Even the concentration of some nurse reviewers on care plans (which might be expected given the importance of this in nursing care) appeared to be more concerned with the content of the plan, rather than whether it was legible, which again tends to suggest a focus on care rather than documentation. In terms of comments about the plan, in most instances there was a judgement about the quality of the plan. Most of the comments about the plan were explicit judgements about it, and, in many instances, the views about the plan were the most explicit comments in terms of judgements made. In aspects of care other than the plan, the judgements were more often implied.

In terms of the areas of care that comments were made on, as might be expected many were about areas that might be considered the responsibility of nursing staff. For example, comments on review including observations, timing of medication administration, involvement of other nursing staff (such as specialist nurses), discharge planning, social circumstances, patient education and nutrition. However, a couple of nurse reviewers took a wider view and made comments on appropriateness of medication, investigations and teams involved in care. In some instances they also queried care provided. Several reviewers also picked up issues of concern, including potential and actual adverse events.

Overall, the nurse reviewers commented much more on the care delivered rather than the documentation, in contrast with the non-clinical audit reviewers. They also utilised the list approach, and, in some instances, it was combined with either implicit or explicit views about care delivered. While many focused on areas more traditionally thought of as nursing realms, one or two did give views (implicit, and, in one case, usually explicit) about medical as well as nursing care. It was easier to get an overall impression of the care delivered, what was delivered and views about the quality of care than with the non-clinical audit reviewers. However, the pictures were still patchy in most instances, and relied in many instances on the reader going with the implications, rather than being given an explicit view.

Doctors

As with the other two staff groups, there was considerable variation in the reviewing style and comments given by different medical staff who undertook reviews. The variation in comments provided ranged from no opinion or very brief opinions of care delivered to several lines that allowed a reasonable picture of the episode of care to be gleaned.

As with other staff groups, some element of listing what was done was evident in the comments from the medical staff. However, where this was done, the items listed were usually much fewer in number. Several reviewers gave limited comments but explicit judgements in almost all cases, so that their comments, as well as listing items, also contain some explicit judgements. If aspects of care had been omitted or delayed or the implication was that care could have been better, medical reviewers often gave further details that almost justified their implied criticism.

One of the most striking differences between this staff group and the other two groups was that almost all of the medical reviewers who gave fuller answers routinely gave explicit opinions, views or judgements about the care delivered. Medical reviewers picked up issues where the care given was queried and alternatives were suggested, and, in some instances, adverse incidents or practice said to be unsafe were picked up. These were more frequent in reviews by medical staff than in nursing/clinical groups and non-clinical staff. It was noted, however, that in instances where care was perhaps not as good as it should be, the reviewers were sometimes less willing to give an explicit judgement, certainly not without adding additional comments (and much more so than when care was deemed satisfactory). In instances where poor care was commented on, the comment often included a statement about it not being clear or mentioned if certain actions/treatments were done. This may, or may not, be a reflection of professional reluctance to criticise.

Overall, the medical reviewers gave explicit views about the care provided, often supplemented with comments that allowed a better picture of the episode of care to be gleaned. Their focus was on medical care rather than on nursing or patient-centred issues, which might be expected, and it might be considered that the comments were on the domains/items that could have a greater impact on patient outcomes.

All reviewers

There was variation in the type and amounts of comments given by reviewers. Some reviewers gave no response, others gave one or two words, while a list type response was given by some and extensive narratives were given by other reviewers. This variation might reflect different levels of understanding of care received, willingness to offer views about the care received, or both.

On the whole, it was most difficult to get a view of care from the non-clinical audit staff. From the nursing/clinical group, more information about care was given, and generally more judgements were given, although they were often implicit and in the area of nursing care rather than care overall. The medical reviewers, while they tended to focus on medical aspects of care, usually gave an explicit view about the care given and picked up issues that were likely to have an impact on patient outcomes.

Except for the information about explanation and justification of actions, proposed alternatives and views about likely impact (which was almost exclusively provided by the medical reviewers), the additional detail from lists provided by the other types of reviewers did little to help build a picture of the episode of care. In each group there was a variation in reviewing quality. The best in each group picked up issues of adverse incidents, unsafe care and gave a good account of the care episode, including their judgement about how good it was. In general the reviews provided by doctor reviewers gave a better representation of the care provided than did those of the nursing/clinical group which, in turn, was a better representation than that of the reviews from the non-clinical audit staff.

Aim

To investigate how much of the variance in outcome for two important clinical conditions (adjusted for measurable differences in risk) can be attributed to known differences in quality of clinical care in acute hospitals.

Research questions

• What is the relationship between the quality of care for individual conditions in hospitals and overall quality of care and quality markers across hospital institutions?

• What is the relationship between case mix or risk-adjusted outcome and quality of care as measured by case record review?

• Is high-quality care associated with high-quality outcomes in risk-adjusted cases?

• Is there a correlation in clinical quality between the management of different conditions in the same hospital, as measured by case note review?

Selection of study hospitals

In our original proposal we indicated that a total of 24 acute hospitals in England would contribute data to this process–outcome study, including the data from with the case note reviews in the initial eight (nine) hospitals in the reliability study. During the analysis of the first part of the study, it became clear that the method of data collection being selected for the outcomes study – using a single physician reviewer per condition per hospital and a compound method of reviewing – meant that the data from the reliability study could not contribute fully to the objectives of this outcome study. Thus, a total of 24 hospitals were separately recruited for this quality and outcomes study.

Selection of the hospitals used the same processes in the phase one reliability study. First, Hospital Episode Statistics33 on 28-day mortality data for COPD and heart failure were accessed through the East Midlands Public Health Observatory. Hospitals were excluded from the selection process if they reported less than 200 inpatient cases per year for either condition, effectively excluding smaller or specialist acute hospitals. There were 136 hospitals in the final data set.

Second, 28-day mortality data for the two study conditions for each hospital were combined, using simple averaging, to create an average 28-day mortality ratio for each hospital. Third, these were then ranked from the lowest mortality to the highest, and the data was split into four quartiles. And, finally, from this ranking, hospitals were randomly selected in each of the lowest- and the highest-mortality quartiles.

The 20 hospitals selected for the reliability study were excluded from the selection process. Thirty hospitals were then randomly selected from the remaining 116 in the data set – 15 from the lower-mortality quartile and 15 from the higher-mortality quartile. Six additional hospitals over the proposed 24 were selected to take account of the likely drop-out rate during the recruitment and fieldwork phases of the study.

At the commencement of data collection there were 25 hospitals included in the study. However, only 20 hospitals in total, 10 each in the upper- and lower-mortality groups, were able to collect data on both COPD and heart failure. Thus only 20 hospitals were included in the overall analysis.

Figures 13 and 14 demonstrate the mortality differences between the two groups of hospital included in the final analysis, each mortality group (upper and lower) containing 10 hospitals.

FIGURE 13.

Box plot of COPD/heart failure mortality, showing differences between the higher- and lower-mortality groups.

FIGURE 14.

Bar chart showing individual hospital-level mortality rate differences.

Selection of review conditions

This quality and outcomes study used the same two conditions as in the phase one study – namely, admissions for acute exacerbation of COPD and of heart failure – in each study hospital.

The working definitions for data collection are shown in Box 6.

Number of case notes for review and of reviews

Analysis of the data from the first part of the study suggested that 40 case notes would be an appropriate number to review at each hospital, for each condition, in a study that was to seek associations between recorded quality-of-care and hospital-level outcomes. Furthermore, review of 40 case notes has been shown in the UK Stroke and COPD national audits34,47 to be practical and to yield useful data. In contrast with the earlier analysis, it was not intended to undertake intra-rater and inter-rater reliability analyses, so it was not necessary for each set of records to be reviewed more than once.

In our proposal for phase two of the study we said that in order to test the reliability of the review process, 10 sets of case notes per condition per hospital would be double reviewed (rather than double reviewing all case notes). However, this target of 10 double reviews proved not possible, partly because of the costs involved in undertaking the double reviews and partly because of inability to recruit second reviewers of the same staff type, beyond the 50 reviewers already required.

Choice and recruitment of reviewers

In our initial research proposal we said that only one review method would be used at this stage of the project, the choice being made following the analysis of the data in the reliability study. Analysis of the reliability study showed that while there were similarities between the criterion-based reviewing skills of the doctor and nurse, there were differences in the type of holistic data captured by the two professional groups and which formed the basis of the judgements about quality-of-care scores. Medically trained reviewers tended to judge the quality of interventions, whereas nurses tended to review from a care pathway perspective. As the study reported here sought associations between process of care and outcomes of care, it was judged that medically trained reviewers in higher-specialty training were likely to provide the most appropriate data.

Following the random selection of candidate hospitals, in each hospital a senior physician working in cardiology (heart failure care), and a senior physician working in COPD care were both asked to recruit a physician reviewer in mid-stage training in their relevant specialty.

Selection and refinement of review methods

One of the subsidiary purposes of the earlier reliability study was to determine which of two review methods was most appropriate for the quality-of-care/outcomes study – criterion-based review or holistic review. We had found that the two methods were complementary, although different, and, therefore, that both methods might have a value in the review of care for the outcomes study. Review of care using evidence-based criterion methods would provide information on the extent to which care, overall, fitted with external standards. Structured holistic review with quality-of-care scores, together with short explanatory comments where reviewers thought it appropriate, would provide information about the nuances of care that could not be identified through the use of preconstructed review criteria. In this second part of the study it was not necessary to separate out the reviewing stages (holistic review followed by criterion-based review), as the results of the two methods were not being compared one with another, as in the earlier study. It was therefore possible to merge the holistic and criterion-based reviews into one process, so that the case notes were only reviewed once and a single data collection form could be used.

Using a form of structured implicit review, the reported quality of care provided to each patient was scored for each of three phases of care (admission/investigations, initial management and pre-discharge care) and for care overall.

In the earlier part of the study of holistic review, a six-point Likert scale was used by reviewers to rate the quality of care in each of the three phases of care, together with a 10-point Likert scale that was used to rate the overall quality of care. Although the 10-point scale was more finely grained, comparison in the phase one study between the six-point phase scores and the 10-point scale added to the complexity of the analysis without providing obvious benefits to the structured review process. A six-point scale was therefore used for all phases and overall care review in this study, with anchors of 1 = unsatisfactory and 6 = very best care. The descriptors for the six points are shown in Box 7.

Two prompting questions had earlier been used to seek textual comments on the quality of care, but reviewers tended only to respond to one or other of the questions. Thus, in this study, only one question was asked of reviewers when they provided textual comment on the quality of care. Box 8 shows the format of the questions used for phases of care and care overall.

The quality of care provided was also measured through the presence of a condition-specific set of review criteria for each of the two study conditions. A number of changes to the earlier data collection methods were required to make the criteria and data capture tools fit for purpose for the outcomes study. During the first phase study it became apparent that a limited number of the review criteria had far higher levels of missing data than other criteria. We presumed that this was because the data was routinely unavailable, or more difficult to find, or that the criteria were more difficult to understand. Whatever the reason, these criteria added nothing to the review process and were removed. Additionally, because of the potential impact of poor quality recording on the ability to assess quality of care, reviewers were asked to rank the quality of the information held in the clinical records on a Likert scale from 1 (poor) to 6 (excellent).

Reviewers were unaware of their hospital mortality ranking – whether they were in the higher- or lower-mortality quartile – and they evaluated the records within their own hospital, as would happen with a local clinical audit. No patient-identifiable data was returned to the study team or used in the analysis.

Reviewer training

Reviewers were trained to use a combination of two forms of case note review – criterion-based (explicit) review and holistic (implicit) review – and also to provide written critical commentary on quality of care received by the patient, including on adverse events. The majority of reviewers were trained at a seminar in which the review methods were presented, using a set of anonymised case notes as the basis for small group and larger directed discussions. Topics discussed included techniques to find relevant information and group discussion to identify good and less satisfactory performance from the case notes.

Some reviewers were unable to attend the main training sessions and they received one-to-one or small group training from study research staff.

Relationship between mortality groups and quality of care

Table 18 shows that most reviewers classify care in the good range [about 41% for COPD, and 46% for heart failure (score 5)], with around 19–21% in the ‘fell-short-of-best-practice’ range (score 1–3) and the lowest scores (1, unsatisfactory) under less than 5% for both conditions across the 20 hospitals.

The relationship between the holistic data (including mean scale scores for overall care, all of the phase of care scores) and the mean total criterion score, correlated with the hospital groups ranked by mortality, was analysed using the two independent samples t-test (Tables 19 and 20). Mean difference is the mean score for each case note review in the higher-mortality group of hospitals minus the mean score for the lower-mortality group of hospitals. A negative mean difference in the table indicates that the lower-mortality group mean score is higher than the mean score for the higher-mortality group.

For both conditions the findings are therefore consistent. Across all phases of care and overall care using holistic review, and total criterion score, there was a trend towards higher mean scores for the lower-mortality hospital groups. However, when account was taken of clustering, there were no statistically significant differences in scores between the two groups of hospitals.

The similarities between the quality-of-care scores in the upper and lower hospitals are further demonstrated when the relationship between mean overall and mean phase holistic scale scores and ranked mortality is explored. Tables 21–24 show the relationships between holistic scores (ranging from 1 = unsatisfactory to 6 = very best care) and ranked mortality. No significant differences in scores were found between the upper and lower mortality groups of hospitals.

The relationship between the mean holistic overall quality-of-care scores for COPD and heart failure in each hospital was also explored using a Bland–Altman plot (Figure 15) to investigate whether the two scores tended to be in the same direction in each hospital. That is, if the hospital was in the lower-mortality group (which, on average, had somewhat higher group mean scores for both COPD and heart failure) was this pattern replicated within each hospital? No clear pattern emerged between the two mean scores for each hospital when grouped for overall mortality, so that it is not possible to see a trend towards higher scores for both conditions in each of the lower-mortality hospitals, or for the lower scores for each condition in each of the hospitals in the higher-mortality group.

FIGURE 15.

Relationship between mean quality-of-care scores for COPD and heart failure within each hospital, grouped by mortality level (note: one hospital in the higher-mortality hospital group was an outlier in all of the analyses and was therefore excluded from the Bland–Altman plot analysis). HF, heart failure.

COPD mean holistic overall score (1 = unsatisfactory to 6 = very best care)

Table 25 shows that there were only two moderate negative correlations. Thus, as the COPD scale score increased (an indication of better quality of care), the frequency decreased at which NHS staff said they were encouraged to report errors (Figure 16), and also when staff said that care of the patient/service user was a high priority (Figure 17). These correlations are in an unexpected direction, as better quality of care might have been expected to relate to both a positive safety culture and to high priority for patient care.

FIGURE 16.

Scatter plot (with regression line) for COPD score by NHS staff survey Q27b: Encouraged to report errors (mean score). RReSQ, Records Review for Safety and Quality (study).

FIGURE 17.

Scatter plot (with regression line) for COPD score by NHS staff survey Q22e: Care of patient/service user is top priority (mean score). RReSQ, Records Review for Safety and Quality (study).

Heart failure mean holistic overall score (1 = unsatisfactory to 6 = very best care)

A total of four variables were significantly correlated with heart failure mean score: quality of services, existing national targets, new national targets, and NHS staff survey question 24a (know how to report concerns) (Table 26).

There is a strong positive correlation between quality of services (fair, good or excellent) and heart failure mean score: those trusts with a fair score had lower mean heart failure scores than trusts with a good or excellent score (Figure 18). Existing national targets (partially met, almost met, fully met) were also strongly positively correlated with heart failure mean scores where an increase in mean scores indicated an increased likelihood of trusts meeting existing national targets (Figure 19). There was a moderate positive correlation between new national targets (weak, fair, good, excellent) and heart failure – mean heart failure scores increased with increased levels of excellence (Figure 20).

FIGURE 18.

Scatter plot (with regression line) for heart failure score by quality of service. RReSQ, Records Review for Safety and Quality (study).

FIGURE 19.

Scatter plot (with regression line) for heart failure score by existing national targets. RReSQ, Records Review for Safety and Quality (study).

FIGURE 20.

Scatter plot (with regression line) for heart failure score by new national targets. RReSQ, Records Review for Safety and Quality (study).

Finally, there was a moderate positive correlation between NHS staff survey question 24a and heart failure mean score: the higher the percentage of responders who knew how to report incidents, the higher was the mean heart failure score (Figure 21).

FIGURE 21.

Scatter plot (with regression line) for heart failure score by NHS staff survey Q24a: Know how to report (% yes). RReSQ, Records Review for Safety and Quality (study).

COPD mean holistic phase score

Only NHS staff survey question 22e (care of patient/service user is top priority) was significantly correlated with COPD mean holistic score (Table 27). As with the overall score analysis above (Table 25), the correlation between the two variables was negative, and as COPD mean holistic score increases the mean score for care of patient/service users being a top priority decreased – a lower score indicates more disagreement with the statement (Figure 22).

FIGURE 22.

Scatter plot (with regression line) for COPD holistic score by NHS staff survey Q22e: Care of patients/service users is top priority (mean score).

Heart failure mean holistic phase score

Seven variables were significantly correlated with heart failure mean holistic score:

1. heart failure mean age [Hospital Episode Statistics (HES) data]

2. quality of services

3. percentage of patients with acute MI receiving thrombolysis

4. existing national targets

5. NHS staff survey question 27e (trust takes action to ensure never happens again)

6. NHS staff survey question 24a (know how to report incidents)

7. NHS staff survey question 22e (care of patient/service user is top priority).

Mean age is negatively correlated with score: as mean age increases mean holistic score decreases (Figure 23). This may be explained by older people having more complex problems, although reviewers in the phase one study showed that quality of care could be scored highly even when a patient did not survive.

FIGURE 23.

Scatter plot (with regression line) for heart failure holistic score by mean patient age.

Quality of service (fair, good, excellent) is positively correlated with mean score; thus, as heart failure mean phase score increases, the more likely a hospital is to be of good or excellent status (Figure 24). There is a moderate positive correlation between the percentage of patients with acute MI receiving thrombolysis and heart failure mean holistic phase score (an important association in heart failure care) (Figure 25). A strong positive correlation exists between existing national targets (partly met, almost met, fully met) and heart failure mean holistic phase score, mean scores increase as the level of meeting targets increases (Figure 26).

FIGURE 24.

Scatter plot (with regression line) for heart failure holistic score by quality of services.

FIGURE 25.

Scatter plot (with regression line) for heart failure holistic score by percentage of patients with acute MI receiving thrombolysis.

FIGURE 26.

Scatter plot (with regression line) for heart failure holistic score by existing national targets.

Additionally, there were moderate positive correlations between mean score for NHS staff survey questions 27e and 22e and heart failure mean holistic phase scores – the higher the heart failure score, the stronger the respondent agreed with the statement that the hospital takes action to ensure that it does not happen again or care of patient/service user is top priority (Figure 27). Finally, there was a moderate positive correlation between NHS staff survey question 24a and heart failure mean holistic phase score, where the higher the percentage of responders agreeing (% yes) they know how to report incidents results, the higher the heart failure mean holistic score (Figure 28).

FIGURE 27.

Scatter plot (with regression line) for heart failure holistic score by NHS staff survey Q24a: Know how to report (% yes).

FIGURE 28.

Scatter plot (with regression line) for heart failure holistic score by NHS staff survey Q22e: Care of patients/service users is top priority (mean score).

Correlations between criterion-based scores and outcome variables

Correlations at the moderate to very strong level are shown in Tables 31 and 32, with accompanying scatter plots. The results of all correlations are presented in Appendices 11 and 12.

COPD mean criterion score (out of 13)

Two variables were significantly correlated with COPD mean criteria score: incidents reported to the NPSA per 100 bed-days and NHS staff survey question 22f (happy with standard of care provided) (Table 31). A moderate negative correlation was observed with NPSA bed-days, as the number of incidents per 100 bed-days decreased the mean criteria score increased (Figure 29). Again this is an unexpected direction for the correlation, as increased reporting is a positive safety culture marker. A moderate positive correlation was observed with NHS staff survey 22f (mean score), where the higher the score, the happier the staff with their standard of care; thus, as the mean response to 22f increased, mean COPD score increased (Figure 30).

FIGURE 29.

Scatter plot (with regression line) for COPD criterion score by incidents reported to NPSA per 100 bed-days. RReSQ, Records Review for Safety and Quality (study).

FIGURE 30.

Scatter plot (with regression line) for COPD algorithmic score by NHS staff survey Q22f: Happy with standard of care provided (mean score). RReSQ, Records Review for Safety and Quality (study).

Heart failure mean criteria score (out of 11)

Two variables were significantly correlated with heart failure mean criteria score: incidents reported to the NPSA per 100 bed-days and NHS staff survey question 24b (knowing the system for reporting errors) (Table 32). In a reversal of results from the COPD criterion analysis there was a significant positive correlation between incidents reported to the NPSA and heart failure mean score, as the number of incidents increased the mean heart failure score increased (Figure 31). There was also a strong positive correlation between the percentage of responders to the NHS staff survey stating they did not know how to report errors confidentially and mean heart failure score. That is, as mean heart failure score increased, the percentage of responders reporting that they did not know how to report concerns also increased (Figure 32).

FIGURE 31.

Scatter plot (with regression line) for heart failure criterion score by NPSA incidents bed-days. RReSQ, Records Review for Safety and Quality (study).

FIGURE 32.

Scatter plot (with regression line) for heart failure criterion score by NHS staff survey Q24b: System for reporting (% no). RReSQ, Records Review for Safety and Quality (study).

Regression analysis

A priori, one of the objectives of this analysis was to fit multiple regression models to COPD and heart failure scores to establish any predictors. With a sample of 20 hospitals the number of variables in a multiple regression model should be restricted to two, as anything over two would result in overfitting a model. Furthermore, consideration should be taken of categorical variables such as existing national targets, where each level of response (not met, partly met, almost met, fully met) counts as one variable in the regression analysis – thus a categorical variable with three or more categories, which is the case for all categorical variables in this data set, would result in overfitting.

It was possible to fit a multiple regression model for COPD criterion score, where NPSA incidents and NHS staff survey question 22f (happy with standard of care provided) were significant predictors of COPD score at the 10% significance level. A one-unit increase in the number of incidents reported to NPSA per 100 bed-days resulted in a 0.5 decrease in COPD criterion score, and a one-unit increase in the staff who were happy with standard of care resulted in a 3.14 change in COPD score. Multiple regression models were fitted using the forwards stepwise method, by which the most significant variable at the univariate stage is selected first.

When the regression model was fitted to the COPD criterion score, this model included only the two variables that were significantly correlated with the score. When included in a regression model together, neither ‘incidents per 100 bed-days’ nor ‘NHS staff survey question 22f’ were significant at the 5% significance level (Table 33).

A multiple regression model was also fitted to heart failure criterion score, where NPSA incidents and NHS staff survey question 22f were significant predictors of heart failure score at the 10% significance level. A one-unit increase in the number of incidents reported to NPSA per 100 bed-days resulted in a 0.21 increase in heart failure criterion score, and a one-unit increase in the percentage of staff reporting ‘no’ to system for reporting errors resulted in a 0.11 change in heart failure score.

When a regression model was fitted to the heart failure criterion score, this model included only the two variables that were significantly correlated with criterion score. When included in a regression model together, incidents per 100 bed-days was no longer significant when included with NHS staff survey question 24b (percentage of staff reporting they did not know of the system) (Table 34).

For the remaining four analyses it was not possible to fit a multiple regression model, either because no two variables were significant in the same model or due to problems with overfitting for categorical variables.

Quality comments

This lady showed early signs of a pending stroke, which could have been prevented.

• No early diagnosis is made and no rapid referral to a specialist service is provided.

• No aspirin is prescribed to prevent the stroke.

• No specialist care in a stroke unit.

• There is a delay in providing a CT scan,

• Dysphagia (difficulty swallowing) is not picked up by early screening, leading to pneumonia.

• She does not receive early therapy input to begin rehabilitation and mobilisation.

• The communication systems between hospital and community fail.

• The community services are unable to provide the rehabilitation she needs.

Quality comments

Evidence shows that the danger of stroke is high after a TIA.

• She receives appropriate rapid referral to a specialist service.

• Secondary prevention with aspirin is commenced immediately.

• Early investigations are carried out.

• Appropriate surgical treatment is provided quickly.

• Secondary prevention and lifestyle advice are provided prior to discharge.

• Her health is monitored regularly by her GP.

Quality comments

• A CT scan is never carried out.

• He does not receive specialist care in a stroke unit.

• His dysphagia (difficulty swallowing) is not picked up, leading to pneumonia and malnutrition.

• Prolonged, possibly unnecessary, use of a catheter leads to complications.

• No early supported discharge from hospital is offered.

• No rehabilitation is provided on his return to primary care.

Quality comments

• A CT scan is carried out early, confirming the diagnosis.

• Secondary prevention with aspirin is commenced early.

• Hospital care is provided in a specialised stroke unit.

• He receives regular monitoring of physiological indicators while in hospital.

• Dysphagia is identified early and managed, reducing the risk of pneumonia and malnutrition.

• Urinary incontinence is managed without the need for a catheter (this is associated with better clinical outcome).

• Mobilisation therapy is begun early and monitored (this is one of the key features of stroke units associated with better clinical outcome).

• Early supported discharge by a specialist team is deemed suitable and is provided (this is proven to result in the same outcomes as hospital rehabilitation and is liked by patients).

Key data

History and patient characteristics

Rating scale results: quality-of-care ratings for the admission phase

Criterion-based review – admission phase

Rating scale results – quality-of-care ratings for the initial management phase

Please note, some of these results are subjective and are the opinions of the individual reviewers.

Rating scale results – quality-of-care ratings for the pre-discharge phase

Please note, some of these results are subjective and are the opinions of the individual reviewers.

Rating scale results: quality-of-care ratings for the overall care

Please note, some of these results are subjective and are the opinions of the individual reviewers.

Rating scale results – ratings for the quality of patient records.