Development of risk models for the prediction of new or worsening acute kidney injury on or during hospital admission: a cohort and nested study

Conceptual model

The conceptual model of CKD is well established, and continues to inform clinical medicine, research and public health. 14–16 In contrast, the concept of acute kidney diseases and disorders (AKD) is relatively new and attempts to map itself onto the widely accepted CKD concept. The definition of AKI, in contradistinction to CKD, describes an abrupt, time-limited reduction of function which has at least the potential to recover.

Both AKI and CKD describe decreased function which can lead to complications, including end-stage renal disease (ESRD) and mortality. Risk factors for AKI and CKD are similar, and there is a conceptual overlap and interplay between the two. AKI and CKD are both risk factors for each other, and also worsen the prognosis of each other.

Underlying biology

If we now consider the situation at a biological level, in the elderly CKD population several factors could lead to increased susceptibility to AKI. Changes in the renal vasculature occur with age, just as in other vascular beds, often owing to comorbidity, but also in the absence of comorbidity. 17 It is suggested that these changes eventually cause cortical glomerulosclerosis, interstitial fibrosis and tubular atrophy, and compensatory hypertrophy and hyperfiltration of glomeruli in the medulla, contributing to the development of CKD. 18 With increasing age and CKD, function in both proximal and distal tubules is compromised, hampering the ability to control fluid and electrolyte balance and affecting tubuloglomerular feedback. 17,19 These changes, related to age and CKD, may exacerbate clinical events such as dehydration and drug toxicity, which carry a high risk of AKI. 18

Contrary to the idea that the diseased kidney is at increased risk of AKI is the ‘intact nephron’ hypothesis. 20,21 In surviving nephrons of a kidney with CKD there remains homogeneity of function and regulatory capacity. The kidney responds in a predictable and organised manner to maintain homeostasis in the face of a number of challenges. There may be fewer functional nephrons available, and a reduced reserve, but available nephrons are functionally intact. This is evident until the late stages of the disease and should, therefore, not produce an increased risk of AKI; however, it may impart an increased severity when it develops, which is therefore more likely to be clinically evident. There is also the concept of priming or conditioning, where the ischaemic or diseased kidney in CKD is more ‘used to’ insults and can, therefore, maintain function.

Given that people with CKD have an increased burden of vascular disease, it may be that less of a vascular insult is required to provoke AKI. There are supportive data from animal models of AKI which suggest that AKI is a form of ‘vasomotor nephropathy’. 22,23 People with CKD and a greater burden of vascular disease may have increased severity of AKI when it develops, which is more likely to be clinically apparent and require hospitalisation and, thus, be captured in epidemiological studies. Patients without CKD, and with less vascular disease, may have less severe AKI, which manifests as ‘silent and discrete’ episodes in the community and may not be captured in existing epidemiological studies, thereby indicating an increased incidence of AKI in CKD.

Further prospective studies are required to assess the true incidence of AKI in patients with CKD and correct more accurately for comorbidity and hospitalisation.

But what happens following AKI? Renal tissue has the ability to recover from sublethal or lethal cellular damage. 24–27 However, function may not be fully restored, which results in the development of CKD. 26 It is suggested that kidney function can be directly related to a cycle of cell injury and recovery following AKI (Figure 1a). 28 This involves renal tubular epithelial cells, damage to which is thought to be extended by renal vascular endothelial injury and dysfunction. It is believed that endothelial repair is important to overall renal recovery and may impact on long-term function. 29 This model, however, considers acute tubular necrosis (ATN) as the cause of AKI. What happens most frequently is limited to the very early part of this process. In patients developing CKD (Figure 1b), the initiating insult leading to damage, inflammation and repair (initiation) may result in fibrosis (extension) and then further damage in a self-perpetuating cycle of progression (maintenance) to ESRD. Early intervention at the stages of initiation and extension may prevent CKD and ESRD, while later intervention during the maintenance stage may only delay progression, with the extent of delay determined by the success or otherwise of intervention. Patients with AKI may or may not have pre-existing CKD (Figure 1c). Okusa et al. 30 (pathophysiological concepts from Sutton et al. 28) suggest that, following AKI, there are four possible outcomes: (1) full recovery; (2) incomplete recovery resulting in CKD; (3) exacerbation of pre-existing CKD accelerating progression to ESRD; and (4) non-recovery of function leading to ESRD. Recovery from AKI may also be incomplete, leading to step-down in glomerular filtration rate (GFR) falling short of CKD. The fact that patients experiencing AKI are likely to also have risk factors for CKD could suggest that patients without a known background CKD who develop AKI already have unrecognised renal disease and reduced functional reserve, which has not yet manifested as CKD. These patients are programmed to develop future CKD, and the AKI episode simply speeds up the development of overt CKD. In this respect, renal outcomes of AKI and CKD are the same, which is further evidence that they are part of the same pathophysiological pathway.

FIGURE 1.

Conceptual model of GFR and cellular pathology over time in AKI, CKD, and AKI and CKD. (a) Model of AKI over time: the cellular phases of AKI leading to repair, highlighting the possibility of initiating a self-perpetuating cycle of inflammation producing fibrosis leading to CKD; (b) model of CKD over time: the phases of cellular injury in CKD. Following an initial insult there is initiation of the inflammatory response with repair. This may then lead to the extension phase with added fibrosis. Past a point of no return, the disease process embarks upon a self-perpetuating cycle of cellular damage and fibrosis (maintenance phase) leading to deterioration in GFR, and progression to ESRD. The figure also shows the effect of intervention on the disease process; and (c) AKI and CKD: the effect of episodes of AKI on the progression of CKD, with three possible outcomes: complete recovery, stepwise progression and inexorable decline. D, damage; F, fibrosis; GFR, glomerular filtration rate; I, inflammation, R, repair.

A key question is whether the ‘I’ in AKI truly stands for injury or actually stands for impairment and/or injury. Is it underpinned by histopathological damage and, if so, when does this become relevant in terms of future CKD or CKD progression? Do undetected episodes of AKI in the community lead to CKD? When patients present with CKD without an obvious cause, is the pathophysiology related to multiple undetected AKI events in the community?

In an ischaemia–reperfusion injury model of AKI in rats, Basile et al. 29 found permanent alterations in renal structure and function associated with the development of features indicative of CKD. They suggest that permanent changes in renal blood flow occur following AKI, resulting in tubulointerstitial fibrosis and altered medullary tonicity (causing impairment of urinary concentrating ability). 29 They also suggest a loss of microvasculature, resulting in a build-up of extracellular matrix, contributing to the development of interstitial fibrosis,29 which in turn leads to development of CKD. They hypothesise that, as long as there is adequate functional reserve, the single-nephron GFR of surviving nephrons increases to maintain a constant total GFR. 29 This suggests that, even in patients in whom creatinine levels and GFR return to baseline, there may be underlying permanent damage masked by compensatory mechanisms. These patients may subsequently have an increased risk of CKD and AKI, owing to underlying ‘subclinical’ damage.

These results are borne out further by studies of renal transplant patients, which demonstrate that delayed graft function (most commonly ATN31) is an independent risk factor for graft survival. 31–33 The kidney has the ability to restore structure and function following AKI, but there are some changes and damage which are permanent. This may lead to development of CKD (or progression of existing CKD) if there is not sufficient functional reserve to compensate. In cases where compensation maintains baseline GFR, there may then be increased risk of future development of CKD. This ‘subclinical’ damage will be important in the management of these patients following AKI, for preventing progression or development of CKD.

Are AKI and CKD biologically part of the same pathway, with eventual glomerulosclerosis and interstitial fibrosis? The discrete episode of AKI leads to fibrosis by setting up the cycle of inflammation and cell repair.

Definition

The first debate surrounding AKI concerned the definition. Over the years, one inherent problem in both diagnosing AKI clinically and reviewing and comparing studies published in the literature has been the numerous definitions used for AKI. 87 The use of these differing definitions in different locations with different populations has only worsened the problem. This also precluded the appreciation of the true problem of AKI in terms of incidence and outcomes.

Risk, Injury, Failure, Loss of function, and End-stage renal failure

In 2003, the Acute Dialysis Quality Initiative group published guidelines to define AKI as a 1.5-fold increase in serum creatinine levels, a decrease in estimated glomerular filtration rate (eGFR) of > 25% or a reduction in urine output to < 0.5ml/kg/hour over 6 hours. 58 They developed the RIFLE (Risk, Injury, Failure, Loss, and End-stage renal failure) classification to define patients by changes in serum creatinine level or urine output criteria. 58 Risk was defined as a 1.5- to 2.0-fold increase, injury as a 2.0- to 3.0-fold increase and failure as a > 3.0-fold increase in serum creatinine level. 58 Loss was defined as a complete loss of kidney function requiring RRT for > 4 weeks and end-stage renal failure as complete loss of kidney function for > 3 months. 58

The Acute Kidney Injury Network

In 2007, the Acute Kidney Injury Network (AKIN) modified the RIFLE criteria, defining three stages of AKI (Table 1): AKI stage 1, equating to the ‘R’ of the RIFLE criteria, with the inclusion of a 1.5-fold or 26.4 µmol/l (0.3 mg/dl) rise in serum creatinine levels; AKI stage 2, representing the ‘I’ of the RIFLE criteria; and AKI stage 3 representing the ‘F’ of the RIFLE criteria. 88 The ‘L’ and ‘E’ (end-stage renal failure) were redefined as outcomes.

The addition of a rise in 26.4 µmol/l to define AKI stage 1 was based on two large studies which demonstrated an independent association between an increase in serum creatinine of 26.4 µmol/l and in-hospital mortality. 34,63 The RIFLE and AKIN are consensus definitions that have now been validated and correlate well with patient outcomes. 89,90

Kidney Disease: Improving Global Outcomes

In the recent KDIGO AKI guideline, AKI is defined as a syndrome, including direct injury to the kidney as well as acute impairment of function. 83

The guideline defines AKI as:

• increase in serum creatinine level of > 0.3 mg/dl within 48 hours, or

• increase in serum creatinine level of > 1.5-fold above the baseline which is known or presumed to have occurred within 7 days, or

• urine volume < 0.5 ml/kg/hour for 6 hours.

The importance of staging AKI (see Table 1) is stressed, as adverse outcomes worsen with increasing stage. 46–49,83,91–93

The definitions of both AKI and CKD are time dependent. For AKI there must be an increase in serum creatinine levels over a period of 2 (AKIN) to 7 (RIFLE) days. For CKD, GFR must be reduced for at least 3 months. These definitions may not capture all cases of AKI and CKD. Certain causes of AKI and CKD may lead to changes in serum creatinine levels and GFR over a time period outside those currently specified, precluding definition. These cases should not be neglected, as intervention may be required. For this reason, the KDIGO AKI Work Group proposed an operational definition for AKD to provide an integrated clinical approach to patients with abnormalities of kidney function and structure, and provide a diagnostic algorithm for defining AKD, AKI and CKD (Table 2). 83

Baseline

Although we now have an internationally agreed on and validated definition of AKI, the time constraints of these definitions raise the big question and primary focus of the present debate in AKI, which concerns how we define the baseline kidney function of a patient. The absolute and relative rises in serum creatinine levels to define AKI are now used widely in clinical practice and research studies, allowing better comparison of data sets. However, what baseline kidney function are these rises from? The AKIN criteria suggest a rise in serum creatinine levels over a period of 2 days, whereas the RIFLE criteria suggest a rise over a period of 7 days. However, a large number of patients presenting acutely to hospital will not have had blood tests in the preceding 2, or in fact 7, days. The question and debate is then twofold: how far back do we look for a baseline kidney function and what value over this time period do we take?

A retrospective cohort study by Lafrance and Miller94 assessed 1,126,636 veterans (US Department of Veterans Affairs healthcare system) who were hospitalised at least once between 2000 and 2005. The highest serum creatinine level during hospitalisation was compared with the lowest using four different baseline periods (in-hospital only, 3, 6 or 12 months before admission). AKI was defined as a rise in serum creatinine level ≥ 1.5 times or an increase of 0.3–0.5 mg/dl over the baseline. 94 The cumulative incidence of AKI ranged from 12.5% (in-hospital baseline) to 18.3% (baseline up to 12 months before admission). By extending the baseline period to at least 3 months, the authors found that the discriminative power increased slightly (the c-statistic increased from 0.846 to 0.855; p = 0.001). They suggested the need for consensus regarding how baseline serum creatinine levels should be determined in database studies.

A clinician’s definition of the baseline kidney function is often achieved through the visualisation of serum creatinine level results graphically represented (Figure 2). The problem with using the lowest serum creatinine levels in the preceding 12 months (which is done to allow a computer algorithm to define AKI from creatinine level results in a structured database) is that a spuriously low result recorded on the pathology database, either from an error or, more often, from an event such as fluid loading (and hence dilution) during a previous hospital admission, will be taken as the baseline kidney function (see Figure 2a). This could, in fact, be significantly lower than the patient’s true baseline kidney function, leading to an incorrect trigger of a diagnosis of AKI on a new blood test.

FIGURE 2.

Errors in defining acute kidney injury as the lowest creatinine in the 12 months prior to the present test. (a) Here, the kidney function (creatinine) can be seen to be stable; however, two spuriously low results, which may be a result of error or more likely fluid loading and dilution may result in an incorrect definition of AKI; (b) here, there has been a stepwise deterioration in kidney function around 4 months previously (likely to be an AKI at that point in time), and since then the kidney function has been stable. However, using a 12-month baseline will continue define this as an acute event; (c) here, a progressive decline in CKD over the course of 12 months. Using the lowest creatinine in 12 months as the baseline will trigger a definition of AKI.

There is also the possibility that a patient with progressive CKD may trigger a diagnosis of AKI based on a baseline defined as the lowest serum creatinine level in the preceding 12 months, when actually the kidney function has slowly deteriorated over the 12-month period, but comparison of the present creatinine level with that of 12 months prior triggers a diagnosis of AKI (see Figure 2c). In the same way, within a 12-month period a patient may have a stepwise reduction in kidney function (probably owing to an AKI) and hence an increase in creatinine that then remains stable. However, a new serum creatinine test, although at the same level as the previous number of months, may be higher than the stepwise increase in creatinine in the past 12 months (see Figure 2b).

In an attempt to tackle some of these issues, other strategies have been suggested: taking the average of values between 7 and 365 days prior to admission;95 back-calculating reference serum creatinine for missing values from an assumed modification of diet in renal disease (MDRD) GFR of 75 ml/minute/1.73 m²,96 and (most recently) a method employing multiple imputation using known comorbidity strengthened by factoring in the lowest admission serum creatinine level. 97

If there are no serum creatinine results in the preceding 12 months, the KDIGO AKI guideline suggests that an estimated creatinine level can be used, provided there is no evidence of CKD. 83 However, there remain cases of CKD in the community that have not been previously appreciated; therefore, estimating baseline creatinine level may lead to diagnosis of AKI in patients with previously unrecognised CKD. These problems with definition make the assessment of AKI and CKD, and their complex interplay, more problematic.

One other point to note is the fact that serum creatinine is a poor biomarker of kidney injury, requiring 48 hours for levels to rise following insult. This emphasises the need for new biomarkers and point-of-care devices to facilitate early identification of patients, which would aid early intervention, as well as more accurate risk assessment to identify patients who may go on to develop AKI.

Although the debate of baseline kidney function in the definition of AKI continues, the now-accepted staging of AKI leads to the following question: when using these accepted definitions, what are the true impacts of AKI?

NHS England patient safety alert

NHS England released a patient safety alert, stage three directive on 9 June 2014, to ensure both that all acute NHS hospital trusts in England are alerting to AKI (by 9 March 2015) and that there is standardisation in the identification of AKI using a single algorithm (www.england.nhs.uk/aki-algorithm). Although similar, our study does not use the same algorithm because it was designed and the analysis was completed before the publication of this directive. In our work following this study we have been using the NHS England algorithm to validate our models so that when we implement the models in clinical practice in the next phase of our work we can ensure standardisation and generalisability.

National Confidential Enquiry into Patient Outcome and Death

From what we already know about AKI, it is apparent that early recognition and effective management of AKI is essential, a concept highlighted in the Renal National Service Framework. However, the 2009 National Confidential Enquiry into Patient Outcome and Death (NCEPOD) in the setting of AKI highlighted systematic failings in identification and subsequent management. 98

The aim of the NCEPOD study was to assess the care of patients who died in hospital with AKI in order to highlight deficiencies in care and provide recommendations for improving clinical management in the future. An advisory group, comprising nephrologists, anaesthetists, intensivists and general physicians, was brought together to review the care of these patients, with a focus on seven main themes:

1. diagnosis and recognition of AKI

2. recognition of risk factors associated with AKI

3. prevention of AKI

4. assessment of patients recognised as having AKI

5. management of established AKI

6. recognition and management of complications of AKI

7. organisational factors relevant to the treatment of AKI.

All NHS hospitals in England, Wales and Northern Ireland were expected to participate. Hospitals in the independent sector and public hospitals in the Isle of Man, Guernsey and Jersey also participated. The inclusion criteria were set as any patient aged 16 years or older who died in hospital between 1 January 2007 and 31 March 2007, inclusive, and who had a coded diagnosis of AKI. Patients were excluded if they were already receiving renal replacement therapy or if their admission was for palliative care from the outset. At each hospital, the NCEPOD local reporter, who acted as a liaison between the NCEPOD and hospital staff, facilitated the identification of these cases within the inclusion criteria, and then facilitated the dissemination of questionnaires and data collation for these patients. For each patient identified, a clinical questionnaire was sent to the clinician responsible for the patient’s care at the time of death and an organisational questionnaire was sent to each hospital. Photocopies of the patient’s case notes were also made and sent to the NCEPOD. The case notes and questionnaires were anonymised before the advisory group reviewed each case.

In total, 1518 patients from 215 hospitals met the inclusion criteria. Of these, 473 were subsequently excluded either for not being indicative of AKI or because the admission was for palliative care at the outset. In a further 69 cases, the case notes were reported as being lost or the consultant in charge of the patient at the time of their death had left the trust. This left 976 patients. A clinical questionnaire and/or case notes were received for 700 patients (72%).

Of the 700 patients, half were from the specialties of general medicine and elderly care medicine. In 14% of patients, it was the clinician’s opinion that the AKI was avoidable. In an overall assessment of care, only 50% of patients were assessed to have received a ‘good’ standard of care and, importantly, in the majority of cases in which the care was considered less than good, there was judged to be room for improvement in the clinical care rather than at an organisational level. This suggests inadequacies in the clinicians’ recognition of AKI and of its subsequent management.

Complications of AKI were missed in 13% of patients and, importantly, in 17% of patients the advisors concluded that the complications of AKI were avoidable. In 22%, the complications were managed badly. In relation to the management and assessment of AKI, the advisors found that there was an adequacy of investigation of AKI in only 67% of patients. The advisors also concluded that in one in six cases there was a failure to recognise the severity of the illness.

In patients who developed AKI post admission, one-fifth were deemed to have been predictable and avoidable, and in 43% of cases there was judged to have been an unacceptable delay in recognising AKI.

Although the NCEPOD report does have its limitations, notably the dependency on clinical coding and its inherent inaccuracies to define AKI, and a patient population in which the outcome in each case was death, the conclusions are very clear: there are currently significant deficiencies in the recognition and clinical management of patients with AKI. The NCEPOD report recommends risk assessment for AKI in all emergency admissions to hospital, and suggests that predictable and avoidable AKI should never occur. 98

The NCEPOD report, with its clear conclusions, has been a key factor in the growing impetus of and focus on AKI and the fact that AKI is now under the spotlight. This also leads to the question of how we can improve the clinical management of AKI. Although strategies can be put in place to alert clinicians to the presence of AKI and allow early intervention to improve clinical outcomes, as with any disease process, our ultimate aim should be prevention.

Before exposure to an insult

In clinical medicine there are certain procedures or treatments that a patient may experience as part of their clinical management that are essential in their care, but carry an inherent risk of precipitating AKI. Some insults may not be avoidable and in these cases the patient’s care can be optimised to reduce the risk of AKI. In some cases, when the risk of AKI is fully appreciated it may outweigh the benefit of the procedure or treatment. In other cases, although the benefit of a procedure or treatment may outweigh the risk, modifications in dosing or exposure may reduce the risk of AKI.

Risk assessing the patient in the context of the expected exposure to the insult can allow appropriate decisions to be made regarding the risk benefit of the procedure or treatment, modifications in dosing and exposure, and clinical optimisation prior to the procedure.

After exposure to an insult

After a patient is exposed to an insult, whether this is iatrogenic or the presenting disease of the patient, as it takes up to 48 hours for the creatinine level to rise in AKI, it may not be immediately apparent. In such cases we can predict which patients are at risk of developing AKI from the insult they have sustained. In these patients, we can provide clinical intervention in terms of both making management changes (Box 1) to aid prevention of AKI following the insult and ensuring that repeat kidney function checks are carried out to monitor for AKI.

On development of acute kidney injury

When we know a patient has AKI, as shown by blood testing, this is not the end of the story. At this point we have the opportunity to intervene to make management changes [often the same as prior to the development of AKI (see Box 1)] to effectively manage the AKI and prevent both the worsening of AKI and the development of sequelae including morbidity and mortality. By risk assessing patients at this point and identifying patients at high risk of worsening AKI or resultant morbidity and mortality, we can focus clinical and specialist care on these patients.

After recovery from acute kidney injury

Following an episode of AKI, a patient may have complete recovery of their kidney function; however, as described Outcomes of acute kidney injury, in some patients this recovery may not be complete, resulting in new CKD or the progression of pre-existing CKD. Patients who have experienced an episode of AKI may also be at increased risk of morbidity, including cardiovascular disease, and mortality in the future.

In this case, risk modelling may be important to highlight patients who require follow-up to diagnose or manage resultant CKD, and other resultant morbidity, following an episode of AKI.

Defining risk in clinical practice

If each patient’s risk of AKI, or worsening AKI if already present, can be defined, then clinicians can be alerted to these patients and management changes and interventions can be put in place in order to prevent, or at least reduce, the risk of the patient developing AKI or worsening AKI.

In any disease process the ultimate treatment is prevention. In the disease process of AKI, risk factors include pre-existing comorbid disease, the presenting illness and also the treatment given for this illness. From these risk factors we can attempt to define a patient’s risk.

In the literature, the majority of the reports focus on the need for RRT after cardiac surgery. One of the first of these was by Chertow et al. ,100 who produced a risk model for predicting AKI after cardiac surgery based on a population of 40,000 patients who underwent cardiac bypass or valvular surgery in 43 Veterans Administration Hospitals in Virginia, USA. A risk stratification algorithm was formulated on the basis of interactions between potential risk factors. There were inherent flaws in the study cohort, specifically a lack of femaled and African American patients. Thakar et al. 101 produced a clinical risk score to predict post-cardiac surgery AKI requiring RRT, based on 33,217 patients who underwent cardiac surgery at the Cleveland Clinic between 1993 and 2002. The scoring system was derived based on 13 preoperative factors, which were weighted; the sum of the scores, ranging from 0 to 17, allowed for stratification of postoperative risk of AKI from low to high. The lowest-risk group (score of 0–2) had a risk for AKI requiring RRT of 0.4%, in contrast to those in the high-risk stratum (score of 9–13), who had a RRT risk of 21.5%. Table 3 demonstrates the risk factors.

There is, however, a paucity in the literature of studies regarding the risk and risk prediction of AKI in unselected emergency admissions to a district general hospital. Finlay et al. 102 published a recent study of risk factors (Table 4) associated with AKI in patients admitted to acute medical units in a study conducted over two separate 24-hour periods in a total of 10 acute medical units. Forni et al. 103 developed a model for predicting AKI in a subset of medical patients admitted to a UK hospital, which included some physiological markers. Alongside this there have been reports in the literature of the development of AKI alert systems, but none presently demonstrate significant clinical benefit. 104,105

Design and theoretical/conceptual framework

This project involved both quantitative and qualitative methodology. Quantitative methodology was used to (1) formulate the predictive risk model and (2) validate the risk model in the East Kent population, and a second population and a NHS trust (Medway NHS Foundation Trust). Qualitative methodology was employed to plan the CDSS development and facilitate effective integration of the CDSS into everyday clinical care.

Setting/context

For risk model development and validation in the first population the study population comprised all patients presenting to the three acute hospitals of East Kent Hospitals University NHS Foundation Trust (EKHUFT; Kent and Canterbury Hospital in Canterbury, William Harvey Hospital in Ashford, and Queen Elizabeth the Queen Mother Hospital in Margate) in the calendar year of 2011, excluding maternity admissions and elective admissions. The renal tertiary referral centre is based at Kent and Canterbury Hospital. The secondary validation population included all patients presenting to Medway NHS Foundation Trust over the same time period and with the same exclusions.

Research governance

Public and patient involvement

We had public and patient involvement in the study design process; however, as this study was a mathematical analysis of retrospective data there was no public and patient involvement during the study.

Data collection

Prior to study commencement and funding applications, an assessment had been made of the data set available for the study in terms of database access and variables available. Following commencement of the study, interfaces were developed with the following clinical systems/databases to enable data extraction.

Data analysis

The main aim of this study was the development of predictive models for identifying and stratifying the risk of AKI at the point of, and during, hospital admission. These models included a large set of potential risk factors identified from secondary care records, as well as admission characteristics of each patient. Both traditional and Bayesian modelling techniques were assessed in order to develop these models. We will describe these methods separately.

Traditional model development

Traditional model validation

Bayesian model development

As the primary variable consisted of a nominal variable with 19 categories, it was transformed into 19 dummy variables, one for each category/type of primary diagnosis. This allowed us to independently access the effect of each diagnosis on the response variable.

We followed the usual process of all other Bayesian analyses; that is, we worked out the likelihood function of the data, placed a prior distribution over all of the unknown parameters and used the Bayes theorem to calculate the posterior distribution over all parameters. The Bayesian framework should more naturally allow for modelling biases and systematic errors, and does not suffer from the large sample constraint (i.e. we do not need to worry about how large our sample should be in order to be able to carry out inference on the parameters). The results of a Bayesian analysis are direct statements about the quantities of interest: in our case, the probability of developing AKI.

The likelihood used here matches that of the classical approach; therefore, the likelihood contribution from the fifth episode is:

where Y is the binary response variable, X₁, X₂, . . . X_p are the predictor variables, and b₀, b₁, . . ., b_p are the unknown predictor coefficients (the unknown parameters). As the episodes are assumed to be independent of each other, the likelihood function over the whole data set will then be the nth product of the above equation.

At the inception of the project we wanted to place an informative prior on the unknown coefficients. However, the study of predicting AKI is in its infancy, particularly with respect to Bayesian methods, and a strong opinion as to what characteristics the prior distribution should have could not be agreed; the approach for variable selection problems is to employ a normal prior. We therefore selected a normal distribution prior for the unknown coefficients and within that incorporated the stochastic search variable selection (SSVS) approach described in George and McCullogh. 107 The SVSS approach specifies a hierarchical mixture prior that in turn is informed by the data to assign a larger posterior probability to the subset of predictors that are more likely to have an effect on the response. The most promising subsets of predictors can be identified as those with higher posterior probability. The Gibbs sampler was then used to indirectly sample from this posterior distribution on the set of possible subset choices. The most promising predictors can be identified by their more frequent appearance in the Gibbs sample. This means that the major advantage of SSVS is the fact that it allows for model averaging while avoiding the problem of calculating probabilities for all 2p subsets (p is the number of predictors). The complete set-up of our Bayesian logistic regression SSVS approach is as follows:

The latent variable is gj and defines if the jth variable is included in the model. A criticism of this approach is that it is necessary to choose the value of s2. If s2 is too large then we are faced with the danger of overfitting. As we have standardised the continuous variables, we chose s2 = 1. We have assumed a priori that five variables should be in the model, adhering to the beliefs of the clinical team. This then implied that p = 5. We feel that this is a reasonable choice as we want to avoid hj being too large.

Bayesian model validation

To assess the predictive accuracy of the models, we randomly divided the data sets in half: half was used for in-sample estimation/development and the other half used for out-of-sample prediction. The out-of-sample prediction was made in terms of sensitivity and specificity; that is, we calculated the probability that the model accurately predicts episodes with AKI and the probability that the model accurately predicts episodes without AKI.

Although there seem to be large numbers of missing data in the analysis, this appears to be attributable to failure to conduct a specific test rather than to data being missing in the usual respect. In order to explore the impact of these data and the fact that they were not missing at random, we created dummy variables indicating the absence or presence of a particular test result and explored the impact of this on the outcomes as a sensitivity analysis.

Sample

The sample consisted of six renal consultants for the individual interviews and six outreach nurses who attended the focus group. All of the consultants worked across the three hospitals within the trust, and there was representation from all hospitals from the outreach nurses; however, the nurses did not work across all three sites. The consultants used the alert system to identify patients with the more serious AKI stage 3 and alert medical teams to offer advice and review if required, and the outreach nurses identified AKI stage 2 patients and provided clinical review of the patients, whose condition was more amenable to preventative action, on the ward.

In terms of the accessibility of the consultant and nurse groups, it was established early on that different qualitative approaches would need to be used on account of their individual availability and potential to meet together. The consultants had to cover three sites and group meetings were difficult to convene; therefore, interviews were the method of choice. Conversely, the outreach nurses were difficult to capture individually but attended a training session as a group once per month, which provided an opportunity for a focus group. Tuning different methods to the requirements of the population group, in order to maximise attendance and enrich data, is considered to be good practice. 108

Instrumentation and data collection

For the focus group, a schedule was developed that explored perceptions of the impact on clinical practice, aiming to identify the best methods for delivering the alerts and recommendations. This covered aspects such as accessibility of information, hardware, whom the recipient should be (junior doctor or consultant), what form the alert should take (additional e-mail or text), how to avoid alert fatigue and alerts being ignored. The focus group lasted 1 hour and was facilitated by an experienced researcher from a university external to the clinical team. Interviews took place at a convenient time and location, and were carried out by the same experienced researcher.

The template and coding framework for both the focus group and the interviews has been included in Appendix 3.

Data analysis

Data consisted of six 30-minute interviews and a 1-hour focus group. The data were transcribed verbatim and subjected to a thematic analysis using a predetermined theme, derived from the interview schedule, as an initial template for analysis. Clinical team members were not involved at all in the data collection or analysis of the data; this was undertaken by a researcher from a university external to the clinical team. Regarding the interview and focus group schedules, sections 1, 2 and 4 are identical. Section 3 has been altered to account for the differing clinical roles in the project and communication experiences (i.e. with different people at different times) regarding exposure to the project. There are, however, similarities, as both sections focus on communication with teams, changes to clinical practice and impacts. Although it is accepted that data obtained from focus groups are influenced by group dynamics and consensus, data sources from different qualitative methodological approaches can be blended and contrasted, provided that they are at first analysed thematically in a separate manner. An overarching, thematic pattern matching can be used to determine an explanation of the data, which is the method used here. 109

The analytical approach taken was Flick’s content analysis,110 whereby themes and subthemes were categorised within a pre-existing template (within the instrumentation). With this approach, however, care must be taken not to artificially represent data within the template but to introduce new themes when identified. This approach required peer review to ensure analytical trustworthiness, which was conducted within the research team. The analysis from the focus group concentrated on the identification of best methods for delivering the alerts and recommendations in order to inform the AKI and AKI risk alert system development and implementation in the follow-up study.

We would aim to have two further focus group waves in our next clinical intervention study to deliver AKI and AKI-risk alerting to clinicians at the point of care. The qualitative analysis described here is, therefore, part of a larger longitudinal qualitative research design. Longitudinal qualitative research involves repeat interviews or observations of, ideally, the same research subjects over time. 111 In recent years, longitudinal qualitative research has been used in a number of health-related areas to generate rich data and a deeper understanding of people’s perspectives and experiences and how and why these may change over time in order to improve practice. 112–115 Rather than comparing findings at a number of distinct moments, longitudinal qualitative research is concerned with the comparison of different, continuous processes of change.

The benefits of the longitudinal design will be that it will permit the same participants to be involved in identifying practice challenges and solutions, in developing methods for how alerts and recommendations can be best delivered for action, and for examining and reflecting on the effects with regard to practice change as well as system evaluation and improvement. The qualitative aspect of both this and the follow-on study will strengthen the production of potentially transferable practice guidelines and system accessibility across the NHS.

The focus group method is a favoured approach in the health-care setting not only for expanding ideas, but also for gaining consensus on views and promoting good practice. 108 This particular research study promotes new and innovative ideas that may benefit from being explored within a group, particularly with a longitudinal approach. Variations in perception and experience will encourage deeper discussion and illuminate impacts, as well as reveal the nature and cause of practice changes in relation to the intervention.

Points of decision-making

The study was designed to develop and validate risk models to define risk of AKI or of worsening AKI during hospital admission. We determined three time points during the period of admission when significant clinical decision-making takes place at which the use of risk models would have the greatest impact on clinical care and patient management. These time points are described below.

Clinical alerting

The purpose of the risk models is to guide clinical management. This will be achieved through alerting clinicians at the point of care. These systems have been developed as part of this project, but are awaiting formal assessment in a trial setting. The clinical alerting system will not solely alert to risk of AKI; it will also alert to patients with established AKI and provide clinical guidance for improving the management of these patients and reducing both the progression of AKI and the development of resultant sequelae. It is therefore important that both alerting to AKI and alerting to AKI risk are brought together in a clinical practice algorithm to guide both the alerting and the clinical management pathway following the alert.

Alerting after 24 hours of admission

For patients who have AKI on admission, the clinical alerting system will suggest appropriate management interventions as above listed (see Points of decision-making), including daily renal function testing. Again, as part of the clinical practice algorithm, patients with AKI on admission were removed from the risk models at 24 hours to predict AKI at 72 hours, as these patients have already been highlighted by the system. Moreover, including AKI on admission as a variable in a model to predict AKI at 72 hours would be highly predictive, dilute the other variables and reduce the clinical utility of the system, especially as these patients will already be flagged up by the system anyway.

This led to the development of the clinical practice algorithm (Figure 5) to guide:

• AKI alerting

• patients to be assessed by the risk models at the point of admission and at 24 hours after admission

• appropriate renal function testing

• appropriate management interventions.

FIGURE 5.

Acute kidney injury clinical practice algorithm. A&E, accident and emergency.

Risk model 1: the point of admission to hospital

Development

After exclusions from the full database (as described in Chapter 2), three-quarters of the data set was randomly selected for the development database. This consisted of 27,532 admissions with available AKI-on-admission data. These admissions occurred in 20,330 different patients.

A series of separate univariable ordinal logistic regression analyses were performed to examine factors associated with AKI on admission. A summary of the analysis results is given in Appendix 3, Table 32. The sizes of effects are reported in the form of odds ratios. The odds ratio is calculated as the exponential of the parameter estimates (beta). For categorical variables, these give the odds of being in the next highest outcome category (e.g. AKI stage 1 rather than no AKI, AKI stage 3 rather than AKI stage 2) for each category relative to a baseline category. For the continuous predictor variables, the odds ratios represent the relative change in the odds of being in the next highest category for a given increase in that predictor. The p-value indicating the overall significance of each variable is also reported.

The results in Appendix 3, Table 32, are interpretable for the categorical predictor variables. The results for the continuous variables are less interpretable, and the results for these variables are best showed graphically.

The graph in Figure 6 shows the relationship between age and the probability of AKI. For graphical purposes, the occurrence of any AKI is plotted.

FIGURE 6.

Risk model 1: relationship between age and probability of AKI in the univariable analysis.

A similar graph for baseline eGFR is shown in Figure 7.

FIGURE 7.

Risk model 1: relationship between baseline eGFR and probability of AKI in the univariable analysis.

The next stage in the analyses examined the joint association between the variables and AKI stage in a multivariable analysis.

The collinearity between the different predictors was examined. The examinations suggested collinearity between platelet count (PLT), white blood cell count (WBC) and haemoglobin (Hb) levels. Additionally, collinearity was found between potassium (K) and sodium (Na). The nature of the collinearity was that these tests were performed on the same patients, and so the ‘not measured’ category was almost equivalent where collinearity was found.

Before the main multivariable analysis was performed, an additional analysis was performed, examining the association between PLT, WBC and Hb levels on AKI stage on the subgroup of patients with all tests performed. The analyses suggested that only Hb was statistically significant, with WBC and platelets not significant after adjusting for Hb. As a result, only Hb was considered for the multivariable analysis. A similar analysis was performed for K and Na in the subgroup where these variables were measured. This analysis suggested that both were independently associated with AKI stage. As a result, a combined variable was derived, consisting of the following categories: one/both tests not measured, both tests normal, Na only abnormal, K only abnormal and both tests abnormal.

A backwards selection procedure was used to retain the statistically significant variables in the final model. This procedure omitted the following variables (in order of removal):

• wound swab/fluid aspirate

• combined Na/K

• calcium

• gender

• faeces

• sputum

• number of outpatient appointments.

The final model is summarised in Table 5.

TABLE 5 - Risk model 1: results of the multivariable ordinal logistic regression analysis to examine factors associated with AKI on admission

CI, confidence interval; CRP, C-reactive protein; CSU, catheter specimen urine; HbA_1c, glycated haemoglobin; MSU, mid-stream specimen urine.

Odds ratio given for a 10-unit increase in age.

Normal range 13–18 g/dl for males.

Odds ratio given for a 20-unit increase in baseline GFR.

Number of the following drugs taken: angiotensin-converting enzyme inhibitors, non-steroidal anti-inflammatory drugs and diuretics.

Variable	Category/term	Odds ratio (95% CI)	p-value
Agea	Linear term	3.35 (1.82 to 6.16)	< 0.001
	Quadratic term	0.89 (0.81 to 0.98)
	Cubic term	1.003 (0.998 to 1.009)
Admissions (last 30 days)	No	1	< 0.001
	Yes	0.82 (0.74 to 0.91)
Admissions (last 2–12 months)	0	1	< 0.001
	1–2	1.28 (1.16 to 1.42)
	3–5	1.54 (1.36 to 1.75)
	6+	2.01 (1.70 to 3.80)
	6+	1.67 (1.52 to 1.84)
Primary diagnosis	Neoplasms	1	< 0.001
	Infectious diseases	1.37 (1.05 to 1.80)
	Blood diseases	0.50 (0.35 to 0.72)
	Circulatory system	0.66 (0.55 to 0.80)
	Digestive system	0.83 (0.68 to 1.02)
	Diseases of head/neck	0.30 (0.15 to 0.63)
	Genitourinary system	1.76 (1.43 to 2.15)
	Musculoskeletal	0.39 (0.30 to 0.49)
	Nervous system	0.53 (0.39 to 0.72)
	Respiratory system	0.90 (0.75 to 1.09)
	Skin	0.87 (0.68 to 1.16)
	Endocrine/metabolic	1.64 (1.26 to 2.13)
	Injury/poisoning	0.55 (0.45 to 0.67)
	Mental disorders	0.79 (0.55 to 1.12)
	Symptoms/signs	0.53 (0.44 to 0.64)
	Other	0.36 (0.16 to 0.79)
CRP	Not measured	1	0.009
	Normal (≤ 10 mg/l)	0.99 (0.86 to 1.14)
	Abnormal	1.19 (1.04 to 1.35)
Hb	Not measured	1	< 0.001
	Normal (female: 11–15 g/dl)b	0.79 (0.70 to 0.89)
	Abnormal	1.10 (0.98 to 1.24)
HbA1c (12-month average)	Not measured	1	0.008
	Normal (≤ 7.5%)	1.06 (0.96 to 1.17)
	Abnormal	1.24 (1.08 to 1.43)
Troponin	0	1	< 0.001
	1	1.27 (1.11 to 1.45)
	2+	1.46 (1.15 to 1.85)
Blood culture	Not taken	1	< 0.001
	Not significant	1.49 (1.29 to 1.73)
	Significant	3.12 (2.36 to 4.12)
Charlson score	≤ 0	1	< 0.001
	1–10	1.08 (0.97 to 1.21)
	11+	1.37 (1.23 to 1.53)
	Not recorded	1.11 (0.96 to 1.29)
Baseline eGFRc	Linear term	0.93 (0.60 to 1.43)	< 0.001
	Quadratic term	0.83 (0.74 to 0.93)
	Cubic term	1.03 (1.02 to 1.04)
Drugs takend	0	1	< 0.001
	1	1.15 (1.04 to 1.27)
	2 or 3	1.38 (1.20 to 1.58)
	Not recorded	0.86 (0.77 to 0.95)
MSU or CSU	Not taken	1	< 0.001
	Not significant	1.05 (0.80 to 1.36)
	Significant	1.37 (1.19 to 1.59)
Proteinuria (worst result)	Not done	1	< 0.001
	1	1.07 (0.96 to 1.19)
	2 or 3	1.38 (1.23 to 1.54)

The results suggested that most factors had a smaller effect after adjustment for the other factors in the model. This is to be expected, as there will be correlations between many of the risk factors. However, the results for the majority of predictors showed a similar effect to the results from the univariable analyses.

The exception was for an admission in the last 30 days. In the univariable analyses patients with an admission in this time period had a higher AKI stage; however, after adjustments for the other factors these patients had a lower status.

The results for the continuous variables (age and baseline GFR) are again difficult to visualise from the odds ratios alone. The adjusted predicted probabilities for age are shown in Figure 8 (assuming ‘average’ values for all other variables). This graph shows a broadly similar relationship for age to that seen in the univariable analyses, although the increased risk peaks around the age of 75 years.

FIGURE 8.

Risk model 1: relationship between age and probability of AKI in the multivariable analysis.

The adjusted relationship for eGFR is shown in Figure 9. After adjusting for the other variables, it seems that those with the highest values have a lower risk, around the same risk as those with the lowest eGFR values. The peak risk is around 100.

FIGURE 9.

Risk model 1: relationship between baseline eGFR and probability of AKI in the multivariable analysis.

Validation

After exclusions from the full database (as described earlier), one-quarter of the data were randomly selected for the validation database. This consisted of 9177 admissions. However, owing to a small number of missing values, predicted probabilities were obtained for 9157 admissions.

First, the development and validation data sets were compared, with the finding that the two data sets were very matched in terms of the model variables. This was not surprising given the selection of the two data sets at random.

The first set of analyses split the patients into risk groups based on the predicted probability of AKI. The ‘expected’ risk of AKI in each category based on the predicted probabilities was calculated, along with the observed occurrence of AKI in each category. Analyses were performed based on the probability of any AKI, and then again based on the predicted probability of either AKI stage 2 or AKI stage 3. The results are summarised in Table 6.

TABLE 6 - Risk model 1: a comparison of the expected with the observed probabilities of AKI in the different risk groups for AKI on admission

Categorisation	Risk category (%)	n	Mean expected (%)	Observed (%)
Any AKI	≤ 10	3325	6.2	5.0
	> 10–20	3057	14.3	14.8
	> 20–40	2178	27.5	28.4
	> 40	597	51.7	51.6
AKI stage 2/3	≤ 2	3962	1.2	0.8
	> 2–5	3358	3.2	3.8
	> 5–10	1294	6.9	7.2
	> 10	543	15.8	11.4

The results suggested reasonably good discrimination between the risk groups for both categorisations. For both models the observed results increased with increased risk, with the categories correctly ordered.

The calibration of the models was also good, with the observed percentages falling within the risk boundaries for both categorisations. In addition, there was a fairly good agreement between the predicted percentages and those observed by the data. The slight exception was for the highest risk subjects for the AKI stage 2/3 categorisation, in whom the observed percentage of AKI stage 2/3 cases was lower than that predicted by the model.

The next set of analyses further examined the discrimination of the two categorisations using ROC curves based on the predicted probabilities. A summary of the AUROC values is given in Table 7.

TABLE 7 - The ROC analyses for the validation of risk model 1

CI, confidence interval.

Categorisation	AUROC (95% CI)	Interpretation
Any AKI	0.75 (0.74 to 0.77)	Fair
AKI stage 2/3	0.75 (0.73 to 0.78)	Fair

The ROC curve analyses suggested that the model had some discriminatory power, although the AUC values were only ‘fair’.

The ROC curve for any AKI is shown in Figure 10.

FIGURE 10.

Risk model 1: ROC curve for the prediction of any AKI.

Figure 11 shows the ROC curve for the prediction of AKI stages 2 and 3.

FIGURE 11.

Risk model 1: ROC curve for the prediction of AKI stages 2 and 3.

The final analyses compared the observed numbers of patients with AKI in each AKI group and the numbers predicted by the model using the Hosmer–Lemeshow test. A summary of the results is given in Table 8.

TABLE 8 - A comparison of the observed numbers of patients with AKI in each AKI group and the numbers predicted by the model using the Hosmer–Lemeshow test for the validation of risk model 1

Or AKI stage 2/3 for second categorisation.

With 2 degrees of freedom.

Categorisation	Risk category (%)	No AKIa (n)	AKIa (n)	χ2b	p-value
		Observed (expected)	Observed (expected)
Any AKI	≤ 10	3158 (3119)	167 (206)	9.4	0.009
	> 10–20	2606 (2619)	451 (438)
	> 20–40	1559 (1580)	619 (598)
	> 40	289 (289)	308 (308)
AKI stage 2/3	≤ 2	3931 (3916)	31 (46)	16.4	0.0003
	2–5	3232 (3250)	126 (108)
	5–10	1201 (1205)	93 (89)
	> 10	481 (457)	62 (86)

The results suggested that for both models there was a significant difference between the observed numbers and those expected by the model. This suggests some lack of fit of the model to the data. The fit of the model for any AKI is slightly better than that for AKI stage 2/3.

For any AKI, the model overpredicts the number of cases of AKI in the lowest-risk group, but provides a good prediction in the upper risk group. For the AKI stage 2/3 predictions, the model overpredicts the number of AKI stages 2 and 3 cases in the upper risk group, as seen in Table 6.

Despite the test suggesting a lack of fit to the data, this test is known to be fairly sensitive to slight deviations between observed and expected frequencies, and so a statistically significant difference often occurs.

Validation in a second population

The same exclusions were applied to the Medway data set as to those applied to the East Kent data for the admission AKI model (risk model 1). After excluding certain episodes, the data set consisted of 4726 admissions. Predicted probabilities of AKI were obtained for all admissions.

Initially, the East Kent data set (the development data set only) and Medway data sets were compared for the admissions suitable for inclusion in the admission model. This revealed some differences between the two sets of data, as AKI was more common in the Medway data set, occurring in approximately 23% of patients, compared with 17% in East Kent. There were also differences in a number of the patient characteristics that were included in the model.

The first set of analyses looked at the occurrence of AKI in each category. Analyses were performed on patients split into risk groups based on the predicted probability of AKI. The ‘expected’ risk of AKI in each category was calculated based on the predicted probabilities, along with the observed probability of any AKI, and then again based on the predicted probability of either AKI stage 2 or AKI stage 3. The results are summarised in Table 9.

TABLE 9 - Risk model 1: a comparison of the expected with the observed probabilities of AKI in the different risk groups for AKI on admission in the second population

Categorisation	Risk category (%)	n	Mean expected (%)	Observed (%)
Any AKI	≤ 10	1961	6.1	10.7
	> 10–20	1597	14.1	23.0
	> 20–40	960	27.1	39.8
	> 40	208	51.1	63.0
AKI stage 2/3	≤ 2	2329	1.1	2.1
	> 2–5	1657	3.2	5.9
	> 5–10	552	6.8	9.6
	> 10	188	15.3	18.6

The results suggested reasonably good discrimination between the risk groups for both categorisations. For both models the observed results increased with increased risk, with the categories correctly ordered.

The calibration of the models was less good, with the observed outcomes in each category higher than those ‘expected’ by the model. In some instances the observed percentage in each category was outside the range of the risk category. For example, for all AKI, in those patients with a predicted risk of 10–20% the observed percentage with AKI was 23.0%, which was higher than the upper boundary of the prediction.

The next set of analyses further examined the discrimination of the two categorisations using ROC curves based on the predicted probabilities. A summary of the AUROC values is given in Table 10.

TABLE 10 - The ROC analyses in the second population for the validation of risk model 1

CI, confidence interval.

Categorisation	AUROC (95% CI)	Interpretation
Any AKI	0.72 (0.71 to 0.74)	Fair
AKI stage 2/3	0.71 (0.68 to 0.75)	Fair

The ROC curve analyses suggested that the model had some discriminatory power, although the AUC values were only ‘fair’. These values compare with values of 0.75 (for both categorisations) that were obtained from the internal validation with the East Kent data.

The ROC curve for any AKI is shown in Figure 12.

FIGURE 12.

Risk model 1: ROC curve for the prediction of any AKI in the second population.

Figure 13 shows the ROC curve for the prediction of AKI stages 2 and 3.

FIGURE 13.

Risk model 1: ROC curve for the prediction of AKI stages 2 and 3 in the second population.

The final analyses compared the observed numbers of patients with AKI in each AKI group and the numbers predicted by the model using the Hosmer–Lemeshow test. A summary of the results is given in Table 11.

TABLE 11 - A comparison of the observed numbers of patients with AKI in each AKI group and the numbers predicted by the model in the second population using the Hosmer–Lemeshow test for the validation of risk model 1

Or AKIN2/3 for second categorisation.

With 2 degrees of freedom.

Categorisation	Risk category (%)	No AKIa (n)	AKIa (n)	χ2b	p-value
		Observed (expected)	Observed (expected)
Any AKI	≤ 10	1752 (1842)	209 (119)	269.6	< 0.0001
	> 10–20	1229 (1372)	368 (225)
	> 20–40	578 (700)	382 (260)
	> 40	77 (119)	131 (106)
AKI stage 2/3	≤ 2	2280 (2302)	49 (27)	66.9	< 0.0001
	> 2–5	1560 (1604)	97 (52)
	> 5–10	499 (515)	53 (37)
	> 10	153 (159)	35 (29)

The results suggested that for both analyses there was a strong overall lack of fit for the models to the validation data set. In both instances the number of cases of AKI and AKI stage 2/3 was higher than that which might be expected by the model. This mirrors the results observed earlier.

Risk model 2: predicting new acute kidney injury at 72 hours

Development

After exclusions from the full database (as described in Chapter 2), the data consisted of 10,075 admissions. Of these, three-quarters were randomly selected for the development database, resulting in 7556 admissions for the development of the model. These admissions occurred in 6626 different patients.

A series of separate univariable ordinal logistic regression analyses were performed to examine factors associated with AKI at 72 hours. A summary of the analysis results is given in Appendix 3, Table 33. The size of effects is reported in the form of odds ratios. For categorical variables, these give the odds of being in the next highest outcome category (e.g. AKI stage 1 rather than no AKI, AKI stage 3 rather than AKI stage 2) for each category relative to a baseline category. For the continuous predictor variables, the odds ratios represent the relative change in the odds of being in the next highest category for a given increase in that predictor. The p-value indicating the overall significance of each variable is also reported.

The results in the above table are interpretable for the categorical predictor variables.

A linear term only was found to be sufficient for age. A 10-year increase in age was associated with a 28% increase in the odds of AKI being in the next highest category. This result is also shown in Figure 14, which plots age and the probability of AKI. For graphical purposes the occurrence of any AKI is plotted.

FIGURE 14.

Risk model 2: relationship between age and probability of AKI in the univariable analysis.

A quadratic (squared) term was necessary for baseline eGFR, and thus the odds ratios are difficult to interpret and the relationship is best viewed graphically. A graph, similar to that for age, is shown in Figure 15 for baseline eGFR.

FIGURE 15.

Risk model 2: relationship between baseline eGFR and probability of AKI in the univariable analysis.

The next stage in the analyses examined the joint association between the variables and AKI stage in a multivariable analysis.

The collinearity between the different predictors was examined. The examinations indicated no strong evidence of collinearity between any of the predictors; therefore, all predictors were considered for the multivariable analyses.

The univariable analyses suggested that patients with normal and abnormal glycated haemoglobin (HbA_1c) (from the 12-month average) had almost equivalent risks of AKI. As a result, for this second stage in the analysis, this variable was recategorised as ‘not measured’ and ‘measured’.

A backwards selection procedure was used to retain the statistically significant variables in the final model. This procedure omitted the following variables (in order of removal):

• calcium

• gender

• B-type natriuretic peptide (BNP) (12-month average)

• creatinine kinase (CK)

• number of outpatients’ appointments in 12 months

• sputum pre admission

• number of contrast radiology scans

• wound swab/fluid aspirate pre admission

• wound swab/fluid aspirate at admission

• Na

• alanine transaminase (ALT)

• mid-stream specimen urine (MSU) or catheter specimen urine (CSU) pre admission

• faeces at admission

• Hb

• sputum at admission

• MSU or CSU at admission

• faeces pre admission

• operation score at 12 hours

• PLT

• admissions within 30 days of current admission

• blood culture

• number of drugs

• amylase (AMY).

The final model is summarised in Table 12.

TABLE 12 - Risk model 2: results of the multivariable ordinal logistic regression analysis to examine factors associated with new AKI at 72 hours

CI, confidence interval; CRP, C-reactive protein.

Odds ratio given for a 10-unit increase in age.

Odds ratio given for a 20-unit increase in baseline GFR.

Variable	Category/term	Odds ratio (95% CI)	p-value
Agea	Linear term	1.78 (1.22 to 2.60)	< 0.001
	Quadratic term	0.97 (0.95 to 1.00)
Admissions (last 2–12 months)	0	1	< 0.001
	1–2	1.41 (1.14 to 1.75)
	3–5	1.58 (1.20 to 2.09)
	6+	2.24 (1.57 to 3.19)
Primary diagnosis	Neoplasms	1	< 0.001
	Infectious diseases	1.53 (0.90 to 2.61)
	Blood diseases	0.58 (0.25 to 1.36)
	Circulatory system	0.77 (0.50 to 1.20)
	Digestive system	0.84 (0.55 to 1.30)
	Genitourinary system	1.52 (0.95 to 2.43)
	Musculoskeletal	0.79 (0.39 to 1.58)
	Nervous system	0.57 (0.23 to 1.38)
	Respiratory system	0.59 (0.38 to 0.91)
	Skin	0.88 (0.46 to 1.72)
	Endocrine/metabolic	0.71 (0.34 to 1.46)
	Injury/poisoning	1.04 (0.67 to 1.60)
	Mental disorders	0.18 (0.02 to 1.32)
	Symptoms/signs	0.78 (0.49 to 1.26)
	Other	1.33 (0.36 to 4.85)
CRP	Not measured	1	0.05
	Normal (≤ 10 mg/l)	0.89 (0.65 to 1.22)
	Abnormal	1.15 (0.85 to 1.55)
HbA1c (12-month average)	Not measured	1	0.03
	Measured	1.26 (1.03 to 1.56)
K	Not measured	1	0.02
	Normal (3.5–5.3 mmol/l)	0.69 (0.50 to 0.94)
	Abnormal	0.86 (0.58 to 1.26)
Magnesium	Not measured	1	0.02
	Normal (0.7–1.0 mmol/l)	0.88 (0.65 to 1.17)
	Abnormal	1.69 (1.14 to 2.50)
Troponin	0	1	< 0.001
	1	1.42 (1.15 to 1.77)
	2+	2.19 (1.67 to 2.87)
WBC	Not measured/normal	1	0.03
	Abnormal	1.18 (1.00 to 1.40)
Charlson score	≤ 0	1	0.05
	1–10	1.14 (0.87 to 1.49)
	11+	1.35 (1.05 to 1.75)
	Not recorded	1.41 (1.02 to 1.95)
Baseline eGFRb	Linear term	0.40 (0.28 to 0.57)	< 0.001
	Quadratic term	1.11 (1.06 to 1.16)
Proteinuria (worst result)	Not done	1	0.001
	1	0.95 (0.73 to 1.22)
	2 or 3	1.52 (1.19 to 1.95)

The final model suggested that a combination of 12 variables could help to predict AKI. A number of these variables [C-reactive protein (CRP) and Charlson score] were only of borderline statistical significance, but were retained in the final model.

The univariable results suggested that only a linear term for age was required. However, after adjusting for the other factors, a quadratic term was found to improve the fit of the model and was, therefore, included. As a result of the inclusion of this term, the results for age are difficult to visualise from the odds ratios alone. The adjusted predicted probabilities for age are shown in Figure 16 (assuming ‘average’ values for all other variables). This graph shows a broadly similar relationship for age to that seen in the univariable analyses, although the increased risk with age tails off for subjects over 80 years.

FIGURE 16.

Risk model 2: relationship between age and probability of AKI in the multivariable analysis.

The adjusted relationship for eGFR is also best viewed graphically, and this relationship is shown in Figure 17. After adjusting for the other variables, the graph suggests that patients with a baseline eGFR of around 90 ml/minute/1.73m² have the lowest risk, with a slight rise in risk for those with higher baseline eGFR values.

FIGURE 17.

Risk model 2: relationship between baseline eGFR and probability of AKI in the multivariable analysis.

Validation

After exclusions from the full database (as described in Chapter 2), one-quarter of the data were randomly selected for the validation database. These consisted of 2519 admissions. However, owing to a small number of missing values, predicted probabilities were obtained for 2514 admissions.

Initially, the development and validation data sets were compared, with the finding that the two data sets were very matched in terms of the model variables. This is not surprising given that the selection of the two data sets was at random.

The first set of analyses split the patients into risk groups based on the predicted probability of AKI. The ‘expected’ risk of AKI in each category based on the predicted probabilities was calculated, along with the observed occurrence of AKI in each category. Analyses were performed based on the probability of any AKI, and then again based on the predicted probability of either AKI stage 2 or 3. The results are summarised in Table 13.

TABLE 13 - Risk model 2: a comparison of the expected with the observed probabilities of AKI in the different risk groups for new AKI at 72 hours

Categorisation	Risk category (%)	n	Mean expected (%)	Observed (%)
Any AKI	≤ 5	709	3.3	4.5
	> 5–10	1023	7.3	7.7
	> 10–20	633	13.7	15.3
	> 20	149	26.9	22.1
AKI stage 2/3	≤ 1	1441	0.6	0.8
	> 1–2	737	1.4	2.6
	> 2–3	210	2.4	3.3
	> 3	126	4.3	2.4

The results suggested reasonably good discrimination between the risk groups for any AKI. For the prediction of this outcome, the observed results increased with increased risk, with the categories correctly ordered.

The discrimination for the prediction of AKI stage 2 or AKI stage 3 was less impressive. AKI stage 2/3 was less common in the lowest-risk group. However, there was a similar occurrence of AKI stage 2/3 in the three higher-risk groups, with this outcome occurring slightly more frequently in the second highest-risk group than in the highest-risk group. However, it should be noted that AKI stage 2/3 was relatively rare in this patient group and the group sizes were relatively small for the higher-risk groups.

The calibration of the model for the prediction of AKI was good, with the observed percentages falling within the risk boundaries. In addition, there was a fairly good agreement between the predicted percentages and those observed by the data.

The calibration for the prediction of AKI stage 2/3 was less good, with an observed occurrence of AKI stage 2/3 in the 1–2% and 2–3% categories above the risk category boundary. Additionally, the observed occurrence of AKI stage 2/3 for the > 3% category was below the category boundary; however, it is noted that the risk boundaries were relatively tight for the prediction of this outcome.

The next set of analyses further examined the discrimination of the two categorisations using ROC curves based on the predicted probabilities. A summary of the AUROC values is given in Table 14.

TABLE 14 - The ROC analyses for the validation of risk model 2

Categorisation	AUROC (95% CI)	Interpretation
Any AKI	0.67 (0.64 to 0.71)	Poor
AKI stage 2/3	0.68 (0.61 to 0.76)	Poor

The ROC curve analyses suggested that the model had there was some discriminatory power, with the lower bounds for the confidence intervals (CIs) higher than 0.5 for both categorisations. Despite the model having some predictive ability, the AUC values were classed as ‘poor’ (although approaching ‘fair’).

The ROC curve for any AKI is shown in Figure 18.

FIGURE 18.

Risk model 2: ROC curve for the prediction of any AKI.

Figure 19 shows the ROC curve for the prediction of AKI stages 2 and 3.

FIGURE 19.

Risk model 2: ROC curve for the prediction of AKI stages 2 and 3.

The final analyses compared the observed numbers of patients with AKI in each AKI group and the numbers predicted by the model using the Hosmer–Lemeshow test. A summary of the results is given in Table 15.

TABLE 15 - A comparison of the observed numbers of patients with AKI in each AKI group and the numbers predicted by the model using the Hosmer–Lemeshow test for the validation of risk model 2

Or AKIN2/3 for second categorisation.

With 2 degrees of freedom.

Categorisation	Risk category (%)	No AKIa (n)	AKIa (n)	χ2b	p-value
		Observed (expected)	Observed (expected)
Any AKI	≤ 5	677 (685)	32 (24)	6.4	0.04
	> 5–10	944 (949)	79 (74)
	> 10–20	536 (546)	97 (87)
	> 20	116 (109)	33 (40)
AKI stage 2/3	≤ 1	1430 (1432)	11 (9)	10.4	0.005
	> 1–2	718 (727)	19 (10)
	> 2–3	203 (205)	7 (5)
	> 3%	123 (120)	3 (5)

The results indicate that for both models there was a significant difference between the observed numbers and those expected by the model. This suggests some lack of fit of the model to the data. The fit of the model for any AKI was better than that for AKI stage 2/3, with the lack of fit only just statistically significant.

For any AKI, the model underpredicts the number of cases of AKI in the three lowest-risk groups, but overpredicts in the upper risk group. The results for the AKI stage 2/3 predictions are similar.

Despite the test suggesting a lack of fit to the data, this test is known to be fairly sensitive to slight deviations between observed and expected frequencies, and so a statistically significant difference often occurred.

Validation in a second population

The same exclusions were applied to the Medway data set as those applied to the East Kent data for the 72-hour AKI model (as described in Risk model 2: predicting new acute kidney injury at 72 hours). After excluding certain episodes, the data set consisted of 1585 admissions. Predicted probabilities of AKI were obtained for all admissions.

Initially, the East Kent data set (the development data set only) and Medway data sets were compared for the admissions suitable for inclusion in the 72-hour model.

This suggests some differences between the two sets of data. However, results for AKI by 72 hours were relatively similar in the two data sets. Any AKI occurred in 8.9% of the East Kent patients, compared with 7.6% of the Medway patients. AKI stage 2/3 was rare in both data sets, but the occurrence was similar in the two groups.

The first set of analyses looked at the occurrence of AKI in each category. Analyses were performed on patients split into risk groups based on the predicted probability of AKI. The ‘expected’ risk of AKI in each category based on the predicted probabilities was calculated, along with the observed probability of any AKI, and then again based on the predicted probability of either AKI stage 2 or AKI stage 3. The results are summarised in Table 16.

TABLE 16 - Risk model 2: a comparison of the expected with the observed probabilities of AKI in the different risk groups for new AKI at 72 hours in the second population

Categorisation	Risk category (%)	n	Mean expected (%)	Observed (%)
Any AKI	≤ 5	760	3.0	3.7
	> 5–10	563	6.8	8.5
	> 10–20	214	13.4	15.0
	> 20	48	26.5	25.0
AKI stage 2/3	≤ 1	1218	0.5	0.7
	> 1–2	262	1.4	0.8
	> 2–3	69	2.4	1.4
	> 3	36	4.3	0.0

The results suggested reasonably good discrimination between the risk groups for any AKI. For this categorisation the observed results increased with increased risk, with the categories correctly ordered. The discrimination was less good for the prediction of AKI stage 2/3. The first three categories are correctly ordered, although there is little difference in the observed occurrence of AKI stage 2/3 in the lowest two groups. The highest-risk group actually has the lowest occurrence of AKI stage 2/3 (no observed cases); however, it is noted that the group is small in number.

The calibration of the models was good for any AKI. The observed outcomes in each category are broadly similar to those ‘expected’ by the model. For the first three risk categories the model underpredicted slightly. In all instances, the observed percentage in each category was within the range of the risk category. The calibration of the AKI stage 2/3 model was less good, with the model overpredicting the occurrence of AKI stage 2/3 in all risk categories.

The next set of analyses further examined the discrimination of the two categorisations using ROC curves based on the predicted probabilities. A summary of the AUROC values is given in Table 17.

TABLE 17 - The ROC analyses in the second population for the validation of risk model 2

Categorisation	AUROC (95% CI)	Interpretation
Any AKI	0.71 (0.67 to 0.76)	Fair
AKI stage 2/3	0.63 (0.52 to 0.75)	Poor

The ROC curve analyses suggested that there was some discriminatory power for the prediction of any AKI, although the AUC values were only ‘fair’. This is comparable with the value of 0.68 obtained from the internal validation with the East Kent data; therefore, this is a slightly better performance than that of the internal data.

The AUROC was poorer for the prediction of AKI stage 2/3, with a value of 0.63, which is a fairly poor value. This is a lower value than that obtained from the East Kent internal validation.

The ROC curve for any AKI is shown in Figure 20. Figure 21 shows the ROC curve for the prediction of AKI stages 2 and 3.

FIGURE 20.

Risk model 2: ROC curve for the prediction of any AKI in the second population.

FIGURE 21.

Risk model 2: ROC curve for the prediction of AKI stages 2 and 3 in the second population.

The final analyses compared the observed numbers of patients with AKI in each AKI group and the numbers predicted by the model using the Hosmer–Lemeshow test. A summary of the results is given in Table 18.

TABLE 18 - A comparison of the observed numbers of patients with AKI in each AKI group and the numbers predicted by the model in the second population using the Hosmer–Lemeshow test for the validation of risk model 2

Or AKIN2/3 for second categorisation.

With 2 degrees of freedom.

Categorisation	Risk category (%)	No AKIa	AKIa	χ2b	p-value
		Observed (expected)	Observed (expected)
Any AKI	≤ 5	732 (737)	28 (23)	4.3	0.12
	> 5–10	515 (525)	48 (38)
	> 10–20	182 (185)	32 (29)
	> 20	36 (35)	12 (13)
AKI stage 2/3	≤ 1	1209 (1211)	9 (6.2)	4.0	0.14
	> 1–2	260 (258)	2 (3.6)
	> 2–3	68 (67)	1 (1.7)
	> 3	36 (34)	0 (1.6)

The results suggested that for both models there was no statistically significant difference between the observed numbers and those expected by the model. This typically suggests a fairly good fit of the model to the data.

Risk model 3: predicting worsening acute kidney injury at 72 hours

Validation

After exclusions from the full database (as described above), one-quarter of the data were randomly selected for the validation database. These consisted of 778 admissions. However, owing to a small number of missing values, predicted probabilities were obtained for 775 admissions.

First, the development and validation data sets were compared. This suggested that the two data sets were closely matched in terms of the model variables. This is not surprising given that the selection of the two data sets was at random.

The first set of analyses split the patients into risk groups based on the predicted probability of worsening AKI. Three risk categories were used (as opposed to four in previous risk models) because of the smaller sample size for this data set and the low occurrence of worsening AKI.

The ‘expected’ risk of worsening AKI in each category based on the predicted probabilities was calculated, along with the observed occurrence of worsening AKI in each category. The results are summarised in Table 20.

TABLE 20 - Risk model 3: a comparison of the expected with the observed probabilities of AKI in the different risk groups for worsening AKI at 72 hours in the second population

Risk category (%)	n	Mean expected (%)	Observed (%)
≤ 4	445	2.6	7.4
> 4–10	244	5.9	7.0
> 10	86	19.7	11.6

The results indicated fairly poor discrimination between the risk groups for worsening AKI. The first two risk categories had little difference in the observed percentage of patients with worsening AKI, with the occurrence in the highest-risk category being not significantly higher.

The calibration of the model was also not particularly good. There was an overprediction of the risk of worsening AKI for the lowest-risk category.

The next set of analyses further examined the discrimination of the model using ROC curves based on the predicted probabilities of worsening AKI. The AUC value was found to be 0.53 (95% CI 0.45 to 0.61). This suggests that the model has little or no discriminatory ability. The ROC curve is shown in Figure 22.

FIGURE 22.

Risk model 3: ROC curve for the prediction of worsening AKI.

The final analyses compared the observed numbers of patients with worsening AKI in each AKI group against the numbers predicted (expected) by the model using the Hosmer–Lemeshow test. A summary of the results is given in Table 21.

TABLE 21 - A comparison of the observed numbers of patients with worsening AKI in each AKI group and the numbers predicted by the model in the second population using the Hosmer–Lemeshow test for the validation of risk model 3

With 1 degree of freedom.

Risk category (%)	No worsening (n)	Worsening (n)	χ2a	p-value
	Observed (expected)	Observed (expected)
≤ 4	412 (433)	33 (40)	44.8	< 0.001
> 4–10	227 (229)	17 (15)
> 10	76 (69)	10 (17)

The results indicated that there was a highly significant difference between the observed numbers and those expected by the model. This suggests a lack of fit of the model to the data.

The model overpredicts the number of cases of worsening AKI in the lowest-risk group, but underpredicts the number in the upper risk group.

Risk model 1: the point of admission to hospital

Development

After exclusions from the full database (as described in Chapter 2), half of the data were randomly selected for the development database. Table 22 displays the variables that were selected via the SSVS method for the Bayesian logistic regression model.

TABLE 22 - Variables selected for the Bayesian logistic regression model: admission

Variable	Median	95% credible interval	Mean
Intercept	–1.1835	–1.2318 to –1.1375	–1.1839
Variables with probability of inclusion > 0.5
Episode	0.1352	0.0800 to 0.1920	0.1353
Primary diagnosis: diseases of the genitourinary system	0.0988	0.0530 to 0.1398	0.0980
Charlson Comorbidity Index score	0.1480	0.0966 to 0.1975	0.1478
Baseline eGFR	–0.2255	–0.2812 to –0.1693	–0.2255
Proteinuria 12-month test count	0.0683	0 to 0.2045	0.0047
HbA1c recent result provided	–0.0839	–0.3052 to 0	–0.0760
Variables with probability of inclusion > 0.25
Primary diagnosis: diseases of the respiratory system	0.0009	0 to 0.1069	0.0290
MSU or CSU in previous 2 weeks	0.0007	0 to 0.1045	0.0335
Platelet result available	–0.0005	–0.1364 to 0	–0.0246

This model considers only the main effects. SSVS calculates the probability of inclusion of variables in the model. A high inclusion probability implies that the variable is important. A good cut-off point for this probability is 0.5. Based on this, the model should include six variables: spell type, primary diagnosis – diseases of the genitourinary system, Charlson Comorbidity Index score, baseline eGFR, proteinuria 12-month test count and whether or not the HbA_1c recent result has been provided. Based on this model we can make the following statements, which are illustrated in Figure 23:

1. The odds of developing AKI are 1.15 times greater for non-elective admission than for elective.

2. The odds of developing AKI are 1.1 times greater for an episode with primary diagnosis = 12; that is, a diagnosis of a disease of the genitourinary system.

3. The odds of developing AKI are 1.16 times greater for an episode with Charlson Comorbidity Index score = 10 than for an episode with Charlson Comorbidity Index score = 9.

4. The odds of developing AKI decrease by a factor of 0.80 when the baseline eGFR increases by 1 unit.

5. The odds of developing AKI increase by a factor of 1.07 when the proteinuria 12-month test count increases by 1 unit.

6. The odds of developing AKI decrease by a factor of 0.92 when the recent HbA_1c blood result is provided.

FIGURE 23.

Graphical representation of variables indicating the probability of their inclusion in the final model.

The effects of a diagnosis of a disease of the respiratory system, of MSU or CSU taken 2 weeks earlier and of the provision of recent PLT are very small, which means that the odds are very close to 1, suggesting that the risk of AKI is the same.

Risk model 2: predicting new acute kidney injury at 72 hours

Development

After exclusions from the full database (as described in Chapter 2), half of the data were randomly selected for the development database. Table 23 displays the variables that were selected via the SSVS method for the Bayesian logistic regression model.

TABLE 23 - Variables selected for the Bayesian logistic regression model: 72 hours

Variable	Median	95% credible interval
Intercept	–1.3377	–1.5866 to –1.2238
Variables with probability of inclusion > 0.5
Charlson Comorbidity Index score	0.1744	0 to 0.2714
Variables with probability of inclusion > 0.25
Outpatient attendances in the last 12 months	0.0010	–1.6156 to 0

This model considers only the main effects. SSVS calculates the probability of inclusion of variables in the model. A high inclusion probability implies that the variable is important. A good cut-off point for this probability is 0.5. Based on this cut-off the model should include only one variable, the Charlson Comorbidity Index score. Based on a cut-off of 0.25 only one other variable is included, the outpatient attendances in the last 12 months. The second and third columns of Table 23 display the median effect and 95% credible interval (for the effect) of the selected variables. Based on this model we can make the following statements, which are illustrated in Figure 24:

1. The odds of developing AKI are 1.20 times greater for an episode with Charlson Comorbidity Index score = 10 than for an episode with Charlson Comorbidity Index score = 9.

2. The odds of developing AKI decrease by a factor of 1 when the outpatient attendance in the last 12 months increases by one additional attendance. This suggests that this variable has a very insignificant impact on the risk of developing AKI.

FIGURE 24.

Overall plot of variables indicating the probability of their inclusion in the final model.

Getting to know the alert system

Some respondents had already used a similar system and so found the transition to using it for the research straightforward:

I’ve used QlikView already so I’m familiar with it . . . [researchers] took me through the system and I basically got on with it.

C6:1

. . . I’d used QlikView previously, some of the trust data’s on QlikView itself . . . I think [researcher] probably discussed it at one of our senior doctor meetings and then it was introduced, and we’d had a chat about using it . . . that the on-call consultant would use it every day to identify patients.

C4:1

I basically was using it in a previous job role . . . to access A&E [accident and emergency] data so we was [verbatim] kind of familiar with it from there. And then . . . I’ve just adapted the bits of it that I’m using really.

ON2:1

The nature of the first introduction appeared to be informal and brief; some participants had difficulty remembering it:

I think it was at a meeting that [researcher] briefly mentioned but it wasn’t a formal you know, unveiling of the system . . .

C1:2

I don’t think there was any formal training that I remember, just opened it and started using it.

C3:1

I can’t remember actually . . . I suppose [researchers] must have shown us, shown me how to use it but I honestly can’t remember.

C2:1

The outreach nurses appeared to have had more formal training, which they rated highly in terms of meeting their needs, and they particularly valued the accessibility of the researcher:

. . . when [researcher] trained us, I certainly felt that it was very adequate. We had several sessions with him. And I feel very much that if I needed anything I could always get what I wanted from [researcher] so it wasn’t . . . it wasn’t just an informal thing for us.

ON1:4

I mean [researcher] is very adaptable and approachable in terms of giving us what we need, I think, as a whole team across the sites in terms of education and updates and stuff.

ON3:4

Some of the consultants had been involved in the project’s conception so they were already knowledgeable about it, as explained here:

I was aware of its development and you know [researcher] would talk about it whilst it was being developed and ask for ideas and suchlike . . .

C5:2

Most respondents felt that the system was relatively straightforward and that they were able to learn experientially, without the need for more formal training:

. . . a lot of it was sort of learning as you tried to use the system, you try . . . and you learned how to access this bit of it or that bit of it . . .

C1:1

. . . it’s not very complicated, but yes we were given passwords for access to QlikView and the AKI system and then I think [researcher] took a quick whizz through and that was it.

C5:1

. . . you learn the tricks as you go, so I think expectations of how you’re taught how a system works are probably too high . . . the practicalities of doing it – actually that experience is quite a useful tool.

C4:3

First reactions to the system

All respondents appeared to be keen to support colleagues and welcomed the alert system. There was also an acknowledgement of its importance and potential contribution to innovation and change in renal medicine:

. . . well I thought it was a good idea because we I mean we have been doing this with chronic kidney disease and advising GP [general practitioner] practices so I thought with acute kidney injury . . . it was quite a good idea and I was keen to continue with it and help out the research process.

C1:1

I think it’s great that there’s a system in place to pick people up.

ON4:26

It’s important work, not only for the trust but it fits with the recommendations of the NCEPOD report for AKI . . . and the general Department of Health Research and Innovation Agenda . . . yeah, so good for patients and practice.

C6:1

In addition to recognising how the system could improve patient care and impact on practice, other respondents discussed the proactive nature of the alert system in bringing this about:

. . . people can be going ‘off’ without anybody realising, so having an alert system that actually picked it up and prompted the doctors looking after them to actually have a think about what might be going on with this patient seemed like a very good idea.

C5:1

. . . it’s our only means of identifying some of the sick patients . . . and quite often some of these groups of patients we will only pick up on because they’ve been identified through the alert system . . . Had we not had that access to that we probably would never even know they’re in hospital.

ON2:5

. . . well I think it’s a very good idea . . . I think is very important and using the benefits of the expertise that we have in the trust for building these platforms I think makes a lot of sense, I’m enthusiastic.

C2:1

One respondent, however, did feel a sense of imposition, suggesting that there was little choice in terms of participation:

. . . we were told that we had to start using it, we were told that it was part of a research and that they were trying to sort of incorporate it into our normal daily activities and practice.

C1:1

The outreach nurses also had some collective reservations about the added workload and goodness of fit with their role, an aspect elaborated on later:

I’m not sure we were totally warm to it because it was just something else to add to our job . . . And we’ve got one of those umbrella jobs where we . . . ‘Outreach will do it anyway’ . . . And then it was like oh. But . . . it is quite appropriate to see the patients very often because they’re sick and they have renal failure . . . But we weren’t exactly, ‘Yay!’

ON1:4/5

I . . . I sort of personally queried why this wasn’t being taken up by the renal satellite unit . . . and I did feel a little bit well I’m not quite sure that that’s appropriate for us.

ON3:5

Summary

Initial responses to the alert system appeared to be encouraging. There was strength of opinion regarding the desire to support colleagues and the research endeavour, and an understanding of its potential contribution to renal medicine. Given that the alert system has been in operation for nearly 2 years, the sentiments expressed by most respondents appeared to be enduring, evidenced by their use of the present tense when describing their views. Alongside this, however, there was a tendency for the outreach nurses to greet the system with a degree of uncertainty regarding their roles. Outreach nurses had more formal training, which they appreciated, and, although seemingly brief and informal for the consultants, the induction process appeared to be sufficient for engaging colleagues given that the technology was not complex.

General comments

General comments about the technology are summarised by the quotes below and include opinions about accessibility of information and the population group:

I think the accessibility is good, the fact that it’s web based so you don’t have to have software installed on a specific computer and the fact that I can access it from my iPad^® [Apple Inc., Cupertino, CA, USA] at home . . . I don’t have a problem with it, it’s quite intuitive, visual impact you know that’s fine again.

C2:3

I use it really just to identify the patients with AKI 3 and those who have not had intervention . . . I only deal with the ones who have not had any comments, so the new ones.

C4:5

With respect to its overall functionality, a frequent comment related to how respondents restricted their use to certain features deemed most relevant:

. . . there are a lot of buttons at the top but I never use most of them . . . most of the information is, the important things, that they’ve got AKI 3, where they are and what team they’re under . . .

C4:8

I appreciate that there are a lot of tabs along the top that I just haven’t explored really through lack of time more than anything else but in terms of the core functionality it’s very easy to use.

C2:1

To be honest I only access the first page and then I go into other systems because they provide for more information.

C5:4

Issues with user friendliness

More detailed exploration with respondents highlighted a number of issues regarding the ease or difficulty of navigating the system and accessing data. The following quotes describe the main difficulties experienced. First, the speed and responsiveness of the system appeared problematic:

. . . sometimes the pages wouldn’t load and if you were trying to filter out things it wouldn’t, it would take a while to . . . get to the filter page which you wanted to look at so sometimes a lot of clicking and logging out of the system and logging back onto it . . .

C1:2

The system is very variable in terms of it does seem to not want to upload or refresh and stuff quite regularly.

ON2:20

. . . there’s a way you can add comments so that your colleagues for the following day know that you’ve already sort of dealt with that patient and that can be a little bit sluggish to upload . . .

C6:4

One respondent also commented on the ‘modernity’ of the system:

. . . it’s not very pretty . . . because actually we’re used to doing most stuff on webpages then you go to Amazon or whatever else you know that’s what we’re used to, and it’s a very different system and I don’t think it’s knowledge user friendly.

C4:9

Issue were also raised about the mechanisms through which identifiable patient data could be retrieved and information added:

. . . sometimes just when you start, when you log in you have to clear all the selections and then start again otherwise you don’t get all the patients quite right . . . you might miss patients.

C4:4

. . . if you were reporting on a certain patient you have to manually enter sort of details, for example the NHS number or whatever and that was a bit time-consuming . . .

C1:2

. . . it’s been hard to be able to print . . . off the lists and then when I’ve gone back at the end of the day to try and put something in I get locked out again or not being able to log in.

ON4:20

if you’re documenting in notes, on a handover sheet . . . then try and do it on the QlikView as well and you think . . . It’s an endless process.

ON2:20

Given that the system was in development, the research team appeared responsive to some of the difficulties respondents were experiencing:

Things that [researcher] has changed which I like is that now it actually updates more quickly when you actually fill in a form and that information goes in there much more quickly now.

C5:2

Timing and workload issues

Although, overall, respondents appeared keen to participate, a further theme that came to light concerned the amount of time it took to use the alert system. The following quotes, from a consultant and an outreach nurse, respectively, summarise the different patterns of system use and associated activity:

. . . an average month about three times, and then when you’re on the wards, which is 1 month every fourth month you’re on the ward, then actually on top of that three you’ll do two weekends so you’ll access it another six times that month.

C5:6

. . . [we access it] Every day. And then we look at the people that are scoring twos and maybe threes. We, you know, we don’t exclude the threes because they might not have been enquired about at the point that we go.

ON3:7

Consultants were particularly keen to highlight the relationship with work plans. Each time they are on call, they encounter between one and eight new cases of AKI stage 3. This respondent appeared reconciled to accommodating the work:

You just incorporate it into your work plan . . . yes, other things get pushed down the list but it’s swings and roundabouts.

C6:4

However, most respondents found the integration of this additional work troublesome in that it was not officially incorporated into their work plans:

. . . there’s been no time put in my job plan to do it . . . it depends on the day but often it’s quite a burden . . .

C5:7

. . . you need a good one and a half to two hours of just you and the computer and the phone . . . I think that it needs to be accounted for . . . I think because it’s a big chunk of work that we have to do . . .

C1:5

A further issue that came to light concerned difficulties prioritising the work, particularly as there was a significant time commitment; reasons for this are explored more fully in a later section (see Interacting with medical teams):

. . . trying to fit it in there was very difficult and it didn’t always happen to be perfectly honest and I used to slightly resent it then but there you go, just one of those things.

C2:4

. . . a lot of time is wasted especially when you’re on call and you have to have other things you know, if you have got a ward round that day, clinic that day, so it takes time.

C3:3

. . . we’ve all got lots of other things going on and OK it’s only once every 10 days so it’s not that often but it’s just a bit irritating having to do it . . .

C4:8

Accessing data and making decisions

An issue of concern related to the fact that the data on the alert system provided information that was up to 24 hours old; this posed problems for locating patients and created uncertainty about their medical condition and concerns about wasted time:

. . . sometimes the ward has changed . . . you have to go on some other system to find where exactly the patient is and sometimes you have to, when we ring a ward they say, well, they’ve moved to the other one.

C3:4

. . . a lot of the time the results will be from A&E and they’ll flag up on QlikView from A&E so you think I’ll go into the computer system and track a patient. Oh where are they now on the site you know. You’ll trawl all the way over there to find out they‘ve just had their bloods done and their result’s better so you’ve wasted like half an hour.

ON2:18

. . . you are in a way acting on yesterday’s data so you’re coming in up to 24 hours later when a lot more things would have happened to that patient in the time period, which is a small problem I think.

C1:3

There was a view among some respondents that this time lag had its advantages:

I’m not sure true real time would be a necessary advantage actually . . . because I think that 24 hours of hindsight allows certain things to sort themselves out . . . the patient that’s admitted clearly dying . . . and me phoning up the team looking after them and saying this patient’s got acute kidney injury . . . would probably not serve any useful benefit to either the patient or the team.

C2:3

So there’s somebody who’s flagged up as [AKI stage] three, but the team have already started treating and if you can wait till the results are back you might see that their results are already improving, so . . . it’s sort of a good thing . . .

C4:4

Respondents described in more detail how patients with AKI were identified and how decisions were made to formalise the alert and contact relevant teams. Given the time lag, it was clear that other data sets were needed to support the decision-making process, as the alert system provided a first-level identification of patients that only included such details as NHS number, AKI stage and the ward:

. . . you would have to log on to other systems . . . the iSOFT system, to sometimes look at the past clinical history from clinic letters and you’d also have to log onto the X-ray system to look at the current images and the historical images.

C1:3

. . . I use all the other trust IT systems . . . so PAS [patient administration system (iSOER)] to identify where they are . . . and you can get in and use the blood results system either on that or DART [pathology database] to see what the latest result is and also the new VitalPAC system I’ll look up their observations and things . . . you know you have to go and find it, yeah you have to open it up . . . there’s no link directly onto QlikView.

C4:5

. . . we can also look on the clinical functions at previous documents where they might have been to clinics so you can get a bit of a background but not all patients have those updated.

ON3:22

For some this was frustrating, and one outreach nurse summed up the general feeling among the nurses:

None of these systems interact with each other and it’s just a nightmare really.

ON3:20

It was considered preferable to have all the data in one system; however, the absence of this was not perceived to be too much of an imposition:

I suspect it would be quite complicated to put interfaces with other systems so I think it, the other information is easily accessible on the net so it’s not too complicated to access it . . . I would say it probably takes me 5 or 10 minutes to look at all the systems and formulate an opinion as to what might be going on with the patient before I try to ring them . . .

C5:5/7

. . . the system identifies the patients, that’s the crucial bit yeah I can access the other information, oh it’s on another, I have to click another box and put in another password or whatever, but yes it’s not the end of the world . . .

C6:6

Summary

Most respondents restricted their use of the technology to the few alert system functions that were necessary for identifying the AKI patients and initiating the alert to medical teams. Observations were made about lack of user friendliness with regard to the sluggishness of navigating the system and inability to access data, and consultants raised the issue of increased workloads that were not built into work plans. Although some felt able to assimilate this extra work, a number found this to be quite onerous. Much discussion focused on the need to access data from other sources to facilitate decision-making about whether or not to alert the medical team, and the pros and cons of having a system that provided data that were 24 hours out of date.

Finding the right doctor

Consultants in particular described this as a ‘chasing game’ that was frustrating, took up valuable time and tried their patience. The following quotes elaborate on typical communication attempts in detail, focusing on difficulties tracking junior doctors and the ‘tyranny’ of bleep identification:

. . . the main problem . . . is mapping junior doctor or junior doctor team to an individual patient and I think that work wasn’t sufficiently developed before the system was rolled out . . . a patient is flagged up as having AKI you then have to . . . go through switchboard to identify which junior doctor is looking after the patient, you may phone the ward where . . . you get given one bleep number, it turns out not to be the right bleep number . . . and then that person doesn’t answer that bleep and so you try a different member of the team and you know it can take half an hour to get hold of the right junior doctor and that’s not an exaggeration . . . that actually gets very tedious when you’ve got lots of other stuff to do to be bleeping about five different people before you get a response and I think that’s a fairly common issue with it.

C2:2

And we have this daft system in this trust where the [junior doctors] look after the wards at a weekend but because they’re not actually on call . . . the switchboards don’t carry their bleep number . . . they haven’t got any number for them so you have to ring up the on-call team to actually ask them which [junior doctor] is on for the weekend and what bleep are they carrying.

C5:11

. . . it’s teams, and it’s partly because teams are so unstable because the registrar’s on holiday, the SHO [senior house officer]’s just done nights so it’s only the house officer and actually they’re in the middle of a ward round for a different team . . . that is the big problem.

C4:12

For all respondents, the frustration of trying to find the team responsible for the patient was clear, and the alert system did not appear to help in this regard:

I mean you can’t get hold of the right person, the person you eventually get hold of does not know the patient . . . you know the conversation doesn’t take very long, it’s just getting hold of people and whether they know about it or not.

C3:3

You deal with it generally out of hours by contacting a doctor who’s not really responsible for that speciality of patient who is already generally busy and probably overworked so it’s not an ideal system.

ON2:12

. . . that’s the real downside of this, I don’t actually mind any of the other bits but I get so frustrated about trying to actually pin down somebody who actually knows something about this person who’s willing to take responsibility and take it forward . . .

C5:8

Very difficult. Very difficult. I think there was a thing, a page where you could look at the contact details of the teams and sometimes I cannot access that information from the actual QlikView dashboards.

C1:5

There were added difficulties when the patient was admitted through accident and emergency (A&E) and when trying to contact certain specialties:

. . . trying to identify which team you’re meant to be talking to when it’s an A&E patient, so somebody comes in with AKI 3 into A&E, it’ll flag up as A&E and then you have to track them down which is, I mean it’s possible using the other IT systems but it’s just a bit frustrating . . .

C4:5/6

. . . surgery – particularly orthopaedics they, you can ring around quite a number of F2s [junior doctors] before you actually find someone who’ll take responsibility or go and see the patient.

C5:7

. . . what you need to do is get the best person that’s going to give the best treatment to that patient and generally unfortunately that’s not always the orthopaedic doctor.

ON2:13

One respondent felt that the European directive on working time contributed towards these difficulties:

. . . what I’ve noticed is, you know, numbers of junior teams who don’t know their patients because they weren’t on call or they weren’t, you know, or they were on a different ward round . . . it’s European working time that’s mucked the whole thing up.

C4:6/7

One respondent felt that more trust-wide communication at the beginning of the project would have been better:

I think it probably wasn’t fully sorted out and like anything you introduce a system and actually you have to anticipate there’s going to be problems and there will be a grumpy clinician somewhere who gets a bit silly about it all . . .

C4:2

Respondents had adapted to these difficulties by developing a range of strategies to improve their chances of accessing the right doctors:

. . . usually I get hold of the right person straight away by actually phoning the ward . . . I guess then that reflects the fact that the nurses just have the local knowledge on the ground on the day.

C2:2

I’m not sure how reliable those bleep numbers are but . . . I don’t even look at them now I still go back to ringing switchboard and trying to find out.

C3:6

. . . it’s very difficult to locate an orthopaedic doctor so I guess that our first port of call isn’t actually the orthopaedic team, it would be the on-call medical team or ITU [intensive therapy unit].

ON3:12

. . . we’re just communicating. We just talk to people a lot of the time.

ON3:19

Discussing the cases

Once the appropriate doctors had been located to formally alert them to an AKI patient in their care, there were differences between consultants’ and nurses’ experiences because of the differing intervention pathways; however, there were also similarities. First, these respondents provide a general overview of their approaches. A consultant describes their general intervention pathway:

I phone them up and I give them advice and make sure that they are repeating the bloods and doing all the appropriate things I think they need to do and it may be a case of just saying ‘are you aware’ and they say ‘yes’ and I say ‘fine you’re doing all the right things’ or it may be much more detailed advice depending on the situation . . .

C2:5

For outreach nurses, there were some pathway differences between the sites because of the professional judgements and discretion of the outreach team members. This did have the potential, however, to cause confusion among doctors who worked across three sites:

. . . we look to see where the patients are . . . and then we go out to the wards to see the patients and speak to the teams so we’ll put a sticker in the notes alerting the team to the patient . . . and we’ll find out . . . what’s happening with the patient . . . then we’ll speak directly with the teams about that. And then if we feel that they need continued monitoring then we’ll retain them on our list. If not, that will be our only contact with them.

ON3:7

Other people on that initial visit, even if it’s basic, might still go on to do a complete full assessment . . . And I think that’s when then you get a differing expectation maybe from the medical teams.

ON2:10

However, respondents found that in most cases action had already been taken:

. . . by and large I think that by the time I get round to speaking to the team they’re usually aware that the patient has AKI. I cannot really recall any instance, maybe one or two instances in the last year or however long since we rolled it out, that I phoned a team and they were genuinely unaware.

C2:9

. . . the doctors that I’ve spoken to will say ‘oh yes we know about the patient, we’ve done A, B and C and we’re awaiting X, Y and Z’, so most of the times they are already on the case and our advice isn’t needed.

C1:5

Although nurses had similar experiences, they felt that the stickers were instrumental in bringing about awareness:

Quite honestly, I think the stickers have been quite successful – generally [the teams have] already identified that they’ve got acute kidney injury, they’ve already started all the processes on the advice sticker and we stick the sticker on the notes anyway, even though it’s all happening.

ON3:7

When it came to reactions from doctors to these interventions, respondents had mixed experiences. This respondent, for example, felt that consultant colleagues were supportive overall:

I have spoken to other consultants, medical consultants in other hospitals and they say it’s a good thing that someone else is keeping an eye and providing that extra layer of advice to their teams.

C1:4

For outreach nurses, despite the sticker often being applied after the event, these respondents were keen to point out that this did not seem to interfere with interpersonal relationships within the teams:

. . . but they’ve identified it, they’ve done it all and then we’ve come along with our sticker! But I think there is sort of quite a . . . a good rapport with them . . . And we just go, ‘Ha, ha, ha. I’m afraid it’s got to go in there,’ because it’s like an audit trail, isn’t it? . . . but I’ve never felt anybody was really cross about it . . .

ON1:9

Not ever had a negative response from anybody.

ON2:14

. . . we’re quite good at . . . at rapport . . . There’s sometimes a bit of a competitive spirit as well. They go, ‘I’ve done it already!’

ON3:15

For other respondents, the general experience of doctors’ reactions varied, with some respondents perceiving some indifference because action had already been taken:

. . . it’s a mixture of positive and indifferent but not negative. I’ve not had anyone getting irritated, cross, abusive or obviously lacking in gratitude. Indifferences maybe one-third and people actually sort of sounding quite positive’s probably two-thirds and that is irrespective of the sort of level if you like, of the person I’m speaking to . . .

C2:5

. . . they’ve got you know other priorities they are busy . . . they just give you the information and if you say something to them they will just note it down but I don’t think they’re hugely interested.

C3:5

Another respondent highlighted that there could sometimes be an issue with understanding the urgency of the situation for the patient:

. . . sometimes you’ll get a registrar who’ll say ‘well I’m in clinic I’ll deal with it later’ and you say ‘no you can’t’ . . . and yeah, so occasionally perhaps there’s a perception issue as to the significance of the problem, it’s how you handle that . . .

C4:10

As intimated previously, some of the respondents did notice a difference between the teams in the trust in terms of how the patients were dealt with:

. . . medical teams usually have it under control, it’s the surgical teams that have the challenges . . .

C6:5

. . . often I find that the ward manager knows far more about what’s going on with Mrs Bloggs than the surgical junior doctors . . .

C5:10

. . . you are dependent . . . on one on-call doctor at a junior level who specialises in people with broken or damaged bones who then have – this is a slightly sweeping statement that I’m going to make, – the inability to do anything about anything that’s not to do with an injured bone! It’s just like, ‘That’s not my job.’

ON3:12

This respondent had a sympathetic opinion regarding these differences:

. . . I am aware that in many instances . . . the surgeons are less aware but they’re surgeons . . . physicians should be looking after these patients . . . surgeons should get on with the job they’re best at . . . they’re not particularly concerned with post-op . . .

C6:3

It was of interest, though, that many of the consultants recognised the potentially intrusive nature of the alert system and were empathetic about how the telephone calls could be received:

I feel very guilty about troubling them . . . the post-take ward round is probably the most stressful point in the week when your workload is at its highest. And to then have somebody bugging you about a patient you already know has got AKI to tell you they’ve got AKI, I feel slightly guilty about to be perfectly honest . . . they are remarkably restrained I think.

C2:7

If I had a patient coming in under my care and a doctor phoned me up asking me basic questions, I would probably get a bit upset but that’s me, every doctor is different.

C1:6

. . . it’s sort of stepping on professional toes a little bit . . . there is perhaps sometimes a worry that we almost take a bit of a holier-than-thou attitude . . . you know, somebody coming in and, not even invited . . . and saying ‘well you’ve got this wrong and this needs to happen’, it could be perceived as you know a little bit arrogant or a bit rude. Not the intention at all.

C4:2

Awareness of the alert system

One of the factors that influenced the response by the teams was the extent to which they were aware of the alert system. Some felt that this was an enduring problem caused by working patterns, especially among junior doctors:

The juniors . . . haven’t had a clue about them at all but once you get to registrar level and above they’re very receptive to it . . .

ON4:14

. . . even now I think a lot of them are basically not certain why we’re ringing them . . . it happens, because there are so many clinicians all over and surgeons, and physicians and nurses, junior doctors. I don’t think everybody knows about it.

C3:2

. . . teams are changing all the time. Junior doctors change every year . . . you just get one group used to the system then a new group arrives with no knowledge, no training in dealing with patients with AKI . . . so it’s a challenge.

C6:11

This respondent highlighted the importance of communication skills:

Sometimes they’ll go ‘ugh, no!’ and don’t know about the system . . . they’re not fully aware that they might get a phone call, so how you introduce yourself in the first instance I think sets a tone of the rest of your discussion . . .

C4:10

For others, there had been more problems at the beginning of the project:

I think in the early days people were surprised that someone else was aware of that clinical issue . . . sometimes if you rang you’d end up speaking to the ward . . . and then they would say ‘sorry, who are you? And what system is this?’ . . . So it’s almost like sometimes I felt I was a bogus caller . . . then eventually, eventually now it’s old hat they generally are aware.

C1:4

Well, we did some work early on in getting round to tell people about the system . . . getting buy-in . . .

C6:3

Summary

There were similarities between how consultants and nurses approached ‘alerting’ and discussing cases with the relevant medical teams; however, for outreach nurses, the extent of professional involvement in this varied between the hospital sites, which could cause confusion among doctors. The difficulties respondents had in tracking down the appropriate junior doctor manifested themselves as a major problem, and it was clearly frustrating and costly in time. Respondents highlighted the many intricate instances of how communication can fail to connect them to the right person in a timely manner. This included failures with how bleeps are managed, the complexity of junior doctors’ working patterns, how their responsibilities are organised, the instability of medical teams, and how the movement of patients is monitored. All respondents had developed strategies to cope with this frustration, which mainly entailed telephoning the nursing staff on the wards, who they felt had more immediate knowledge of who to contact, and maintaining good channels of communication.

In the face of these difficulties with contact, the general experience of the respondents was that once the appropriate doctor was located and cases were discussed, most found that teams were already aware and were dealing with the situation. Outreach nurses noted the success of the stickers in bringing about awareness. Some respondents felt a degree of discomfort with ringing colleagues uninvited for little justification, noting that responses sometimes implied disinterest. On the whole, however, responses were not negative and there was recognition of the importance of interpersonal communication skills. It appeared that there were differences between specialties; not only did some respondents seem to have greater difficulties accessing surgical and orthopaedic teams than physicians, but surgical teams were, by and large, less aware of their AKI patient and how to action the alert.

When it came to colleagues’ awareness of the alert system, respondents were mixed in their responses. Some felt uncertain about whether or not there was full understanding of why they were being contacted. The shifting nature of junior doctors’ employment explained some of this. Others felt that there were problems at the outset but that there was now generally a good understanding.

To follow up or not to follow up?

The responses were varied; these quotes from consultants, for example, describe experiences in which respondents were particularly concerned about inappropriate management and took action:

. . . we had one last week where I felt very strongly that the patient had been inappropriately managed . . . my colleague went and saw the patient, you know, absolutely agreed that this was an avoidable situation, that really they should have been more aware of the risks and the fact that the patient already was developing AKI . . .

C2:6/7

. . . there are a couple of cases where I’ve been really, really worried about somebody . . . then I will actually follow it up and look at it the next day and the next day to make sure that actually they’re getting better . . .

C5:10

. . . probably once every 6 or 9 months then you know I feel actually this hasn’t been appropriately managed and there needs to be a bit of follow-through for this.

C2:6

As intimated in earlier sections, outreach nurses’ involvement varied across sites according to the condition of the patient and professional judgement:

. . . in my workload . . . they either get a sticker and we’d never ever promise to do anything more than one visit, or they are a documented outreach patient, so they’ll have a full assessment.

ON1:10

. . . if they have already identified AKI then . . . then that’s it and we don’t really necessarily go back.

ON3:11

. . . you’ll find people that are more complex, that have got a lot of other ongoing problems, that have got maybe multiple potential causes of their AKI that are very sick . . . you keep on the outreach list because they’re sick, not necessarily just because they’ve got an AKI.

ON2:9/10

These consultants, however, felt that the teams should be accountable following the alert communication:

I really don’t see the need for further monitoring, once it’s handed over then it’s the responsibility of that team . . .

C6:5

I think really the ball should be in their court, whose patient it is, that they should be just chasing up and let us know if there is progress or lack or it rather than, you know, we chasing them all the time which is happening at the moment . . .

C3:5

Whether or not they followed up cases, all respondents made efforts to ensure that a communication was recorded to alert the teams to their involvement, or that a mechanism was in place should more advice be needed:

. . . following up, no . . . I try to remember . . . put that as a final comment you know, registrar or SHO [senior house officer] informed, discussed blah blah blah, team aware to contact renal if they have any further concerns or issues.

C4:8

. . . if I was going to carry on monitoring or seeing a patient from a trigger . . . I would write something at the end of the notes along the lines of ‘we will continue to monitor this patient’s progress’ and something along those lines.

ON2:10

. . . obviously the team can contact you but sometimes the team will contact the on-call doctor as well or would have already done that so there is a way of, for the team to follow up those cases.

C1:3

Many of the patients eventually came under the care of the renal team:

. . . a lot of them do end up on dialysis and they eventually come to our ward anyway so there is that follow-up. I think out of all the cases I’ve done there was one that I was particularly interested in and kept contacting the team and . . . then eventually they came over to our ward . . .

C1:3

Acting on advice

There were distinct differences between the consultants and the outreach nurses on this subject. For consultants, the extent to which advice was acted on was not generally known because of work patterns. One respondent stated:

. . . you generally aren’t really aware of whether it’s been carried out or not – your recommendation – because the next day you’re not on call again . . . On one occasion I did ring to speak to the doctors and I advised them to call me back, and they never did. And for whatever reason they probably felt that they’d finished the case effectively and they didn’t need my advice, I’m assuming.

C1:4

A further concern related to this was the failure of teams to make a written note of the discussions with respondents, which impacted on the audit trail and clinical decision-making process. This respondent summarises consultant experiences:

What I hadn’t done until recently was actually ask them to document the discussion that had been had with them . . . There’s no documentation of these discussions happening, and I suspect that’s because often the discussions happen when the doctor’s not in front of the notes because we’re ringing them and . . . catching them wherever they are.

C5:8

The outreach nurses found that there were clear comparisons between different medical teams, and significant concerns were raised again regarding the lack of appropriate trauma and orthopaedic team responses and accountability. The following responses highlight the extremes to which the nurses feel they need to go to ensure that patients are appropriately managed:

. . . the medics are really very good at dealing with the acute kidney injury. The area that we have a huge problem with . . . is the trauma and orthopaedic areas . . . you know, they . . . they will watch their patient’s renal function deteriorate and then the nurses will call the outreach team to say, ‘This patient’s got a really big problem,’ and then we’ll tip up and start the whole ball rolling really . . . in extreme cases they’ve already contacted the renal team and they completely ignore the advice. And that’s really an area that needs a lot of work . . .

ON3:11

We’ve started using words like ‘life-threatening’ ‘this patient will die’ and then . . . and then the next thing we do is we say, ‘Shall we phone [the renal team] for you? Here you are; they’re on the phone’.

ON2:15

. . . we’ve highlighted it, shown the doctors the stickers in the notes and said to them, ‘This is actually quite serious now and this patient looks like they have a potential to deteriorate. We feel that you need to take this further,’ and the doctors have walked past us and carried on talking to other nurses and . . . and we’ve had to wait and wait our turn to then come back to them and say, ‘Did you quite understand that this is serious? Can you please listen to us?’

ON4:15

Summary

There were differences in consultants’ and outreach nurses’ experiences regarding monitoring patients and acting on advice given. For consultants, monitoring or follow-up of cases was conducted on few occasions and only when patients gave cause for concern and confidence in managing the patient was not high. Some felt quite strongly that their involvement should end following the alert, as teams should take over responsibility. Ongoing monitoring by outreach nurses varied between the three sites and was dependent on professional judgement and the condition of the patient.

Most consultants were not aware whether or not advice had been acted on, and there were some concerns regarding the documenting of the advice given. However, all respondents, including nurses, agreed that they would ensure the availability of a contact for further advice if needed. Outreach nurses focused on challenges with trauma and orthopaedic teams and highlighted the difficulties with passing on the management responsibility of AKI patients to them.

Making a difference to patient care

Some respondents could point to circumstances in which they might have had some positive impact on patient outcomes, and outreach nurses tended to have a firm conviction that they were essentially contributing to patient benefit:

. . . for the patients. I know . . . I mean I don’t know how many we’ve seen . . . but I know that there are some that we’ve changed things for.

ON1:24

. . . how can it not be a benefit? It has to be, doesn’t it? It has to be.

ON2:25

. . . most AKI is . . . secondary to sepsis or dehydration, or excess diuresis, you know, so actually if you can alert the teams to it and get them to intervene early, then actually you will significantly improve outcomes.

C5:12

I think sometimes . . . when we pitch up a lot of the relatives and the patients are, ‘Oh thank goodness somebody’s going to come and do something,’ because they’ve got . . . they’ve noticed that there’s something wrong and it looks as if somebody’s coming to sort it out so that’s a patient benefit, isn’t it, from their perception.

ON3:26

But consultants were largely sceptical about the benefits. This was mostly because of the observation that either teams were already aware or full-scale intervention was not clinically appropriate:

I question whether phoning a junior doctor team or consultant . . . about a patient admitted the day before with acute kidney injury and who, they must be aware if they’ve looked at the blood results, has AKI, I wonder whether that actually provides any added value at all.

C2:7

. . . I’d say three-quarters of the patients you ring up . . . they’re either dying and there is nothing you can do or the team’s got it in hand and they know exactly what they’re doing and they’re doing fine. So you know it does feel sometimes like quite a lot of work for very little benefit.

C5:11

One respondent indicated that the lack of perceived benefits stemmed from advice not being followed:

I think that generally speaking it doesn’t improve patient care . . . there was one case only where I did give advice which was the correct advice and then we subsequently found that the advice wasn’t followed and that patient was eventually transferred here for dialysis therapy so I feel that it doesn’t really add to patient management . . .

C1:5

Comments were also made that suggested a need for proper comparative research and larger samples to better estimate benefits:

. . . once you’ve, you know, given the information and instructions, what happens it’s not really followed up and whether it would have made any difference you would have to compare it with somebody who has not had that, you know, data so you have to compare it with something to know whether there is a difference or not.

C3:5

I think there are too few numbers of patients to really point to whether it has a benefit or not, but it’s brought AKI to people’s attention . . . we just need to have the means to track patients . . .

C6:10

The following respondent indicated that more benefits could potentially be achieved through the outreach team, who focus on patients with the earlier stages of AKI:

I think the outreach team have been looking at going out to see people with acute kidney injury stage 2 and they’ve actually physically seen patients and put information in the notes . . . I suspect that probably the biggest benefits will have come from that intervention, getting in there a little bit earlier and actually steering people in the right direction.

C5:12

Impacts on clinical practice

Most respondents saw both positive and negative impacts on clinical practice. The following statements refer to cases in which intervention can be positive and serve an educational purpose:

I think that there are certain groups in the hospital that are very poor at managing the sick patients and particularly patients with AKI . . . so I think that actually, I think that’s where I see that we have the benefit . . . when I ring up some, one of the surgeons to say did you realise your patient has deteriorating renal function . . . they usually haven’t got a clue . . . so it does prompt them to go and actually look and get their medical review.

C5:9/10

. . . sometimes it’s not direct intervention, it’s almost preventing intervention . . . you know, facilitating the team making a clinical judgement on what is or isn’t appropriate for the demented ninety seven year old with multiple other illnesses . . . So some of them are sort of much more soft interventions about sort of management strategy rather than medical treatment . . . some of it’s educational as well.

C4:13

The outreach nurses specifically identified the positive and proactive contribution of the stickers, as well as the ability of their involvement to reduce unnecessary work:

And they’re actually now approaching us when we go onto the unit and say, ‘Can we have a sticker?’ and they will have people that haven’t even been identified on the database as acute kidney injury and they want stickers for those notes.

ON3:7

. . . whoever’s on the nightshift will go round in the early hours of the morning, we’d go into QlikView, get the list out, go and identify these patients and their notes so that when they’re being post-take in the morning it’s identified to the teams . . . Which is . . . certainly cutting out one ward round and potentially 1 day’s worth of medical input.

ON2:17

In addition, outreach nurses in particular saw the educational opportunities for themselves:

I think for me it’s been positive because I’ve . . . it’s done a lot for my knowledge and maybe . . . maybe I see patients in a different way now. Sharing skills which is part of our job, people show interest in the stickers and you’re able to tell them what you’re looking for and why so they might pick things up about the drugs that you’re looking at and so on.

ON1:24

The following comment from a consultant indicates the benefits of an AKI database, again in terms of enhancing educational opportunities in the trust:

. . . the system does mean that we’ve now got a very good database of actually what sort of AKI we have in the trust and where it is . . . which groupings is it happening under and maybe an idea as to you know, who’s doing well and who’s not doing so well and to allow us to focus some education on those areas that aren’t doing well.

C5:11

Conversely, other comments related to the negative personal impact on practice brought about by the alert system:

. . . I think it adds to my busy day already when I’ve put you know, I can actually do other things . . .

C1:6

. . . I’m not sure it’s the best use of my time to be perfectly honest . . . a lot of it’s an administrative task which is pretty tedious.

C2:9

However, all respondents highlighted concerns regarding clinical responsibility for the patient and the impact of the alert system. This comment describes what this means in practice for consultants:

I have quite a lot of concerns about ownership of the patients. So for example if a patient was admitted on my take, I would assume a hundred per cent responsibility for . . . acting and looking into each problem and dealing with it. And when you include a system like this I feel that number one ownership could be eroded and doctors . . . could become more nonchalant and lethargic and say ‘oh there is this computer system that could pick this patient up and because they have renal failure it’s not my problem any more’ and I feel strongly about that.

C1:6

As has been noted, outreach nurses had particular concerns about where their involvement began and ended, and how some doctors interpreted this:

I think ownership of the . . . ownership of the alert system from my feeling is . . . becomes an issue. My feeling is that once you document in the notes ‘Outreach’, you then become the first point of contact for anything that ever goes wrong or problems . . . you then become embroiled in the rest of the unfolding of the events.

ON2:6/7

. . . some doctors will see the sticker and your signature and think that every day you’re going to make sure their bloods are done and you’re going to check the results. Other teams won’t. Other teams will just manage what they do but it’s very varied.

ON1:9

Even from the medical teams; ‘Mrs so-and-so – you’ve seen this person?’ ‘I’ve kind of not really; I just stuck an . . . alert sticker in the notes and signed it.’

ON3:6

This respondent made particular reference to the establishment of a specialist orthogeriatric system that compounded the problem of patient accountability within certain surgical teams:

. . . part of the problem with trauma and orthopaedics is now that there is this orthogeriatric system in place, it’s almost exacerbated the problem of them now completely ignoring anything . . . there’s almost a mentality now that; out of hours – oh that can wait, the orthogeries will be on in the morning.

ON2:15

There were also concerns that the alert system had generally impacted on their professional roles in a negative way, reducing it to an unsatisfactory screening process as a result of being provided with by ‘lists’:

I think the problem for us is that we have so many parts of our role and we’ve . . . we’ve become very much a list and screening culture, so we . . . we start off every morning with a list from the alert system, we have a list from the VitalPAC, we have our list of patients and it feels very much as though we’re trawling around the hospital to keep a lot of lists . . . you obviously have to go through that process to get to people that you’re going to have effective interactions with but it’s beginning to feel like we are part of this list culture . . . that’s not very nice really.

ON3:17

Continuing with outreach nurse perceptions of professional issues, respondents were aware that there were numerical differences in patient interactions brought about by the alert system between the three hospital sites. This was explained by an administrative problem with recording on two of the sites (the numbers recorded did not in fact reflect activity in reality) and that there were fewer identified patients on the third site:

. . . we go and see these patients but we don’t always at the end of the day sit down and put them on the system . . . Now, unless we keep all of that documentation we can’t go back retrospectively and put it on the system so we . . . we miss that time slot. So there are a lot of interactions we do on this site that we don’t put on the system.

ON3:19

Probably at [site] our patient numbers are lower . . . And it is doable within our working hours.

ON1:19

Perceptions of cost-effectiveness

It was generally difficult for respondents to express the extent to which they felt that the alert system saved money, but the following comments demonstrate that it is certainly an aspiration:

Well it costs £150 to dialyse a patient, and that’s around £30,000 per patient per year . . . so it would be nice to think it was making some impact on that . . .

C6:10

If you’re highlighting early people that have got an acute kidney injury that’s then being treated and managed it’s got to save money in terms of length of stay and ongoing treatment or higher levels of treatment.

ON2:26

. . . so do we save money? Don’t know . . . do we improve patient outcomes? One would hope to think so and actually you could then say well improving outcomes is going to be a cost saving in terms of if you can prevent somebody needing to go onto dialysis or needing to go up to ITU for haemofiltration, yes you will save money, you will shorten their patient inpatient stay.

C4:14

This respondent, however, was more sceptical:

. . . if somebody’s already in ITU and being managed and they know about AKI I don’t think that will save money just to tell them that he has AKI.

C3:7

Other, more comparative views focused on the cost of respondent time:

I don’t know how much QlikView costs, I’m assuming it’s part of the great IT budget of the hospital so in terms of everything else it’s probably a small cost, but in terms of time at the moment it’s quite costly on time.

C1:7

. . . well it’s highly cost-effective at the moment because it’s not included within any of our job plans to my knowledge, so essentially the trust is getting this additional work for – for nothing!

C2:8

Summary

Despite some respondents, largely outreach nurses, perceiving positive proactive, preventative and educational benefits for patient outcomes and clinical practice, other respondents were not wholly convinced of the value of the system in terms of inducing these benefits, nor of being cost-effective. This scepticism resulted from the observation that most clinical teams already had the situation in hand, or that intervention was not indicated. In addition, the small number of patients made impacts difficult to assess. An issue of concern centred on the potential of the alert system to divert perceived clinical responsibility of the patient away from the clinician who was actually accountable, and outreach nurses in particular felt that this aspect impacted on their professional roles. It was also revealed that discrepancies in outreach nurse system activity between the sites were largely caused by an administrative recording issue.

Improving trust-wide knowledge of the alert system

Knowledge of the alert system could be improved by the following means: inclusion of a user manual; an initial comprehensive and wide exposure of clinicians and nurses to the new system; an ongoing programme of updates; a focus on junior doctors’ education; and greater visibility of the stickers.

maybe a user’s guide . . . you know, an idiot’s guide how to quickly get to the reporting page . . .

C1:2

. . . perhaps if they would have maybe sat down with everybody at the beginning and explained how the system works and what to do and how to go through the system.

C1:1

I think that it would have probably been advisable to have made it more obvious, more visible to other staff in the trust . . . you’d have to kind of redo that every time you induced a new load of junior doctors across multiple sites, I’m not sure that would be practical so maybe some, you know some pre rollout announcements and so on . . .

C2:2

I think [junior doctors] need to be made aware of that but that will have to be a very extensive education programme, because it’s across three sites and it includes all specialities surgical, medical and the juniors change as well.

C6:4

I have to say the stickers are good. I do think the stickers need to be a bit more in your face, though, because I sometimes put them in the notes and think mmmm, do you know what; I’d just like something a bit more out there.

ON2:23

Improving the technology

Risk models in the setting of acute kidney injury

In the literature, the majority of the reports focus on the need for RRT after cardiac surgery. One of the first of these was by Chertow et al. ,34 who produced a risk model for predicting AKI after cardiac surgery, based on a population of 40,000 patients who underwent cardiac bypass or valvular surgery in 43 Veterans Administration Hospitals in Virginia, USA. A risk stratification algorithm was formulated on the basis of interactions between potential risk factors. There were inherent flaws in the study cohort, specifically a lack of female patients and African-American patients. Thakar et al. 101 produced a clinical risk score to predict post–cardiac surgery AKI requiring RRT, based on 33,217 patients who underwent cardiac surgery at the Cleveland Clinic between 1993 and 2002. The scoring system was derived based on 13 preoperative factors which were weighted; the sum of the scores, ranging from 0 to 17, allowed for stratification of postoperative risk of AKI from low to high. The lowest-risk group (score of 0–2) had a risk of AKI requiring RRT of 0.4%, in contrast to the high-risk stratum (score of 9–13) who had a RRT risk of 21.5%. However, there is a paucity in the literature of studies about the risk and risk prediction of AKI in unselected emergency admissions to a district general hospital. Finlay et al. 102 published a recent study of AKI risk factors associated with AKI in patients admitted to acute medical units in a study conducted over two separate 24-hour periods at a total of 10 acute medical units. Forni et al. 103 have developed a model for predicting AKI in a subset of medical patients admitted to a UK hospital. Their model included some physiological markers.

NHS England initiatives in acute kidney injury

NHS England has recently made considerable progress in improving patient safety with respect to AKI. A national algorithm has been developed to standardise the definition of AKI. A patient safety alert was issued on 9 June 2014 requiring all hospital trusts to embed the algorithm in routine pathology reporting. Although this will go a long way towards the identification of established AKI, it does not address the issue of prevention of harm in the first place, hence the requirement for the development of risk assessment tools. In addition, the reporting of AKI will not require a response by the attending medical team; in other words, there is currently no requirement for a standardised response to alerts to established AKI. Although this work goes along way towards highlighting established AKI, it does not address the issue of assessing the risk of AKI. This work and a future study examining the impact of an automated alert will inform further national strategy with respect to AKI prevention.

Our study and further studies examining the complexities of implementation of auditable clinical alerts have informed these work streams and in the future will provide more clarity in a complex area.

Intervention studies

To date there have been no intervention studies in comparable unselected populations that have shown a reduction in episodes of AKI. A formal health economic analysis of the impact and costs of such intervention needs to be performed, along with a technology appraisal to maximise the benefit from electronic alerting.

Quantitative analysis

Using a number of combined hospital databases, we have identified a number of key variables that predict AKI; this methodology can therefore be used to highlight patients at risk of AKI. We have validated our findings in second data sets from the same hospitals and then a further data set from a second hospital trust with very different population characteristics. The key variables used in these models are available in most UK hospitals and the majority of them are not susceptible to coding bias; however, the potential for such bias is always a concern. The use of physiological parameters to predict AKI for inpatients may pose a problem for some hospitals across the NHS because of the lack of electronic recording. Many hospitals are adopting electronic recording of physiological parameters as part of their patient-safety programmes, so these data are likely to become more ubiquitous in the future; the application of these models across the NHS is therefore a feasible proposition. As discussed previously, the model would benefit from further refinement and analysis.

Qualitative analysis

The aim of the qualitative arm of this project was to provide personal insight into, and explanation of, the processes involved in using the alert system through an experiential narrative gleaned via semistructured interviews. A total of 12 members of staff participated and, given this low number, generalisability cannot be applied to measure rigour nor to infer fittingness of the findings to the wider population; this is not the intention of qualitative research. Instead, the study has conformed to two leading qualitative principles to ensure the collection of rich experiences and to achieve credibility. First, there was a high level of adequacy regarding representation within the sample; the majority of consultant and outreach nurse users across the trust participated, thereby ensuring a full complement of views concerning the alert system. Second, the data collected through the use of semistructured interviews were, on the one hand, rich and diverse, but, on the other, clearly revealed consensus of opinion through thematic representation. Agreement of the strengths and weaknesses of the system, alongside contextual issues that helped or hindered, was reached in a number of core areas; thus it can be implied that data saturation was achieved. The high level of user representation alongside consensus of opinion suggests that this arm of the project has added a credible explanatory component to the wider study in terms of taking the system forward to the next stage.