Development of processes allowing near real-time refinement and validation of triage tools during the early stage of an outbreak in readiness for surge: the FLU-CATs Study

CURB-65

‘CURB-65’ is a validated predictor of mortality from community-acquired pneumonia in adults but it was never intended for use in children. 7,9 CURB-65 was not designed to predict mortality from non-pneumonic presentations. 8–10

Pandemic Medical Early Warning Score

Challen et al. 11 proposed the PMEWS as a clinical triage tool to aid hospital admission decisions for adults in a pandemic situation. The PMEWS score attributes an ordinal value to ranges of physiological measurements (respiratory rate, oxygen saturation, heart rate, systolic blood pressure, temperature and neurological assessment) and patient characteristics (age, social factors, chronic disease and performance status) to generate a score of between 0 and 20 (Table 1). They validated PMEWS in adults presenting to hospital with community-acquired pneumonia and found that it was better than CURB-65 for predicting the need for admission and higher levels of care, but had limited ability to predict mortality. CURB-65 and PMEWS pose problems for use in primary care: CURB-65 is reliant on a contemporaneous serum urea value, PMEWS could be computationally complicated for some, and both are designed for use only with adults. Many people presenting with ILI are children. Other severity scoring tools exist but these are not suitable for use in primary care because of a greater dependence upon laboratory or radiological investigations.

Department of Health Community Assessment Tools

In 2009, the Department of Health in England published a package of care that included paediatric and adult CATs and patient pathways for use by the national health services of the UK nations in a severe pandemic event, in primary and secondary care, and matched hospital care pathways. 12 CATs were developed to help non-specialist front-line staff identify, when resources are limited, which sick children and adults are most likely to benefit from interventions and levels of care that are available only in hospitals. CATs were developed by paediatric and adult expert clinical development groups drawing on evidence that supports the recognition of severe influenza and severe pneumonia in the community in adults and children in resource-limited settings, severe chronic obstructive pulmonary disease in adults, potentially serious feverish illness in children and severe bronchiolitis in infants. 5,13–22 Clinicians were warned not to use the CATs and the pathways unless the local situation precluded normal admission and discharge processes.

CATs use six objective criteria and one subjective criterion based on simple (binary) clinical assessment (Figures 1 and 2). Meeting any CATs criterion warrants referral and admission to hospital. Criteria are (1) severe respiratory distress; (2) increased respiratory rate; (3) oxygen saturation of ≤ 92% on pulse oximetry breathing air or oxygen; (4) respiratory exhaustion; (5) severe dehydration or shock; (6) altered consciousness level; and (7) other clinical concern. Although criteria fields are common to paediatric and adult CATs, the abnormal physiological thresholds and clinical signs are age appropriate. Like PMEWS, there is no requirement for laboratory investigation to complete the assessment. However, CATs were intended for use only ‘during severe and exceptional circumstances when surge demand for health-care services leads to a need for strict triage’12 and, as such, were not deployed during the 2009–10 pandemic.

FIGURE 1.

Paediatric Community Assessment Tool for children aged < 16 years. © Crown Copyright, Department of Health, 2009, reproduced under Open Government Licence for public sector information.

FIGURE 2.

Adult Community Assessment Tool for adults aged ≥ 16 years. © Crown Copyright, Department of Health, 2009, reproduced under Open Government Licence for public sector information.

Developing the real-time surveillance system

Eligible patients for the study were identified during GP consultations using LEPIS loaded on participating GPs’ computers. LEPIS monitors the consultation record for entry of any eligible diagnostic Read code (see Appendix 1). When a participating GP enters a relevant diagnostic Read code, a pop-up window appears prompting them to consider recruiting the patient; not recruiting; or suspending further recruitment prompts for a period of time (Figure 4). If recruited, a form appears for structured entry of data related to the consultation (i.e. signs and symptoms, physiological measurements and decisions relating to treatment or hospital referral). The linkage between the LEPIS and Vision^® systems enabled automatic triggering of tailor-made data entry forms based on age and sex for men, women and children (age < 16 years), with additional conditional questions relating to pregnancy status if female and age > 12 years. These forms are the electronic case report forms (eCRFs) for the study. The eCRFs were designed to encourage both positive and negative reporting. The eCRFs underwent revision after the first winter season to record if the GP actively decided not to make an assessment because the GP considered the feature to be grossly normal or abnormal. This important feature avoids bias in the analysis of data that might otherwise be considered missing at random, when, in reality, the data are missing because the GP considered the feature to be grossly normal or abnormal. This addresses one challenge of collecting data from routine consultation records, when many clinicians record only a limited number of important negative or positive features based on their personal practice.

FIGURE 4.

Screenshot of LEPIS pop-up window indicating an eligible patient.

Age- and gender-specific eCRFs were developed. Patients aged < 16 years are classified as ‘children’ and patients aged ≥ 16 years are classified as ‘adults’. Data collection forms for female patients > 12 years include a query on pregnancy status. A screenshot of a sample female adult LEPIS form is provided (Figure 5). The data collection form for children is identical to that for adults, except for three questions that are absent in the children’s form – blood pressure measurement, social isolation status and premorbid performance status – as these criteria are used only in the adult PMEWS triage tool.

FIGURE 5.

Sample data collection form for adult female.

Sociodemographic and relevant medical history data were extracted by a background process from the routine EHR to minimise GP workload.

The flagging system and web-based data collection form was user-tested by GPs at five practices. Sample web-based forms can be viewed at www.cprd.com/flucats/ (accessed 5 August 2015); these will be maintained for the duration of the study.

Data entry checks were incorporated in the eCRF so that measurement values could be entered only if they were within physiologically plausible ranges. To further minimise duplication of tasks, after the completion of each form, GPs are presented with a summary of their assessment along with any decisions made (to treat with antivirals and/or antibiotics, and referral to hospital). This can be cut and pasted, or downloaded in text or portable document format (PDF) for entry into the patient’s EHR. A screenshot of a sample GP summary generated following the submission of the data collection form is shown in Figure 6.

FIGURE 6.

Sample GP summary screen.

Eligibility criteria

All people (any age, any sex) presenting with ILI (regardless of date at onset of disease and prior medical history) to GPs participating in CPRD are eligible for this study. The challenge was to identify a set of Read codes that LEPIS would monitor and pop up to prompt GPs to recruit people with ILI, while not overburdening the GPs with irrelevant prompts.

Eligible Read codes were identified by systematic analysis of the CPRD database using a ‘reverse engineering’ process. First, we identified patients of any age admitted to hospital in the winter of 2010–11, who had a diagnosis of ‘influenza’, ‘an influenza-related condition’ or ‘complication of influenza’ in the HES data set, and extracted their CPRD/HES linked data. To identify these patients, we searched for terms such as ‘influenza’, ‘influenza-like illness’ and ‘acute respiratory infection’ (see Appendix 1 for initial list of eligible Read codes) in the Read-coded clinical terms dictionary. Next, the output was restricted to patients who had visited their GP in the 7 days prior to admission. Then we queried the CPRD data to list all Read codes used by GPs in these consultations that preceded the hospital admission. A total of 831 different Read codes were recorded for these GP consultations. Many of these consultations were unrelated to ILI. These Read codes were reviewed for relevance by PM and MGS, without sight of their frequency, to yield a shortlist of 39 codes. These codes were then listed by frequency and reviewed by the study steering group (see Appendix 2). We decided to use all of these codes at least initially as a previous consultation with GPs suggested that GPs prefer syndromic codes for ILI to reflect the diagnostic uncertainty in the absence of laboratory confirmation of influenza. It is our intention to refine the LEPIS trigger list at an interim analysis during the final phase of the study, that is, during a pandemic. 33

Recruitment of general practitioners

General practitioners from practices that were already contributing to the CPRD were invited to participate in the study by the CPRD team initially by post and followed up electronically. Further recruitment of CPRD GPs was conducted via the NIHR Primary Care Research Networks (PCRNs), and by CPRD and investigator activities at regional and national meetings of GP research groups. GP practices were required to install the LEPIS software on their practice computer system and were given remote IT support. GPs were briefed on the eligible Read codes so that they had the option of proactively triggering the LEPIS pop-up window. No monetary incentives were provided to GPs for participation in the study. Following correspondence with the Royal College of General Practitioners (RCGP), they were sent certificates to include in their continuing professional development portfolios.

Data linkages, validation of primary outcome measures, influenza surveillance data and expansion of study to other electronic health record systems

The study protocol included plans to validate the primary and secondary outcome events using HES data for ‘hospital admission within 24 hours of the consultation’ and ONS data for the outcome ‘death (all causes) within 30 days of the consultation’. In addition, at the request of NIHR reviewers, the feasibility study scope was expanded to explore linkages with Public Health England (PHE) data sets for data on microbiology and virology. The feasibility study also explored options for expansion of the study-specific technological infrastructure to all CPRD and non-CPRD GP practices in the event of a pandemic.

Automation of data extraction, analysis and reporting

Algorithms or ‘do files’ were developed and tested using Stata version 13 (StataCorp LP, College Station, TX, USA) to automate data linkage, extraction, analysis tasks and reporting to facilitate timely production of weekly and monthly reports on the incidence, progression and outcomes of pandemic influenza or a similar pandemic caused by a respiratory virus manifesting as ILI (see Appendix 3 for ‘do files’).

Evaluating user experience

Feedback on the ease of use of the study flagging and data collection system was obtained from participating GPs via e-mail consultation and one-to-one telephone interviews. Interviewees were provided a financial incentive for participation. A semistructured interview guide was used for the telephone interviews (see Appendix 4). Interviews were audio-recorded and transcribed verbatim for thematic analysis. Data were analysed using a thematic analysis approach that enables the researcher to identify, analyse and report themes or patterns in the data collected. The analysis followed the steps outlined by Braun and Clarke,34 beginning with familiarisation of the data through listening to the recording of the interviews, transcribing the data then reading and re-reading each interview, making initial codes in the margins.

Coding was conducted systematically, taking each interview initially in turn and then subsequently moving back and forth across the data set. In order to establish consistency in coding, and thereby enhance dependability and credibility, the transcripts were independently double-coded by two of the authors (SV and PRM). Initial themes were reviewed and refined further, and a revised thematic table of candidate themes was produced.

Aim

To investigate the predictive performance of the triage tools CATs and the PMEWS for hospital admission and death in patients presenting with ILI to primary care, using a novel near real-time data collection, collation, linkage and analysis process.

Population and recruitment

Patients of all ages presenting to study GPs during the reporting period 19 March 2013 to 31 March 2015 (data upload 13 April 2015) with ILI were eligible for inclusion in the study and no exclusion criteria were applied. LEPIS was activated. Patient recruitment was determined by individual GP convenience rather than a specific sampling protocol. We estimated a minimum sample size of 1000 patients, assuming an event rate of 5% (based on the UK mortality and need for interventions in hospitalised influenza patients during the 2009 A/H1N1pdm2009 pandemic26) to test the hypothesis that the triage tools being tested would have an AUROC value of > 0.05, demonstrating the ability of the tools to discriminate between patients who experienced an outcome of interest and those who did not. 35,36

Design

Open cohort study involving follow-up for 30 days after the initial consultation for ILI.

Predictor variables

These include criteria from both the adult and child CATs (see Figures 1 and 2) and the adult PMEWS (see Table 1). Covariates that were extracted from the routine EHRs included patient sociodemographic characteristics (age, sex), comorbidities associated with an increased influenza risk (cardiovascular disease, chronic liver disease, neurological conditions, chronic renal disease, chronic respiratory disease, diabetes, immunosuppressive conditions), previous prescriptions of statins, antibiotics, influenza-specific antivirals, inhaled and oral corticosteroids, history of seasonal influenza vaccination, pneumococcal vaccination and Haemophilus influenzae type B vaccination. The detailed code list for disease covariates is provided in Appendix 5.

Outcome variables

There were two primary outcome variables to be collected from the GP EHR: hospital admission within 24 hours of GP assessment (binary categorical variable, coded as yes/no) and death (from all causes) within 30 days of GP assessment (binary categorical variable, coded as yes/no). Secondary outcome variables included ‘any need for augmented level of care (admission to high-dependency units or intensive care units coded as a binary categorical variable)’, length of hospital stay [three binary categorical variables were created using different thresholds of stay, < 48 hours (yes/no), ≥ 6 days (yes/no), ≥ 12 days (yes/no)], GP decision to refer to hospital (binary categorical variable, coded as yes/no), GP decision to prescribe antibiotics (binary categorical variable, coded as yes/no) and GP decision to prescribe influenza-specific antivirals (binary categorical variable, coded as yes/no).

Statistical analysis (planned and actual)

A descriptive analysis of patient sociodemographic and clinical characteristics was conducted. The planned analyses included an investigation of the association between each CATs criterion and the various outcome measures. Each was investigated, in turn, using univariate logistic regression analysis. In addition, a multivariable model was planned to identify which of the CATs criteria were significant independent predictors of outcome when included in the same model. A separate investigation of the association between other covariates (as listed above) and outcomes using a similar approach was also planned. The analysis plan also considered predictive performance of a combined CATs score (both non-weighted and weighted, which would incorporate all CATs criteria), as well as the effect of varying individual criteria and combined score thresholds (e.g. respiratory rate of > 30, > 35, > 40 breaths per minute or a combined CATs score of > 3, > 4, > 5). Finally, a comparison of predictive performance with the PMEWS score was planned for adults only. Predictive performance would be assessed using measures including sensitivity, specificity, positive predictive value, negative predictive value and AUROC curve values. Separate analyses were run for paediatric and adult patients. All statistical analyses were conducted using Stata.

Development of real-time analysis and reporting

Three distinct stages of work were required:

1. Data cleaning and management Some data management of the weekly data as provided by CPRD was necessary to enable analyses. Most of the consultation-specific data collected via the web-based forms were provided as string variables, and data management mainly involved converting them to numeric variables or date variables where appropriate. These tasks have been automated and tested for accuracy in Stata 13 to enable an analyst to complete these usually labour- and time-intensive tasks in < 10 minutes once the data are received.

2. Analyses After cleaning, the following analyses were planned: descriptive analysis involving tabulation and summarising of the data; logistic regression analyses (unadjusted and adjusted) to investigate the association between each of the triage criterions and the primary outcomes; and an assessment of the predictive performance of different triage tools using AUROC curve comparisons. All analytical tasks have been automated using Stata 13 and can be completed in < 1 hour by an analyst.

3. Reporting Automated analyses and reporting mechanisms have been set up within Stata to report key findings from the weekly and monthly data sets. Findings currently being reported include frequency tables and charts, and results of unadjusted and adjusted logistic regression analyses. Weekly data set findings are reported in a Microsoft Excel^® version 2010 (Microsoft Corporation, Redmond, WA, USA) spreadsheet and monthly findings in a Microsoft Word^® version 2010 (Microsoft Corporation, Redmond, WA, USA) rich text format (RTF) document. This task has been semi-automated in Stata 13 to facilitate the production of weekly and monthly reports in Word and PDF format for policy-makers.

In addition, a Stata ‘do file’ was developed to extract background data from the CPRD database on comorbidities and select medication for all patients who underwent a FLU-CATs Study consultation. This has been tested by the CPRD team and has now been automated to provide the monthly data extracts. Detailed descriptions of each of the above three steps, along with instructions on how to use the Stata ‘do files’ are contained in the FLU-CATs Handbook (see Appendix 6). All ‘do files’ have been tested and validated with each of the weekly and monthly data uploads since the first weekly data tranche was received on 14 July 2014. The April 2015 data set (13 April 2015 tranche of weekly data) was the last tranche of data that was analysed at the time of preparation of this report. All findings presented in this report relate to the FLU-CATs data as of April 2015. Detailed results are provided below (see Phase 2: pilot study, data collection, extraction and statistical analysis).

Continuous weekly testing of these three processes has allowed us to reduce data management, analytical and reporting times to 1 day. By adapting the CPRD EHR structure we have also eliminated a separate electronic data entry step and streamlined data collation while maintaining data security.

We currently do not have sufficient data to report results from the AUROC analyses, even although we have recruited 863 eligible patients so far because influenza activity has been low and very few of our patients have progressed to the outcomes of interest.

Data linkage

A ‘do file’ was written to extract relevant variables (hospital admission, need for augmented level of care, length of stay and death) from HES and ONS data. This ‘do file’ has been tested by the CPRD team using historic HES and ONS data sets. CPRD have confirmed that the ‘do file’ is automated to periodically extract HES data for patients with FLU-CATs GP consultations. However, as of April 2015, no HES or ONS linked data have been received from HSCIC since January 2014. The HES/ONS data received in January 2014 related to the HES period up to the third quarter of 2013 and preceded the first winter influenza season that this study included. Therefore, validation of the few observed outcome events using the FLU-CATs pilot study data has not been possible.

Linkage to PHE data sets was explored and abandoned because of inability to anonymously link patients in CPRD with PHE records with confidence, mostly due to virology requests from primary care not including sufficient unique patient identifiers. As an alternative solution, linkage is being explored at a practice and patient level in collaboration with the RCGP Research and Surveillance Centre.

General practitioner participation and user experience

As of April 2015, a total of 30 GP practices participated in the FLU-CATs Study, although at any one time only 25 GP practices were actively recruiting. There were a total of 704 adult FLU-CATs consultations (702 single consultations and two repeat consultations) and 159 single consultations for children.

All 30 participating GPs were invited to be interviewed to evaluate the user experience. A total of six GPs agreed to be interviewed (Table 2). The interviews are summarised below.

The purpose of conducting these interviews was to explore GPs’ experiences and views of being involved in a real-time influenza surveillance research project in order to help inform future consultations. GPs who participated in the FLU-CATs Study were sent invitation letters by CPRD: 11 GPs returned expressions of interest, of which six were interviewed over telephone. Each interview lasted between 10 and 20 minutes. A payment of £40 (based on the PCRN recommended rate of £80 per hour) was made to the GPs for their participation in the qualitative interviews.

The main finding that emerged from these interviews was that the LEPIS trigger pop-up and FLU-CATs data collection forms were easy to use. The simplicity of the eCRFs encouraged GPs to participate in the study despite there not being any financial incentive for participation. However, the setting up of the LEPIS system to enable data collection for the study was fraught with technical difficulties. Although the support from CPRD in resolving any technical issues surrounding LEPIS installation was appreciated by the GPs, technical difficulties persisted for some and this reduced patient recruitment. The FLU-CATs process was quite easy to conduct and did not interfere with the routine GP consultation. All interviewed GPs agreed that the FLU-CATs system, with a few modifications, was ready to be used in a pandemic scenario.

1. Introduction to, and involvement with, the FLU-CATs Study:

   1. Of the six GPs interviewed, three were introduced to the FLU-CATs Study through the research networks (NIHR PCRN and other local research networks), one through a research coordinator and one directly by CPRD. The remaining GP did not remember exactly how he/she was introduced to the study but thought that it might have been through an e-mail.

2. Lack of financial incentive as a barrier to participation:

   1. One GP said that the lack of financial incentive for the GPs’ participation in the FLU-CATs Study was ‘inevitably, a barrier’, whereas another GP said that it was not. One GP thought it might have been a barrier if the FLU-CATs Study was the first ever CPRD study in which the GP was participating but, as he/she had participated in a few before it was not a barrier for him/her. The consensus among the interviewed GPs, however, appeared to be that as the FLU-CATs Study was relatively straightforward and not ‘too onerous’, as one GP put it, the lack of a financial incentive was not a barrier to their participation. Interestingly, one GP said ‘My partners were a bit reluctant because there was no remuneration at all, but we thought it was relevant because it was relevant to the influenza season and I just thought it would focus our minds on how to manage respiratory tract infections’. One GP said the following about the lack of incentives for participation in the study: ‘I think the only difficulty, as I was saying at the beginning, with that is it’s a bit difficult to say why would I recommend doing it and my argument is because it makes me feel like a better person because I do, but does that make somebody else feel that it makes them feel like a better person if they do. I think the other thing is if you are thinking about incentives, I don’t think in general practice, the incentive needs to be particularly large, it just needs to be an appreciation of the fact that there are costs involved in doing things and the amount of money we got for [another study name] and [another study name], I would say was well in excess of what was necessary to reimburse us for the amount of work that we had done. And, I think if you were pitching something like FLU-CAT, looking for a price, then I think you’d be talking something like a couple of pounds a form not a couple of hundred pounds a form. Again, I think for the future of CPRD, I don’t think they would have to have terribly large incentives to make it a reason why people would want to do it. It’s perhaps where there has been some difficulty about it, it’s just the total lack of an incentive. And, basically, in general practice, if somebody isn’t paying you anything it’s costing you’.

3. LEPIS system installation:

   1. LEPIS pop-up window According to one GP, this issue was serious enough to affect patient recruitment significantly: ‘I don’t think it pops up with consultations, so I don’t think we’ve hardly recruited anybody. So, it’s a real shame because it was easy when it worked, it was dead easy’.

   2. Two GPs reported issues with the clinical IT system Vision while using the LEPIS system. According to one of them, ‘It was very difficult because we are on EMIS-WEB now but we were on Vision and it took a long time to get your software to work. Fortunately, we had a medical student who worked in our office temporarily who was sort of, you know, he was doing a Masters, who was very IT aware and he spent a lot of time talking to Vision and talking to you and trying to get it working and eventually he got it to work, but that was a real pain. And, now we’re on EMIS-WEB we can’t do it all because apparently it doesn’t work with EMIS anyway’. Another issue reported was that the LEPIS system would ‘hang’ while entering the FLU-CATs form and the GP did not use the system for a while because she did not want the screen to hang during a consultation. (EMIS^® is an alternative EHR system.)

   3. One recommendation to improve the system was to change the point at which LEPIS was triggered: ‘The biggest problem though I think with it is the fact that what triggers LEPIS is the receiving of the code not the selecting of the code and I don’t know, I mean I have said this before, that we use DXS which is the decisions support system and as soon as you select a code it activates DXS but if you select, let’s say flu-like illness, and then you type to the end of your clinical note, you then save it and LEPIS triggers, you’re less likely to want to interact with LEPIS than you would be if as soon as you’ve selected the code flu-like illness, it triggered it. And, that I think is the biggest obstacle to FLU-CAT being done, where it triggers, because you see by the time it triggers you’ve finished the consultation, you’ve finished the recording of the consultation and you either decide am I going to bother going back and doing it. . .I think that’s an inherent problem of the way that LEPIS works at the moment, that you really want to as soon as you put in that somebody’s got I don’t know, haematuria, you want it to bring up your haematuria study. You don’t want it to let you write the whole of your clinical history down and then trigger it’.

   4. Another GP had reported technical difficulties with the LEPIS system, and had stopped recruiting for the study and intended to resume once the issue was resolved. The GP remarked that he/she would have found it useful to have received an e-mail or some communication indicating that the issue was resolved, as he/she had realised the system was fixed only on the morning of the telephone interview when checked to see if worked.

   5. Despite having experienced difficulties with the LEPIS system, all interviewed GPs acknowledged the support from the CPRD team in helping resolve their technical glitches.

   6. According to the CPRD team member responsible for the web development: ‘Yes, I mean it’s a mixed bag really, because you find some practices that are computer literate and some of them not computer literate and so we try to say this is how it works and some of them don’t actually grasp what’s required for the study, so they would say well it’s not working but they haven’t actually done the preliminary installation or for example like, if we take the example of the FLU-CATs, because although there is a wide range of Read-codes, perhaps they put a Read-code which is not in the list, so therefore obviously it would not pop out to say this patient is eligible . . . Yes, I mean you would know more than me when you describe the flu, everyone describes it differently, so that’s why we have included more and more Read-codes because if you. . .what we had done for flu example we say, oh this is only the ten Read-codes, but perhaps another doctor will describe it differently and it is another Read-code, although it means the same thing, but it is another Read-code, so we have worked towards including more Read-codes in order to encompass everything, or we try to educate . . . a big word educate, you can’t educate a doctor. . .but tell them that basically if they want to recruit for this study, if you want to take part, this is the list of the Read-codes. So, therefore there is that communication when you go’.

4. Nature of the FLU-CATs consultations:

   1. All six GPs agreed that the FLU-CATs consultations were very similar to routine consultations and that the FLU-CATs eCRF was quite easy to complete. One GP said that he/she would fill out the FLU-CATs form after the consultation with the patient and remarked, ‘I supposed I had to make note of the blood pressure, which I might not have done had it been a normal viral illness, you know I wouldn’t necessarily have done that, yes’. Two GPs said they filled out the FLU-CATs forms during the consultation while the patient was still present. According to one of the GPs, ‘One of the main people that did our FLU-CATs study returns has been our nurse practitioner and she quite likes it, it’s quite helpful to her and I mean, I find it’s not at all unhelpful to me, provided I get it triggered at the right time’.

5. Readiness of the current FLU-CATs system for use by all GPs in a potential pandemic situation:

   1. All six GPs thought that the FLU-CATs system would be ready for use in a pandemic situation with minor modifications. Two GPs expressed the need for the data entry to be better integrated within their clinical system; according to one of them, ‘I mean it doesn’t take that long. It [FLU-CATs form] was a couple of minutes extra I guess to do it compared to doing the consultation without it. So, in the middle of a pandemic that sort of translates to quite a lot of extra work I suppose. If you could capture straight into the notes without having to sort of. . .it’s a bit faffy at the end, sort of copying and pasting and it wasn’t the way the data was dropped into the notes would have been fine but it was pretty much impossible for anyone else to use really. But, otherwise it was okay’.

   2. Another GP thought that the lack of pulse oximeters in every consulting room would need to be addressed before the FLU-CATs system is recommended for use in a pandemic situation.

   3. All six interviewed GPs said that they would be very interested in reading any outputs that result from the FLU-CATs Study.

The association between Community Assessment Tools criteria and outcomes of interest

The distribution of events across various criteria and covariates resulted in a low number of events in some categories despite the pilot study reaching nearly 86% of the a priori sample size estimates, and so running a fully adjusted logistic regression model (adjusting for all confounders) was not possible. Therefore, an unadjusted and a multivariable model including all of the seven CATs criteria, adjusted for each other, was run. Generally, because the data were sparse, wide 95% CIs and variable omissions (from the analyses) were seen. Results from the children’s analyses are presented in Table 8 and results from the adults’ analyses are presented in Table 9.

Outcome 1: decision to treat with influenza-specific antiviral drugs

Given that only five adult patients were prescribed influenza-specific antiviral drugs, odds ratios (ORs) and 95% CIs could be obtained for only three criteria in the unadjusted and adjusted analyses. CATs criteria F and G showed strong positive associations (OR 174.5, 95% CI 9.2 to 3303.9 and OR 15.4, 95% CI 1.7 to 139.0, respectively) in the adjusted analyses, but none of the associations was statistically significant in the adjusted analyses.

None of the participating GPs decided to treat any of the recruited children with influenza-specific antiviral drugs, so no further analysis for this outcome was possible for this group.

Outcome 2: decision to treat with antibiotics

In adults, CATs criteria B and G were statistically significantly associated with this outcome in the unadjusted analyses, and remained statistically significant with consistent estimates in adjusted analyses. After adjusting for other CATs criteria, adults presenting with CATs criterion A showed a 10% decreased odds (95% CI 0.9 to 0.9) of being treated with antibiotics and those presenting with CATs criterion G were associated with an increased odds of being treated with antibiotics (OR 6.4, 95% CI 3.2 to 13.0).

In children, although CATs criterion G showed a 10-fold increase in odds of treatment with antibiotics (OR 10.1, 95% CI 2.3 to 43.9) in the unadjusted analyses, this was not seen to be statistically significant after adjustment for other CATs criteria.

Outcome 3: referral to hospital

In adults, CATs criteria A, C, E and F were positively and statistically significantly associated with referral to hospital in the unadjusted analyses. CATs criteria C and E remained statistically significant in the adjusted analyses, with increase in odds of hospital referral by 11.7 times (95% CI 1.7 to 81.2 times) and 49.0 times (95% CI 6.1 to 396.3 times), respectively.

In children, CATs criteria A, D and F were significantly associated with the outcome, although statistical significance was lost after adjustment for other CATs criteria.

Phase 1

The CPRD FLU-CATs team identified a number of technical and practical issues during the course of the feasibility study. These were captured via one-to-one interviews with the study team members and are listed below:

1. Monetary incentives would be necessary to increase GP participation unless there was a statutory requirement to systematically collect data on all possible influenza cases during a pandemic.

2. Funding for system maintenance and updating between study phases was not requested in the original grant. There is a cost associated with keeping the system updated and compliant with the evolving technological infrastructure, and to facilitate rapid re-activation in case of a pandemic.

3. External issues: bureaucratic processes, which are expected to be resolved in time, stalled provision by the HSCIC of contemporary HES/ONS linked data. Access to virology data from PHE is unlikely to be resolved because of lack of adherence to data standards by those requesting virology swabs tests in the community. Collaboration with the RCGP Research and Surveillance Centre may provide linked–anonymised virology surveillance data at a practice level and possibly at patient level.

4. Briefing of GPs on the study and LEPIS software installation: installing the LEPIS software was generally seen as being quite difficult and time intensive for practice staff. Many practices required regular technical advice from CPRD on this matter. Dedicated IT support is required to aid GPs at set-up.

5. LEPIS did not install at some of the practices, so these GPs were not able to participate in the study.

6. The reorganisation of primary health care from Primary Health Care Trusts to Clinical Commissioning Groups was associated with preferences for which EHRs are used across Clinical Commissioning Groups. The popularity of Vision appears to be declining in favour of EMIS. It is estimated that 50% of GPs now use EMIS. LEPIS does not work on EMIS and so an alternative solution will need to be developed. Thus, most importantly, to conduct the study in a pandemic situation we would need a flagging system that could work across several clinical IT systems.

7. Any decision aid based on the validated criteria would better be delivered separately on an open web-based platform or mobile phone ‘app’.

Phase 2

The two main lessons learnt regarding data analyses relate to missing data, clinical data not being measured and the delays in obtaining HES and ONS linkage data.

1. Missing clinical measurements Up to 74% of some clinical measurements (blood pressure in adults in this instance), were not recorded as part of GPs’ routine assessment of an adult person presenting with ILI. We would question the utility and adoption of triage tools that depend upon a clinical measurement that is not used in the routine assessment of ILI for use in a time-pressured pandemic situation. During the early phases of the pilot study, the low recording of clinical measurements prompted the researchers to modify the form to include two further categories: ‘not measured – grossly normal’ and ‘not measured – grossly abnormal’. This was based on informal feedback that the clinical examination did consider factors such as fever, respiratory rate, blood pressure, etc., but that formal measurements were triggered only if they appeared to be grossly abnormal. Our study was not an interventional study and therefore tried to approximate the ‘routine’ clinical consultation. Although failure of exact measurements resulted in poor recording of PMEWS criteria, we were able to derive the CATs criteria for the majority of patients. Therefore, it is reasonable to assume that CATs offers a more user-friendly and efficient option for GPs over a more data-intensive triage tool such as PMEWS.

2. Linkage with HES and ONS In the absence of contemporary linked data from HES and ONS, it was not possible to prospectively validate the primary outcomes (hospital admission and death) or study secondary outcomes (e.g. admission to intensive care, length of hospital stay) or examine cause of death. However, it is important to note that the CPRD primary care data routinely records death data and hospital admissions, so while there may be a slight underestimation of these data, it would still be possible to study these outcomes using primary care data alone (although possibly with a 1-month lag without active follow-up of flagged patients).

The LEPIS software and accompanying web-based data collection form have generally been regarded as being quick and easy to use after the initial set-up, taking an experienced user about a minute to complete. However, the research team has identified a number of limitations. One is that the LEPIS software is specific to GP practices that use the Vision system and incompatible with GP practices using EMIS, SystmOne^® (Horsforth, Leeds, UK) or other EHR systems. Although the popularity of Vision is reducing, there are still about 650 practices using this system and contributing to CPRD. If all of these GPs could be deployed in the event of a pandemic, there could be a population-based surveillance system providing near real-time reports of the course of the pandemic. There were technical problems with compatibility of the LEPIS software with some configurations of the GP computer systems, which need to be resolved. Moreover, although being very easy to use, LEPIS was seen as being quite difficult to set up, and technical assistance was necessary to help GPs to install the system. Both of these points are important limitations that would prevent a rapid national level rollout using the existing LEPIS software to trigger data collection. A decision aid based on the validated criteria would better be delivered separately on an open web-based platform or mobile phone ‘app’.

A lack of any financial incentive was cited by several practices as a reason for not participating. Several GPs felt that even a minimal payment of a few pounds would be sufficient remuneration for the research activity. We will add a small payment for each case submitted in future seasons.

This study is one of the UK NIHR portfolio of pandemic preparedness studies. The investigators are funded to establish capacity and processes during an inter-pandemic period in readiness to run the study during the early stages of a future outbreak of pandemic influenza or other outbreak of severe acute respiratory infection of public health concern. Our results show that a web-based triage tool, which uses simple clinical assessments, can be used with ease during routine consultations, and that linkage to the patient’s EHR with automated background processes can be used to define the performance of triage criteria triage tool against outcome events. Using the processes that we have developed, the demonstration of the proof of concept indicates that the study will be of value during the early phases of a novel pandemic and capable of providing a valid triage tool in readiness for surge. If surge occurs, the web-based process could be switched from data collation to provide a decision aid. This would best be provided on common web-based platforms or mobile phone ‘apps’.

To scale up the study during a pandemic period will be challenging, as we recognise that about half of willing CPRD practices in the pilot had difficulties installing the LEPIS software, and half of all practices in England and Wales use a different EHR system. That said, the EMIS system also has capability to monitor consultations and trigger to structured data entry forms. The investigators are in negotiation with EMIS to develop a solution for that system.

This study adheres to the five principles of Dynamic Risk Assessment, as applied to the management of emergency situations by UK Government agencies (Evaluate, Select, Assess, Refine, Reassess). 37 That said, the proposed use of triage tools by the NHS in the UK during pandemic surge has been criticised, in particular because of lack of evidence of engagement with the general population and with local ethics committees. 38

The data collection period of the study to date covers two calendar years, including the set-up and pilot in sequential winter seasons. At most, only 25 GPs were recruiting cases. PHE described influenza activity in the UK as low in 2013–14 and moderate in 2014–15. 39 Perhaps as a result of these factors, and the low-to-moderate influenza activity, only a few of the 863 patients had primary outcomes of interest (death n = 15 and referral to hospital for admission n = 17).

It is possible that GPs chose to ignore the LEPIS prompts when patients with severe illness presented, thus introducing bias to the study. However, the hospital referral rate observed for this study of 19.7 per 1000 consultations is within the ranges quoted for several larger studies that examine variation in GP referral rates in the UK. 40

A potential limitation of studies such as this relates to possible misclassification of criteria as a result of clinicians’ preference to record only certain important negative or positive features. We avoided misclassification of data seen in our previous studies by use of required data entry fields with exclusive options.

This was a non-interventional study. Although the clinical assessment data collection tool (eCRF) was structured to capture evidence-based criteria, recognised in national guidance, it did not require the practitioner to make any change in his/her normal assessment – only to record the assessment. We thought that the presentation of a structured assessment form may result in personal reflection on the nature of the GP’s own assessment and so alter the completeness of future assessments. We were therefore surprised to find that some clinical measures are not consistently taken with many patients during consultations. Three-quarters of adults did not have their blood pressure taken, and 60% of adults and 43% of children did not have their respiratory rate measured. Our data collection system did allow GPs to record some clinical assessments, such as respiratory rate, as ‘not measured as grossly normal’ and ‘not measured as grossly abnormal’, which reflects pragmatic or usual practice. This finding is important, as it strongly suggests that criteria that rely upon a measurement such as blood pressure, and which is not being used consistently in routine assessment of ILI, should not be included in a triage tool that is designed for use during surge, when time and resources are in even shorter supply.

Denial of access to linked HES and ONS data by the HSCIC has meant that, at the time of writing, it has not been possible to validate the primary outcomes (admission to hospital and death) or secondary outcomes (admission to intensive care and length of hospital stay). A retrospective analysis of the existing data will be possible when the linked data are made available.

Weekly reports

For weekly reports, the ‘clean’ weekly Stata file is to be used. The input and output directories will need to be checked and modified if necessary. Once this has been done, the ‘do file’ may be executed. This should result in a complete run with no errors and a new Excel spreadsheet (titled ‘Results.xls’) with the weekly report should be created in the specified output folder. It is recommended that the results Excel spreadsheet is copied and re-saved in a different directory before running the next set of analyses, as these the ‘do files’ are programmed to overwrite the existing results spreadsheet.

At the time of writing this handbook, the latest tranche of weekly data for children (uploaded on 27 April 2015) does not have sufficient observations for outcome #1– ‘decision to treat with antivirals’ to obtain ORs, unadjusted or adjusted, for association with each of the CATs criteria. Because of this, the automated weekly report ‘do file’ terminates at this point. Until sufficient numbers for this outcome are obtained, the following is recommended:

1. Run the ‘do file’ until the error message is seen (until Outcome #1).

2. At this point, select portions of the ‘do file’ from Outcome #2 onwards and execute.

Monthly reports

From the monthly tranche of FLU-CATs data, the ‘master file’ and the ‘patient file’ will be used to generate a word document summarising findings for each month. As with the weekly data report ‘do files’, the directories with have to be checked and modified if necessary. Once this has been done, the ‘do file’ may be executed. This should result in a complete run with no errors and a new Word document (titled ‘mydoc1.rtf’) with the monthly report should be created in the specified output folder.

Weekly data set: adult (as provided by Clinical Practice Research Datalink)

Weekly data set: children (as provided by Clinical Practice Research Datalink)

Monthly data sets

Four data sets are uploaded on to Dropbox by CPRD each month: two form data sets (one each for adults and children), one master file and 1 patient file. The two form files contain cumulative data from 4 weeks in one ‘monthly’ data set each for adults and children. The variables in the form files are identical to the variables in the respective weekly data extracts. Descriptions for the master file and the patient file are given below.

Protocol amendment

The Award Board asked the investigators to consider add an extra resource to include virological diagnosis in a sub sample of patients.

We will explore linking GPRD to the HPA supra-regional microbiology network database to provide linked-anonymised results on microbiological investigations for respiratory pathogens. This will still require NIGB approval and Caldicott compliance. Preliminary enquiries have discovered that a supra-regional database is used by several of the nine HPA regions. This includes patient linked data on communicable diseases including the HPA/RCGP influenza virus swab results. GPRD is now linked to ten NHS datasets (including Hospital Episode Statistics, death certificates, cancer registry data, data from the Myocardial Infraction National Audit Program [MINAP]). These linkages are done on a regular basis using a Trusted Third Party and are all subject to NIGB approval. The project submitted involves already established links between GPRD, HES and Death Certifications. Establishing linkage to the HPA data-set would be a significant additional task for the study but one that would bring great benefit and avoid need for additional swabs to be taken.

Further details of protocol amendments including revised costs to resource linkage to the HPA are given at the end of this document.

Has the project changed since the Expression of Interest was submitted? Yes

The title has changed to reflect that the aim of the study is to prospectively refine and validate the triage tools in real-time ready for use in the event of surge.

We have added Dr Jamie Kirkham, Centre for Medical Statistics and Health Evaluation, University of Liverpool, as a co-investigator to provide methodological and statistical support to the study at 3% WTE.

We have added a full time pre-doctoral research assistant based at the University of Nottingham to conduct statistical scripting and run analysis in year one. In lieu of this substantial change and additional salary costs, University of Nottingham have agreed to make a 100% institutional contribution to their estates and indirect costs.

Work underpinning this study

Goodacre and colleagues (2010) conducted an evaluation of the discriminatory value of the CURB-65 score, PMEWS and CATs for predicting severe illness or mortality in patients with suspected pandemic influenza, but were unable to draw any conclusions regarding their clinical utility in a pandemic situation due to insufficient case numbers especially of adults, and a low incidence of severe outcome.[20]

Semple, Myles, Van-Tam and other members of the UK Pandemic Influenza Clinical Information Network (FLU-CIN) characterised PCR-confirmed pandemic influenza disease in a much larger cohort of 1520 people (1040 adults, 480 children (age < 16 years)) admitted to hospital.[21] (Thorax submitted) FLU-CIN compared the clinical validity and utility of CATs, PMEWS and CURB-65 as predictors for interventions normally only available in hospital, higher levels of care, and death using area under the Receiver Operating Characteristic (ROC) curve (AUROC) comparisons with 95% confidence intervals (paper submitted).[22] CATs showed the best predictive performance for level 2/3 admissions in both adults [AUROC: CAT 0.77 (0.73, 0.80); CURB-65 0.68 (0.64, 0.72); PMEWS 0.68 (0.64, 0.73), comparison of AUROCs p<0.001] and children [AUROC: CAT 0.74 (0.68, 0.80); CURB-65 0.52 (0.46, 0.59); PMEWS 0.69 (0.62, 0.75), p < 0.001].

While the FLU-CIN cohort is limited to patients admitted to hospital with severe influenza and its complications, the data show that triage tools are capable of predicting higher levels or care and or death in children and adults. However the FLU-CIN analysis does not include assessment of triage tools in primary care.

Appropriate use of such triage tools in the community could expedite referral to hospital and where scores are high, immediate admission to level 2/3 care. Prompt admission and allocation of higher levels of care may be associated with improved patient outcomes. Another study by FLU-CIN found that delayed admission to hospital (≥ 4 days after symptom onset) was significantly associated with increased likelihood of admission to critical care and death.[23]

The validity and utility of using triage tools in the community remains untested. Morbidity and mortality rates were low during the H1N1(2009) event when compared to some previous influenza epidemics such as the one in 1989/90 [24] and the use of anti-viral therapy was generally low in the FLU-CIN cohort despite it being widely available at the time. A more severe pandemic may be associated with a greater acceptance of anti-viral therapy and this may impact upon need for higher levels of care and death. Consequently criteria threshold values may need to be adjusted to optimize the ROC for each criterion and the AUROCs for the various triage tools.

Justification of this study

The validity and utility of the various triage tools needs to be assessed in a large community-based prospective study of patients presenting with influenza like illness to General Practitioners (GPs), to give confidence to clinicians who make be asked to use such tools in the event of surge, and policy makers who may need to recommend their use to GPs.

Linking GP assessment and management decisions to hospital episode data enables assessment and comparison of the validity and utility of triage in the community in relation to patient relevant outcomes (hospital admission, length of stay, higher levels of care and death).

It would not be possible to conduct this study during a pandemic without prior development of processes, feasibility and pilot studies.

The Health Protection Agency timeline for the UK 2009 pandemic shows only 12 weeks between identification of person-to-person transmission in the UK (first week May) and peak influenza activity (last week July) in the first pandemic wave. Prospective data collection with near real-time iterative and cumulative analysis is the only method for validating triage criteria and tools against a novel pathogen in such a short time.

Since pandemics are unpredictable and infrequent, limited but potentially useful information will be gained from prospective feasibility and pilot work conducted in primary care during seasonal influenza while H1N1v is still circulating.

Conducting this study in real-time during the early stages of a pandemic, when the characteristics of the novel virus are not fully understood, is important as it allows refinement and validation of triage tools against the novel pathogen in preparation for possible surge. This cannot be done until a novel virus emerges. Dame Diedre Hine has recommended that population based studies be established that, in the early stages of a future pandemic, can measure the severity of the pandemic and support decision making.[25]

The study method adheres to the five principles of Dynamic Risk Assessment and as applied to the management of emergency situations by UK Government agencies (Evaluate, Select, Assess, Refine, Reassess) (Home Office Guide to Operational Risk Assessment - Generic Risk Assessment 3.2. [Version 2 September 2008]).

If, as in the 1918/19 epidemic, the behaviour of the virus is markedly different in terms of severity between the first and subsequent waves or evolves to cause severe disease in a particular organ system; then triage criteria may need to be adapted to reflect the consequent changes in health care demand and clinical presentation.

Aim

To establish processes now, that can be used in the early stages of a future pandemic event to provide valid community triage tools capable of assisting hospital referral decisions for people of all ages for use if health care demands exceed health care capacity (surge).

Primary objective

From the start of a future pandemic event, to describe on a weekly cumulative basis the association between various triage tools and pandemic influenza outcomes. This will be supplemented as appropriate, with receiver operator characteristic (ROC) curves analysis and predictive values (positive and negative) of various community triage tools (Adult and Paediatric CATs, PMEWS) to predict outcomes (hospital admission and or death) in people of all ages presenting with pandemic influenza like illness to a large number of general practices in the UK; and to feed this information back to policy makers via the Pandemic Influenza Clinical and Operational Advisory Group’s Clinical Sub-group (PICO) and medical members of the Scientific Advisory Group for Emergencies (SAGE).

Secondary objectives

1. To provide reassurance that it is safe for people who do meet triage criteria and so would not be referred for hospital admission to continue to be managed in the community with advice.

2. To describe the associations between specific triage criteria and outcomes (hospital admission, length of stay, need for higher levels of care and or death).

3. Using sensitivity analysis to refine threshold values for specific triage criteria and outcomes (hospital admission, length of stay, need for higher levels of care and or death).

4. To describe demographics and clinical features of patients that meet threshold criteria and are referred for admission by GPs but are declined admission on assessment at hospital.

5. To describe demographics and clinical features of patients that present with influenza like illness to GPs and after assessment are not considered to need referral to hospital, yet die in the community within 30 days.

6. To describe demographics and clinical features of patients that present with influenza like illness to GPs and after assessment are not considered to need referral to hospital, yet die in hospital within 30 days.

We will analyse:

1. separately by age group: adults and children (< 16 years)

2. cumulatively for primary objectives.

3. weekly for primary objectives from initiation of study.

4. three monthly for secondary objectives.

Policy related

To provide evidence and reassurance to clinicians, policy makers and Ministers that any triage tools recommend for use in the event of surge are validated against the novel pathogen, are safe to use and clinically justified.

Study design

Prospective observational maximum capture study using anonymised electronic general practice data linked to extended hospital episode summary data.

It is important to note that there will be no specific interventions made by the study.

None of the clinical features that constitute triage criteria are novel.

All clinical feature of interest are examined albeit with variations of adherence in current community practise.

Recruitment population

Any person presenting to a GPRD participating practice for consultation with a General Practitioner or Primary Care Nurse Practitioner regardless of date of onset of disease and prior medical history.

GPs participating in GPRD are committed to improving the health of their communities by providing anonymised data for a number of research and quality of service audit purposes.

Data collection templates with reminder flags are regularly uploaded to LEPIS and linked to Vision^® in-practice software for such purposes.

Inclusion criteria

Any person presenting to a GPRD participating practice with influenza like illness regardless of date of onset of disease and prior medical history.

Exclusion criteria

People who present with any condition other than influenza like illness.

Proposed sample size

A formal sample size calculation is not possible and will depend to a great extent upon the clinical attack rate and hospitalisation rate (severity) of the pandemic strain. The clinical attack rate and hospitalisation rate of a future pandemic strain cannot be guessed, however the recent ‘mild’ pandemic does provide useful information.

In the first wave of the 2009 pandemic the clinical attack rate varied considerably between regions from 446 cases per 100,000 in the rural South-West to 1344 cases per 100,000 in London.[31] Most people experienced a mild, typical influenza like illness and the overall rate of hospitalisation ranged from 1.3% to 2.5% of those affected.

In the event of a pandemic and activation of phase 3 of the study, the study will recruit from all 629 practices participating in the GPRD currently actively caring for approximately 5.14 million people. Based on the limits of the clinical attack rates during the 2009 event, between 23,000 and 69,000 of these people would be expected to present in the first wave. Approximately 10,000 to 30,000 of these patients will present in the first 3 months before peak activity; forming the study cohort available for assessing, refining and validating triage tools in preparation for surge. Based on the limits of the hospitalisation rates previous described, 130 to 750 of these people would be admitted to hospital forming the group with a positive primary outcome measure in this study. In comparison, the FLU-CIN study was able to validate triage tools using similar method of analyses in a cohort of 1520 patients admitted to hospital where the positive primary outcomes groups comprised 250 (16.5%) people who required higher levels of care (level 2 HDU or 3 ITU) and 80 (5.3%) who died. Thus we are confident that this study will be sufficiently powered to assess, refine and validate the triage tools in the timetable described.

Data Sources: The General Practise Research Database (GPRD) and linked Hospital Episode data

The GPRD research group are named co-investigators in this application.

The General Practice Research Database (GPRD) is the world’s largest database of anonymised longitudinal medical records from primary care. The Secretary of State for Heath has owned the database since 1994. The GPRD is managed by Medicines and Healthcare products Regulatory Agency (MHRA). The database comprises near real-time comprehensive observational data from over 1000 GPs working in 629 general practices. It is a valuable tool for academic research in a broad range of areas including clinical epidemiology, disease patterns, disease management, outcomes research, and drug utilisation. The quality and reputation of the GPRD data make it an invaluable resource for researchers. There have been over 890 research papers published in peer-reviewed journals using data from this database. In July 2011 data was being collected on 5.14 million active patients of research standard or 8.3% of the UK population attending. The GPRD closely reflects the age, gender, and geographic distribution of the UK.[32] The quality of GPRD data has recently been reported in a systematic review. [33] While some acute musculoskeletal and metabolic conditions are not well recorded in GPRD, the majority of diagnoses were reliably coded and there is good agreement between more common diseases and other datasets. Importantly fro this study, influenza incidence rates derived from GPRD agree closely with data from national influenza surveillance systems.[30]

Detailed Hospital Episode Summary (HES) data is linked to GPRD every three months. HES data includes: detailed disease codes (ICD-10), procedure codes, levels of augmented care, length of stay and death. The ability of GPRD to conduct research across linked databases using this detailed clinical record data has been proven in a population-based study on incidence, risk factors, clinical complications and drug utilisation associated with seasonal influenza.[34]

GP clinical assessments and immediate management decisions are recorded on GPRD/LEPIS at time of consultation. This data is downloaded nightly and collated for access by researchers on a weekly basis. Data on hospital admission and death is typically entered on GPRD with in 48 hours of receipt of notification. Recently Jick et al. (2011) described the ability of GPRD to provide near real-time analysis of the epidemiology of pandemic influenza 2009.[35]

GPRD research group has gained ethical and National Information Governance Board (NIGB) approval to provide anonymised data and other healthcare data linked via the patient’s NHS number, sex, date of birth and post code.

GPs who contribute to the database use Vision^® (In Practice Systems Ltd); a computer software package designed by an established user-group of experienced GPs, to make contemporaneous records of their routine consultations and enter study specific data in LEPIS; a linked application maintained by the GPRD research team.

General Practitioners participating in the GPRD will be prompted on screen to record their assessment and management of patients presenting with influenza like illness. A flag will appear during consultation requesting the GP to complete their assessment on the LEPIS study website. This is the limit of any variation from routine practise; which is to record assessment and key positive and negative rule-out features in in free text fields in Vision^®. It is important to note that GP are not expected to diverge from their routine clinical assessment of patients and that GP management decisions are already routinely entered in Vision^®.

Hospitals are required to transmit discharge summaries to GPs within 48 hours or suffer financial penalties. These contain high-level data: primary diagnosis, date of admission, date of discharge or death. This data is manually input to Vision^®, usually within 48 hours of receipt.

Linked anonymised Vision^® and LEPIS data is downloaded nightly in a silent background application to the GPRD research group. Weekly extracts of the data are provided to the wider research community as the GPRD.

Linkage to Hospital Episode Summary (HES) data is done by an external NHS group in a way that the GPRD research team does not see any identifying details. HES data includes: method of admission, disease coding (ICD-10), length of stay, level of augmented care, procedural coding (OPCS classification of interventions and procedures, v4.6), cause of death as certified, prescribing and pharmaco-economics. Levels of augmented care are defined by the Department of Health and Intensive Care Society Standard (2009). These high-level data in HES have been mandated for return from all NHS hospitals in England for all patients admitted to since April 2006 to support payment by results. HES data is linked to GPRD at three monthly intervals and are provided to the wider research community.

The Office for National Statistics (ONS) collects information on causes of deaths from death certificates. ONS data is linked to HES monthly and to GPRD every three months.

Notification of death is made by hospitals to GPs via standard operating procedures that ensure this data is handled as a priority to prevent inappropriate communications and facilitate appropriate support to bereaved relatives. Thus death as a status is entered on Vision^® with little delay. Notice of ‘hospital admission’ is normally available within 48 hours of admission. The primary outcome measures of hospital admission and death are therefore available for use by GPRD following a nightly download. It is recognised that there is typically a delay of 48 hours after discharge before high-level hospital admission data is available from faxed hospital discharge letters. At the time of writing this proposal there is a 3-month interval in linkage of GPRD to HES and ONS certified cause of death data.

Part of our work in phases 1 and 2 will be study feasibility, conducting necessary validation and if necessary, initiate improved systems of data collection in the case of a pandemic.

Data management

The GPRD research group will undertake data collation and secure data warehousing in line with NIGB requirements. The GPRD research group will provide direct access to the GPRD warehouse for the statistician and epidemiologist. The researchers will have no access to patient identifiable data. All data analysis and will be maintained on secured computer servers within the University of Nottingham and University of Liverpool, only accessible to the research team and appropriately protected in accordance with Caldicott principles.

Definition of influenza like illness

Influenza like illness (ILI) will be defined per the World Health Organization criteria: ‘any person with sudden onset of fever (> 38 °C) and cough or sore throat in the absence of other diagnoses’. This closely matches the Centres for Disease Control (CDC) USA definition used for community incidence studies: ‘fever or recent history of fever (≥ 37.8 °C) plus cough and or sore throat in the absence of a known cause other than influenza’. This definition will include those people meeting the British Infection Society/British Thoracic Society/Health Protection Agency/Department of Health (England) guidance criteria (2007): ‘presence or recent history of fever (T > 38 °C) plus new cough in the context of influenza circulating in the community’. We have chosen not to adopt the Royal College of General Practitioners’ Syndromic Surveillance System algorithm as it depends on self-reporting of muscle ache and or headache, which are poorly reported by young children.

It is accepted that the definition of ILI may be revised in the event of a future pandemic, either on the basis of on-going surveillance information or in the event of emergence of a novel pneumonic pathogen. This is not a concern. This study’s aim is to establish automated processes capable of assessing and validating community triage tools in near-real time, which can be applied regardless of any variations of the case-defined entry criteria.

Predictor variables

Clinical features of patients presenting with influenza like illness as recorded by GPs on LEPIS during their routine consultation which include the criteria used by various triage tools:

1. Respiratory Rate (value)

2. Oxygen saturation (value)

3. Any sign of Severe Respiratory Distress (yes/no). Prompt ‘any of: lower chest wall indrawing, sternal recession, grunting, noisy breathing when calm, use of accessory muscles, supra-clavicular recession, tracheal tug, unable to complete sentences in one breath or feeling of suffocation’.

4. Respiratory Exhaustion (yes/no) or Apnoea reported (yes/no), apnoea defined as a ≥ 20-second pause in breathing

5. Blood pressure (systolic and diastolic values)

6. Sternal Capillary Refill Time (normal/> 2 secs)

7. Severe dehydration (yes/no). Prompt: ‘any of: reduced skin turgor, sunken eyes or fontanelle’.

8. Adults only: New altered conscious level (yes/no). Prompt ‘any of: new confusion or disorientation in person, place or time; AVCPU score < A; or ‘Mini (abbreviated) mental test score < 8’.

9. Children only: New altered conscious level (yes/no). Prompt ‘any of: new confusion or disorientation in person, place or time; AVPU score < A; or Mini mental test score < 8/10’.

10. Strikingly agitated (yes/no)

11. New seizures (yes/no)

12. Floppy infant (yes/no)

13. Social isolation (yes/no). Prompt: ‘any of: lives alone; no fixed abode’.

14. Chronic disease (yes/no). Prompt: ‘any of respiratory, cardiac, renal, metabolic, immune suppressed or sickle disease’.

15. Causing other clinical concern to their GP (yes/no). Prompt with ‘For example a rapidly progressive or an unusually prolonged illness’ (yes/no).

Outcome measures

Detailed cost breakdown for research group data acquisition, processing and reporting

Notes