PRImary Care Streptococcal Management Study (PRISM): in vitro study, diagnostic cohorts, and a pragmatic adaptive randomised controlled trial with nested qualitative study and cost-effectiveness study

Background

Using accurate and easy-to-use rapid antigen detection tests (RADTs) to identify group A beta-haemolytic streptococcus (GABHS) sore throat infections could reduce unnecessary prescription of antibiotics. Although widely used in Finland, France and the United States, uncertainty regarding RADTs’ sensitivity and ability to impact on prescribing decisions has resulted in limited uptake in the UK.

Objectives

We aimed to evaluate current RADTs available in the UK in controlled, parallel in vitro trials, in order to eliminate the spectrum bias and variability in sampling associated with many clinically based studies.

Method

We compared the ease of use of five UK RADTs and their ability to detect different concentrations and strains of GABHS: OSOM^® Ultra Strep A (Bio-Stat Limited, Stockport, UK), QuickVue^® Dipstick Strep A test (TK Diagnostic, Oxford, UK), Streptatest^® (DECTRA PHARM, Strasbourg, France), Clearview^® Exact (Inverness Medical Professional Diagnostics, Bedford, UK) and IMI TestPack^® Plus Strep A (Inverness Medical, Bedford, UK). We also measured whether the RADTs falsely identified common throat commensals as GABHS. All kits were tested with single-tipped polyester swabs, rayon swabs and the kit swabs provided by the manufacturer.

Results

The IMI TestPack was the easiest RADT to use. The ability to detect all positive GABHS (the sensitivity of the RADTs) varied considerably between kits from 62% [95% confidence interval (CI) 51% to 72%] for Clearview to 95% (95% CI 88% to 98%) for the OSOM and IMI TestPack at the highest GABHS concentration. None of the RADTs gave any false-positive results with commensal flora – they were 100% specific. For most of the kits, the supplied swab performed well with the exception of the Clearview pack, which performed much better with a polyester swab.

Conclusion

The IMI TestPack is suitable for use in primary care, as it had high sensitivity, specificity and was the easiest to use. If Clearview is used, a polyester swab rather than the manufacturer’s swab is preferable.

Rapid antigen detection kits

The kits tested represented the most commonly used kits in Europe, the most widely available kits in the UK and those that performed reasonably in a previous study:4 Clearview Exact Test, IMI TestPack Plus Strep A, OSOM Ultra Strep A, QuickVue Dipstick Strep A test and Streptatest (Figure 1).

FIGURE 1.

The five RADTs evaluated. Dipsticks, left to right: OSOM, QuickVue Dipstick and Streptatest. Cassettes, top to bottom: IMI TestPack Plus and Clearview. Showing positive reactions for all.

Sample size for kit sensitivity testing

Previous studies have shown that some RADTs are able to detect between 80% and 90% of GABHS infections. 5 Assuming that the best RADT in this study would achieve a sensitivity of 85–95%, and to estimate with 95% confidence that the sensitivity of a RADT was within ± 5% (i.e. to be confident that the sensitivity was not < 80%, which would be less useful clinically), we estimated that 1460–3920 samples were required for all five RADTs.

Sensitivity testing: ability to detect group A beta-haemolytic streptococcus

Four Streptococcus pyogenes strains were used that are associated with clinical sore throat infections from the National Collection of Type Cultures (NCTC) (Health Protection Agency, Colindale, UK) (Table 1). Streptococcal dilutions ranging from 10² up to 10⁸ colony-forming units (CFU)/ml were tested against RADTs in duplicate (for detailed culture methods, see Appendix 1). To mimic conditions used to undertake RADTs in a GP surgery, each streptococcal dilution was administered as a 100-μl aliquot on to a swab and these swabs were then used to perform the RADTs (as per manufacturers’ instructions).

These trials showed that all RADTs detected GABHS at 10⁷ CFU/ml, whereas some kits failed to detect streptococci at concentrations < 10⁶ CFU/ml. These tests were duplicated to ensure accuracy. Subsequently, each S. pyogenes stock solution was adjusted to achieve four dilutions of GABHS within this range: 2.5 × 10⁶, 5 × 10⁶, 7.5 × 10⁶ and 10 × 10⁶ CFU/ml. Each RADT was tested 20 times using the swab technique outlined above at each dilution, with four GABHS strains (320 tests per RADT). The final result for each RADT investigation was interpreted in conjunction with the manufacturer’s instructions: positive, slightly positive, negative or invalid (see Figure 1).

Specificity testing: assessment of false-positives due to commensal flora

A panel of 23 commensal control strains normally found in the throat were obtained from the NCTC, the American Type Culture Collection and the National Collection of Pathogenic Fungi (see Table 1). Strains were evaluated in seven test solutions, which included several organisms from similar or identical genera. The concentrations of these organisms were adjusted to represent the upper limits often found in clinical samples, with each organism equal to 10⁷ CFU/ml. 6–9

Each RADT was tested 10 times with seven commensal groups (70 tests per RADT).

Ease of use of rapid antigen detection tests

A biomedical scientist and consultant microbiologist evaluated 10 characteristics of each RADT (Table 2) – these included packaging, shelf-life, test procedure, controls, interpretation and timing of results. RADT instruction inserts were assessed on six main features: layout, font size, general clarity, stand-alone clarity of visual instructions, clarity of instructions for determining final results and inclusion of all pertinent information (see Table 2 for scoring). Each test kit could score a maximum total of 22 points. Price per test (2008) was calculated excluding company discounts for bulk purchases.

Swab type

The manufacturers’ instructions for the RADTs evaluated specifically recommended the use of either polyester or rayon swabs (see Appendix 2). Consequently, all kits were tested in vitro with single-tipped polyester swabs (Ref: 170C, Copan Diagnostic, Barloworld Scientific, Staffordshire, UK), rayon swabs (Ref: 141C, Copan Diagnostic) and the kit swabs provided by the manufacturer.

Sensitivity results: ability to detect group A beta-haemolytic streptococcus

All the test kits performed better with increasing concentration of GABHS (Figure 2). All RADTs were positive at the highest concentration of GABHS of 10 × 10⁶ CFU/ml, whereas at lower concentrations of GABHS the kits varied in their ability to give positive results. At a GABHS concentration of 10 × 10⁶ CFU/ml, the OSOM and IMI TestPack detected 95% of the test samples as positive, whereas Streptatest detected 79% (95% CI 67% to 85%), QuickVue 70% (95% CI 59% to 80%) and Clearview 62% (95% CI 51% to 72%). The Streptatest, QuickVue and Clearview were all at least 15% less sensitive than OSOM and IMI TestPack at all concentrations of GABHS. Detailed results are available in Appendix 3.

FIGURE 2.

Sensitivity (with 95% CI) for five RADTs, in relation to swab type used; all GABHS strains and concentrations.

GABHS strain type caused only minor fluctuations in RADT sensitivity; strain 8312 was associated with an overall sensitivity of 41% (95% CI 38% to 44%), strain 8308 a sensitivity of 48% (95% CI 45% to 51%), strain 12,969 a sensitivity of 49% (95% CI 46% to 52%) and strain 10,867 a sensitivity of 51% (95% CI 48% to 54%).

Specificity

None of the RADTs gave any false-positives when tested 10 times with each commensal group. Thus, the specificity of each RADT was 100% (95% CI 72% to 100%).

Swab type

Combining the results from all RADTs demonstrated the impact of swab type: kit swabs were associated with the highest sensitivity results overall at 55% (95% CI 53% to 57%), compared with polyester at 52% (95% CI 50% to 54%) or rayon at 37% (95% CI 35% to 39%).

Figure 2 presents the compiled sensitivity results for all GABHS strain types and GABHS concentrations, and demonstrates the most important two-way interaction between RADTs and swab type. The kit swabs provided produced the highest sensitivity results with the OSOM, QuickVue Dipstick, IMI TestPack Plus and Streptatest kits. However, Clearview sensitivity was notably better with polyester swabs at 77% (95% CI 67% to 87%); in comparison, the kits swabs provided had a sensitivity of only 38% (95% CI 28% to 48%) and rayon swabs of 23% (95% CI 13% to 33%).

Figure 3 shows the effect of swab type on RADTs when the influences of strain type and GABHS concentration are removed. Overall, RADT sensitivity was best when using the kit swabs provided at 54%. This is in comparison with polyester swabs at 52% and rayon swabs at 37%.

FIGURE 3.

Sensitivity (with 95% CI) for each swab type; all RADTs, GABHS strains and GABHS concentrations.

Figure 4 demonstrates RADT sensitivities at all GABHS concentrations, minimising the effect of the least important variable – GABHS strain. At 7.5 × 10⁶ CFU/ml the OSOM/kit swab combination achieved a sensitivity of 95% (95% CI 87% to 100%), which was matched only by Clearview using polyester swabs. However, Clearview and polyester swabs achieved 100% (95% CI 96% to 100%) sensitivity at 10 × 10⁶ CFU/ml, whereas the sensitivity of OSOM and kit swabs remained at 95% at this concentration. However, it is worth noting that the sensitivity of Clearview when using the kit swabs provided was considerably reduced, reaching a maximum sensitivity of only 62% (95% CI 51% to 72%) at 10 × 10⁶ CFU/ml, which is lower than any other RADT.

FIGURE 4.

Using ‘best’ swab type, RADT sensitivity at each GABHS concentration; combined results for all GABHS strains.

Ease of use/storage, cost and instruction clarity (see Tables 2 and 3): suitability of rapid antigen detection tests for general practice

As the study evaluated a selection of cassette and dipstick formats, all the kits had different characteristics, with each RADT having a variety of strengths and weaknesses.

The OSOM Ultra kit had a shorter shelf-life than all the others and contained the most tests per kit (n = 50). This may be important if surgeries are undertaking less than one test per week, as the tests may not be used before the expiry date.

The Clearview and IMI TestPack kits were provided in larger boxes than those of the other RADTs. This could present a storage problem for GP surgeries, particularly those practices with minimal storage space or those storing numerous RADTs kits.

The IMI TestPack needed an additional sample extraction step, not required for the other RADTs, which was simply the addition of another extraction reagent. However, this additional step was straightforward, adding only seconds on to the total test procedure.

The IMI TestPack had an end point that could be read at between 5 and 10 minutes, with the availability of the final results confirmed by a novel ‘end-of-assay window’ (see Figure 1). The other four kits had timed end points at 5 minutes, and two of the four kit instructions stated it was essential to read the results at 5 minutes. The IMI TestPack was deemed the easiest format for routine clinical practice because of the flexibility in reading the end point.

Overall, the cassette formats were preferred to the dipsticks, as they were easier to handle and discard. All test reagents are absorbed into the cassette test device, thus eliminating the hazard of test reagent spillages, a clear advantage for any point-of-care testing kit. Overall, the IMI TestPack achieved the highest score for general ease of use, totalling 18 out of 22 points (see Table 2). Streptatest and Clearview both scored 15 points, QuickVue Dipstick 14 points and OSOM 13 points. Overall, the two cassette formats achieved higher scores than the dipsticks.

Clarity of kit instructions and price per test

The instruction clarity of the RADTs is outlined in Table 3, along with price per test. Notably there was no standard format used by the RADT manufacturers and, consequently, the quality of the kit instructions varied between RADTs. Streptatest was felt to have the best instructions overall: clear and simple, with a logical layout. The IMI TestPack was a close second, losing marks on small font size and the inclusion of too much information.

Overall, OSOM was the most expensive and judged to have the poorest instructions. Streptatest was the cheapest kit, with the best instruction clarity. The IMI TestPack performed well overall, marred only by price per test. Clearview was the second cheapest RADT, and had an average instruction manual. The difference in price per test was more than threefold: Streptatest was the cheapest at £1.38 and OSOM the most expensive at £3.46.

Strengths and limitations

Clinical throat swabs are notoriously imprecise samples: two simultaneous swabs often vary in the number of GABHS collected. 10,11 With no standard method for throat swab collection, swab material, transport medium and environmental factors can all affect GABHS collection and survival, and ultimately RADT performance. Our in vitro study removed these biases by using precise GABHS concentrations and directly comparing the performance of each RADT.

The GABHS strains used for sensitivity testing were chosen because of their association with sore throat episodes and included two mucoid strains. Testing more strains of GABHS may have altered our sensitivity findings; however, this seems unlikely, given that the four very different strains included in this study had little impact on RADT sensitivity.

This study was undertaken by a biomedical scientist with laboratory training, as we wanted to minimise any user bias in our method. Previous research has shown that the professional skills of the person performing the RADT can affect its sensitivity. Consequently, the accuracy of these tests may be affected when performed by untrained personnel in a general practice setting.

A significant proportion of streptococcal sore throats are group C or G streptococci which probably present with similar clinical features to group A streptococci. 12,13 Investigating sore throat infections with highly specific RADTs will result in cases of group C and G infections being undiagnosed and untreated. The significance and clinical relevance of this issue will be investigated further in the diagnostic studies (see Chapters 2 and 3).

Main findings and comparison with existing literature

The OSOM and IMI TestPack were 15% more sensitive (detected 15% more positive samples) than Streptatest, QuickVue and Clearview at all concentrations of GABHS. The OSOM kit detected more strains of GABHS at lower concentrations than the IMI TestPack. However, the OSOM kit was the most expensive kit and had poor instructions. Overall, the simple and practical IMI TestPack outperformed the other kits for ease of use, demonstrating the kit’s suitability for the general practice setting. None of the RADTs gave any false-positive results with the commensal flora [specificity 100% (95% CI 72% to 100%)]. Variation in GABHS strain type had little effect on RADT sensitivity, whereas increasing streptococcal concentrations improved the sensitivities of all kits.

Only one similar in vitro study has been identified, a French-language paper by Charlier-Bret et al. published in 2004. 4 Our findings were in line with this study, confirming that RADT sensitivity typically falls between 10⁵ and 10⁷ CFU/ml. Overall, Charlier-Bret et al. evaluated four of the five kits assessed in our study, reporting the most sensitive RADT as the IMI TestPack, followed in decreasing sensitivity by Streptatest, QuickVue Dipstick and Clearview. The OSOM kit was not evaluated.

The cost of each RADT is likely to be a barrier for its use in the health systems such as UK primary care, in which doctors working in primary care do not pay for laboratory diagnostic tests. If health-care providers wish to encourage the use of RADTs in primary care in similar environments, they are likely to need to reimburse physicians for their use. This may well be a cost-effective policy if RADTs can be shown to reduce antibiotic prescribing more effectively than the alternative strategies.

Swab type

A previous report by Bourbeau14 noted that swab fibre composition, swab tip preparation, swab tip characterisation (fibre, foam, flocked), shaft type (hollow, solid) and transport medium can impact on swab performance. In light of this information, and because the clinical validation part of the PRISM study used rayon swabs for validation of all kits, despite three of the manufacturers recommending polyester swabs (see Appendix 2), users of RADT should note that the ‘best’ swab type is not always that provided with the RADT kit or recommended by the manufacturer.

Background

Management of acute sore throat is often based on features associated with GABHS, but the features that best predict GABHS require clarification. Non-group A streptococcal strains share major similarities with group A strains, but their clinical presentation and incidence has not been clarified.

Objective

The aim of this study was to assess the incidence and clinical features associated with streptococcal infections.

Design

This study comprised a prospective diagnostic cohort.

Setting

The setting was UK primary care.

Patients

The patients included in the study were aged ≥ 5 years and presented with acute sore throat and clinical signs in the pharynx (acute pharyngitis).

Methods

The presence of pathogenic streptococci isolated from throat swabs was documented.

Results

Pathogenic streptococci were found among 204/597 patients (34%; 95% CI 31% to 38%). Of these, 33% (68/204) were non-group A streptococci – mostly C (29), G (18) and B (17); and rarely D (3) and S. pneumoniae (1). Patients presented with similar features whether the streptococci found were group A or non group-A. The features that best predicted A, C or G beta-haemolytic streptococci were the patients’ self-reported assessment of severity (odds ratio for a bad sore throat 3.31; 95% CI 1.24 to 8.83); absence of a bad cough (2.73; 95% CI 1.56 to 4.76), absence of coryza (1.54; 95% CI 0.99 to 2.41); muscle aches rated moderately bad or worse (2.20; 95% CI 1.41 to 3.42); and clinicians’ assessment of severity (severely inflamed tonsils 2.28; 95% CI 1.39 to 3.74).

Conclusion

Non-group A strains commonly cause streptococcal sore throats, and present with similar symptoms and clinical features to group A streptococci. The best features to predict streptococcal sore throat presenting in primary care deserve re-examining.

Inclusion

Health professionals in general practices in the south and central areas of England recruited adults or children aged ≥ 5 years presenting with acute sore throat (< 2 weeks), when the sore throat was the predominant clinical feature (or when the clinician felt that the pharyngitis was driving the illness presentation), and with an abnormality on examination of the throat (erythema with or without pus and anterior cervical glands) – similar to a previous study in primary care. 23 Exclusion criteria were as follows: other non-infective causes of sore throat (e.g. aphthous ulceration, candida, drugs) or unable to consent (e.g. dementia, uncontrolled psychosis).

Clinical data

Following informed consent, baseline clinical data were collected by the recruiting health professional. 24–26 The case report form collected information on age, gender, current smoking status and past history of quinsy27 as well as data on symptom severity for the symptoms of sore throat, difficulty swallowing, fever, cough, coryza (‘runny nose’), headache, muscle ache, abdominal pain, diarrhoea, vomiting, earache (each symptom was rated by the patient as follows: 0 = no problem, 1 = slight problem, 2 = moderately bad problem, 3 = severe problem). The doctor or nurse documented examination findings for oral temperature using Tempa•DOT™ thermometers (3M, St Paul, MN, USA)28 the severity of tonsillar and pharyngeal inflammation, and the presence of cervical glands, tonsillar exudate, fetor and palatal oedema. 24–26

Throat swabs

At the training session in each practice, clinicians were instructed in standard study procedures, including how to take a throat swab. Swabs were taken by the clinician and sent to a central laboratory for culture of and sensitivity testing to all significant pathogens, in line with national standard operating procedures. 29,30 Mean time between specimen collection and receipt at laboratory was 2.9 days (data incomplete for 13 samples). The swabs were inoculated on to a blood agar plate and staph/strep agar plate (E&O Laboratories Ltd, Bonnybridge, Scotland) and spread for single colonies. Plates were incubated anaerobically for 48 hours. 29,30 Plates were read after 24 hours of incubation and negative cultures reincubated for an additional 24 hours. Suspected beta-haemolytic streptococcal isolates were identified via visual analysis of colony morphology and Lancefield grouping (PathoDx Strep Grouping Kit, Oxoid), in accordance with the national standard operating procedures. 29,30 Antibiotic sensitivities were conducted using disc diffusion techniques. 31

Sample size

In order to determine the predictive value of clinical variables, we estimated that a subgroup of 139 patients with a clinical presentation not associated with streptococcal infection would provide estimates of a negative predictive value of 90% with 95% CIs of ± 5%, and a subgroup of 93 individuals with streptococcal infection would provide estimates of a positive predictive value of 60% with 95% CIs of ± 10%. We estimated that a sample size of 455 patients would be sufficient to detect an odds ratio of 2 (assuming α = 0.05 and β = 0.2) for variables with a prevalence of 20–65% among patients without streptococcal infection.

Analysis

Clinical variables were included in a logistic regression model to assess their association with the presence of streptococci. Forward selection was used: variables were included if significant at the 10% level and retained in multivariate analysis if they remained significant at the 5% level, with no evidence of collinearity. All variables significant in univariate analysis were checked again in the final model. Cases with missing data for a particular analysis were excluded. For variables with several levels (e.g. sore throat), to facilitate use in a simple clinical score (i.e. ease of implementation), a cut-off was normally made at or near an odds ratio of 2. Continuous variables were dichotomised using previous cut-offs (age ≤ 10 years; prior duration longer than the median of 3 days). 25 For duration, there was a progressive reduced likelihood of infection with group A streptococci. With longer prior duration, however, we dichotomised at the median for ease of implementation. We also present a version of the final model with more categories for each variables (i.e. not dichotomised, using ordered categorical variables). Such a model could potentially be used with computerised practices to document more precisely risk of streptococcal infection (Appendix 4). Although for non-group A streptococci to date we assumed Lancefield groups B and D and also Pneumococcus were not to be counted as significant pharyngeal pathogens (Tables 2 and 3 and Appendix 4), given the ongoing debates about this issue,13 we also present the multivariate analysis when these streptococci are included as potentially significant pathogens (see legends to Tables 2 and 3). Given the higher asymptomatic carriage rates of streptococci in children, we did not include age in the final multivariate models.

Recruitment bias

Comparing patients of higher-recruiting doctors (higher than the median – average 11.8 patients per month) with patients of lower-recruiting doctors (an average of 2.6 patients per month), there was no difference in the number of features that predicted streptococcal infections in multivariate analysis (see Table 6 for significant features, respectively a mean of 3.3 features and 3.4 features), suggesting little or no recruitment bias based on clinical characteristics.

Strengths and limitations of the study

The power to detect variables associated with groups C and G streptococci was limited; however, this is one of the largest studies to assess a broad range of clinical variables, and we found similar features to a smaller study reported previously. 11 Missing data were minimal (< 5% for any analysis), and, although consecutive recruitment of cases was difficult to enforce in practice, very little evidence of recruitment bias was found when recruitment rates were compared with expected rates of recruitment from national samples. Selection bias is a potentially important issue among low-recruiting doctors, but we found no evidence of clinical differences between patients of higher- and lower-recruiting doctors. Overall, fewer children than expected from historical data sets2 were recruited, which probably reflects the reluctance of parents and/or GPs to expose children to a throat swab, but, as we elected not to include age per se in the model, the impact of this should be slight. The time between taking a swab and receipt at the laboratory was slightly longer than expected, but Lancefield groups A, C and G are not particularly sensitive to transport conditions, and we found relatively high percentages of streptococci compared with previous literature. As there was no indication from sentinel practices or microbiology laboratories of a streptococcal epidemic, the higher streptococcal percentages and more florid clinical signs than previous studies in a similar geographical area33 may indicate changing consultation thresholds. Although the study indicates that clinical features not traditionally incorporated in making diagnostic assessments may possibly not be important, further data sets are needed before recommending a key variable set. The way variables are operationalised may also be important: McIsaac et al. 34,35 use tonsillar swelling or exudate, Centor et al. 24 just exudate and we chose exudate, as swelling is not necessarily an acute feature. We used intermediate cut points when indicated rather than the extremes of each scale; however, as the judgement of intermediate points may be more variable, using the extremes (none or very) may be more reliable for developing a clinical prediction rule.

Main findings and comparison with existing literature

Traditionally clinicians have been predominantly interested in GABHS because of their association with major non-suppurative adverse outcomes – particularly rheumatic fever. 1 Therefore, the clinical predictors of GABHS infection1,25,36 – especially pus, cervical nodes, a history of fever and no history of cough37 – have been widely used in clinical guidelines. 16,17,38 Historical comparisons tentatively suggest that these variables may identify a group of patients who are more likely to benefit from antibiotics. 15 We confirmed the importance of cervical glands and the absence of a bad cough and of fever. 25,34 However, this study documents that, although the feature of purulence is associated with the presence of GABHS in univariate analysis, in this data set it is not independently predictive – and other features may be important, particularly the severity of both the sore throat and the inflammation, the prior duration (reflecting a more rapid, severe onset), muscle aches and possibly the absence of coryza. Some of these features were identified previously in studies in typical primary-care settings,12,25 but previous studies were limited by a lack of multivariate analysis or limited power.

The clinical presentation of infection with group C and G streptococci suggests strongly that not only are these presentations unlikely to be due to commensal carriage, but they are causing a similar clinical syndrome to GABHS. This supports those studies that observed similar symptomatic presentation. 12,13 If group C and G streptococci are clinically important, then rapid streptococcal antigen tests (which are targeted at GABHS only) will miss a significant proportion of streptococcal infection.

Background

Clinical variables traditionally associated with pathogenic streptococci – such as the presence of cervical glands – and clinical scores derived from these variables are commonly used to minimise antibiotic use for individuals with a low likelihood of pathogenic streptococci.

Objective

The aim of this study was to assess the association between features of pharyngitis and the growth of Lancefield group A, C or G streptococci from culture of a throat swab in two cohorts.

Design

This was a diagnostic cohort study.

Setting

The setting was general practices in the UK.

Patients

Patients included in the study were aged ≥ 5 years and presented with acute pharyngitis. Patients were recruited for a second cohort (cohort 2, n = 517) consecutively after the first (cohort 1, n = 606) from similar practices.

Results

A, C or G beta-haemolytic streptococci were found in 40% of participants in cohort 2 (compared with 34% in cohort 1). The clinical features that predicted the presence of these streptococci in multivariate analysis in both cohorts were as follows: rapid attendance (short prior duration of ≤ 3 days; multivariate-adjusted odds ratio 1.92 cohort 1, 1.67 cohort 2); fever in the last 24 hours (1.69, 2.40); and severe inflammation as assessed by doctor [severely inflamed pharynx/tonsils (2.28, 2.29)]. Absence of coryza or cough and purulent tonsils were also significant predictive variables in univariate analysis in both cohorts and in multivariate analysis in at least one cohort. A five-item score, based on Fever, Purulence, Attend rapidly (≤ 3 days), severe Inflammation and No cough or coryza (acronym FeverPAIN), had moderate predictive value [bootstrapped estimates of area under receiver-operating-characteristic (ROC) curve: 0.73 cohort 1, 0.71 cohort 2] and performed well in identifying a substantial number of participants at low risk of streptococcal infection (38% in cohort 1, 36% in cohort 2 scored ≤ 1, associated with streptococcal percentages of 13% and 18%, respectively). A Centor score of ≤ 1 identified 23% and 26% of participants, with streptococcal percentages of 10% and 28%, respectively.

Conclusions

Items widely used to help identify presentations of streptococcal sore throat in primary care may not be the most valid. A modified clinical scoring system (FeverPAIN), which requires further validation, may be helpful clinically in identifying individuals who are unlikely to have major pathogenic streptococci.

Sample size

In order to determine the association of clinical variables with streptococcal infection, assuming that at least one-third of individuals would have streptococci infection (based on our first data set), and that variables in the streptococcal group were found in 30–80% of individuals, the detection of a variable with an odds ratio of 2 required 407 individuals with complete results.

Analysis

As we found previously from the first data set that patients with group C and G beta-haemolytic strains presented with similar clinical features to individuals with GABHS, we assessed the independent clinical features associated with combined group A, C and G streptococci in both data sets. Clinical variables were included in a logistic regression model to assess their association with the presence of Lancefield group A, C and G streptococci. Forward selection was used: variables were included if significant at the 10% level and retained in multivariate analysis if they remained significant at the 5% level. Missing variables were not imputed. Continuous variables were dichotomised using previous cut-offs (age ≤ 10 years; prior duration longer than the median of 3 days). 25 Given the higher asymptomatic carriage rates of streptococci in children, we did not include age in the final multivariate models. Clinical scoring systems or clinical prediction rules are most likely to be useful if they are simple to remember and use, which suggests few variables should be used – preferably using a simple count of the predictive variables. We estimated the increase in area under the ROC curve starting with the most predictive variables, with the aim of maximising the area under the curve without including unnecessary variables, and generated a basic model using variables that were significant in multivariate analysis in both data sets. However, a clinical score using very few variables will potentially limit the grading of risk (as there will be fewer categories) and variable performance of one item in different cohorts will unduly affect reliability. Therefore, we also generated an expanded score to include variables that were significant in univariate analysis in both data sets and multivariate analysis in at least one of the data sets.

Because any new model developed from a single data set may be overfitted, bootstrapped estimates are provided for the area under the ROC curve for internal validation for the new model (see Table 8). 43 For the Centor criteria (an established model), non-bootstrapped estimates are provided.

Strengths and limitations of the study

These data sets are some of the largest from a typical primary-care setting to have assessed the importance of the range of streptococci, and to explore the range of potential clinical predictors of streptococcal infection. There were few missing data (< 5% for any analysis), and little evidence of recruitment bias, either in recruitment rates or in clinical characteristics. The conventional approach to develop and validate a diagnostic model is to develop it in one data set and test it in another. However, the variability of the performance of variables in these data sets – particularly striking for some variables such as cervical glands – suggests that such an approach is unlikely to provide the most valid method of variable selection for a clinical prediction rule, which is supported by similar findings in the development of clinical prediction rules for other acute infections. 44 The reason for this variation is unclear, but it may reflect varying infective agents, populations or clinician factors (e.g. reliability of history taking or examination). The variability suggests that the choice of variables to include in clinical prediction models should be based either on very large single cohorts or on multiple cohorts at different times and/or different settings. Over and above the most basic model (short prior duration, severe inflammation, fever) the choice of additional variables to include (pus and ‘absence of cough and coryza’) was determined by consensus, including a consideration of the strength of prior evidence. Although we have provided bootstrapped estimates of the area under the ROC curve to limit overfitting, nevertheless the proposed model should have further validation. 12

Main findings in the context of previous literature

GABHS have dominated previous literature because of their association with major non-suppurative adverse outcomes – particularly rheumatic fever and glomerulonephritis. 1 Therefore, the clinical predictors of GABHS1,25,36 – especially pus, cervical nodes, a history of fever and no history of cough – have been widely used in clinical guidelines. 5,16,17 Trials using these as inclusion criteria may have larger effect sizes for antibiotics than trials using less selected patients – although the validity of historical comparisons is questionable. 15 We were unable to confirm the importance of cervical glands as a predictor of streptococcal infection in the second data set, and in the first data set we were unable to confirm the importance of purulence. 25,34 From these two data sets the features that may be most important are the speed of presentation (i.e. symptoms developing rapidly resulting in short prior duration of illness), the severity of inflammation and fever. These variables have been identified in studies from typical primary-care settings,12,25 but previous studies have been limited by a lack of multivariate analysis or limited power.

Clinical utility

Scoring systems are most helpful clinically for reducing antibiotic use if they identify as large a group as possible of individuals unlikely to have streptococcus. From these data sets, the Centor criteria are likely to identify relatively few such individuals who do not have streptococci: only 23% in the first data set and 26% in the second data set had a score ≤ 1, and, of these, in the second data set the percentage of patients with streptococci was high (28%). A low count (≤ 1) using a modified score (Fever SPIN) identified > 35% of patients in both data sets as unlikely to have streptococci (between 13% and 18%).

Objective

The aim of the study was to compare clinical scores and RADTs with delayed antibiotic prescribing.

Design

This was an open, adaptive, pragmatic, parallel-group randomised controlled trial.

Setting

The setting was UK primary care.

Patients

Patients included in the study were aged ≥ 3 years and had acute sore throat.

Intervention

An internet program randomised patients to targeted antibiotic use according to (1) delayed antibiotics (control group), (2) clinical score or (3) RADT used according to clinical score.

Outcomes

The outcomes were as follows: self-reported antibiotic use, symptom duration and severity on seven-point Likert scales (primary outcome was mean sore throat and difficulty swallowing score for the 2–4 days following the consultation primary outcome).

Results

A preliminary score to predict streptococcal infection (score 1; n = 1129) was replaced by a more valid score [score 2; n = 631; features: Fever during previous 24 hours, Purulence, Attend rapidly (≤ 3 days), very Inflamed tonsils, No cough/coryza (FeverPAIN)]. For score 1, there were no significant differences between groups. For FeverPAIN, symptom severity was documented in 80% of patients [delayed 168/207 (81%); clinical score 168/211 (80%); RADT 166/213 (78%)]. Severity was lower in the clinical score group than in the delayed prescribing group (–0.33; 95% CI –0.64 to –0.02; p = 0.039; equivalent to one in three rating sore throat a slight rather than moderately bad problem), and a similar reduction was observed for the RADT group (–0.30; –0.61 to –0.00; p = 0.053). Moderately bad or worse symptoms resolved significantly faster (30%) in the clinical score group (hazard ratio 1.30; 95% CI 1.03 to 1.63), but not in the RADT group (1.11; 0.88 to 1.40). In the delayed group, 75/164 (46%) used antibiotics, and 29% fewer used antibiotics in the clinical score group (risk ratio 0.71; 0.50 to 0.95; p = 0.018) and 27% fewer for the RADT group (0.73; 0.52 to 0.98; p = 0.033). No significant differences in complications or reconsultations were found.

Conclusion

Targeting antibiotics for acute sore throat using a clinical score improves symptoms and reduces antibiotic use. RADTs used according to a clinical score provide similar benefits, but no clear advantages over a clinical score alone.

Development of clinical scores

The clinical score development has been reported in Chapter 3. In brief, two diagnostic cohorts prior to this trial cohort were used to develop clinical scores to predict streptococcal infection.

Trial recruitment

Patients presenting with acute sore throat were recruited by health professionals in general practices in south and central England (mainly GPs but also triage practice nurses).

Intervention groups

The aim of the trial was to compare methods of targeting antibiotics, by comparing a RADT or a clinical score with the empirical strategy of patient choice in the use of a delayed prescription.

1. Delayed antibiotics (control) A prescription was prepared and left in reception with advice to the patient to collect the prescription after 3–5 days if symptoms were not starting to settle, or sooner if symptoms became significantly worse. 33 The rationale for using delayed prescribing is that it is safe, it should, according to previous data, result in rates of antibiotic use similar to those when there is no initial offer of antibiotics, it changes belief in antibiotics as effectively as not prescribing and it modifies reconsultation more effectively than not prescribing (based on our previous studies both of sore throat46 and lower respiratory tract infections47). It has been incorporated widely into routine practice in the UK since our 1997 trial33 without any increase in complications of sore throat. 48

2. Clinical score Antibiotics were not offered to those with very low scores (0 or 1). For high scores (≥ 4, 63% streptococci), immediate antibiotics were offered, and for intermediate scores (2 or 3, 39% streptococci) delayed antibiotics were offered.

3. RADT group Health economic modelling of the phase 1 results was used to estimate the most efficient use of RADTs. The modelling indicated that RADTs would be best targeted at those with intermediate and high clinical scores, for whom prescription of antibiotics was most likely. Thus, using score 2 (FeverPAIN), those with low clinical scores (0 or 1) were not offered antibiotics or a rapid test (< 20% streptococci from the first phase); those with a clinical score of 2 (33% streptococci) were offered a delayed prescription and those with higher scores (≥ 3, 55% streptococci) had a rapid test on surgery premises, and following the test patients with negative results were not offered antibiotics. All patients – as in the other groups – were advised to use analgesia (regular paracetamol and/or ibuprofen). The original intention was that the choice of RADT (the IMI test pack) would be based on both a clinical study and an in vitro study. However, there was sufficient evidence of validation characteristics from in vitro studies, which provided better evidence of validity by avoiding sampling biases. Ease of use was another important consideration and was assessed during the in vitro study and also confirmed by a small panel of GPs and nurses (see Appendix 7 at the end of the report). Taking the swab and performing the test takes approximately 5–7 minutes to get an answer.

As this was a pragmatic trial, clinicians were asked to use the intended strategy when this could be agreed with the patient, but clinicians were given flexibility to negotiate other strategies, as would happen in practice. Thus, for example, not all those randomised to delayed prescribing were given a delayed prescription.

Data collection

Patients were blind to the precise details of the groups being tested, but the pragmatic interventions, and the key outcomes (reported symptom severity and duration), made full blinding impossible. Data collection by the research team (telephone or notes review) was blind to group as far as possible, but details of patient management were available in the notes. No changes in planned outcome measures were made following trial commencement.

Analysis

The analysis plan was finalised by the trial management group prior to performing the analysis. Analysis of covariance was performed for the main continuous outcomes (diary scores) and Cox regression for the duration of symptoms rated moderately bad or worse. Logistic regression was used for dichotomised outcomes [e.g. belief in the need to see the doctor for future episodes and return to the surgery (controlling for follow-up time)]. Odds ratios were converted to risk ratios using the method of Zhang and Yu. 50 The models controlled for baseline severity (a strong predictor of all outcomes), and for potential confounders if appropriate (in this case, fever during the last 24 hours). The primary analysis was an intention-to-treat analysis based on complete data sets – given the problem of imputing modest differences for rapidly changing symptomatic outcomes when no outcome data are available. Although a per protocol analysis was initially considered, given the pragmatic nature of the study and the leeway given to GPs to negotiate management, it was difficult to operationalise what per protocol might mean, so a per protocol analysis was not performed. The clinical and demographic characteristics of those not followed up were compared to assess, respectively, possible selection and non-response bias. Non-differential selection bias to the trial was assessed by a comparison of the clinical features in the current trial with those in the previous diagnostic study, and also with a parallel observational cohort among a wide range of practices which used the same clinical proforma [the MRC (Medical Research Council) DESCARTE (decision rule for severe symptoms and complications of acute red throat in everyday practice) study, Little et al. , University of Southampton, which recruited nearly 12,000 patients]. No interim analysis was performed, and no subgroups specified in advance. To assess the differences between the two phases of the trial, interaction terms were used in the models, and, as significant interactions were found, the data from both phases were presented separately. The study team agreed in advance that, if there were significant differences between score 1 and score 2, score 2 results would be presented separately as the main results (i.e. using score 2, FeverPAIN), with the detailed results from the first part of the trial being documented in Appendix 6.

Evidence of differential effectiveness of the first and second parts of the trial

Improvement in symptom scores in the first 3 days following the index consultation was significantly greater with the second clinical score than with the first score (interaction term –0.38, 95% CI –0.76 to –0.01; p = 0.043), and less with the rapid test group (interaction term –0.18, –0.55 to 0.19; p = 0.345). There was also a significantly larger effect on symptom resolution when using the second score (interaction term hazard ratio 1.35, 95% CI 1.01 to 1.79; p = 0.043), but little difference when the rapid test was used (interaction term hazard ratio 1.01, 0.76 to 1.35; p = 0.93). Similarly, there was a significantly greater effect on antibiotic use when using the second score (interaction term odds ratio 0.43, 95% CI 0.24 to 0.76; p = 0.004) and a lesser effect for the rapid test group (0.74, 0.42 to 1.32; p = 0.31). Thus, the results of the trial when using the second score (FeverPAIN) are presented in the text as the main findings, and the results of the first score shown in Appendix 6 – which show no significant differences for any outcome despite good compliance with the intended strategies in each group (see below).

Baseline table

Main findings

Selection and attrition bias

There was no evidence of clinical selection bias when comparing the patients in the two parts of the trial: score 2 and score 1 patients were similar for mean severity of sore throat and difficulty swallowing at baseline (3.20 vs. 3.12, respectively), as well as mean prior duration of illness (4.77 vs. 4.41 days), purulence [163/628 (26%) vs. 294/1124 (26%)], fever [358/630 (57%) vs. 645/1126 (57%)], mean age (29 vs. 29 years) and female gender [404/629 (64%) vs. 732/1128 (65%)]. Follow-up for diary information was also similar [502/631 (80%) vs. 901/1129 (80%)].

Based on clinician report, most patients who were assessed and approached agreed, which suggests the major issue for selection bias is likely to be which patients clinicians assessed and approached. The most common reasons given for not approaching patients were too little time to consent and recruit (87% of health professionals rated this the most common reason), followed by patients being too unwell (10%). Similarly, the most common reasons given for patients declining were having too little time to complete study materials (75%) or too little time to consent and be recruited (16%). The characteristics of patients recruited by GPs at high-recruiting surgeries [> 80 patients per year, contributing 49% (309/631) of score 2 patients] were similar to those of patients from lower-recruiting surgeries (mean sore/difficulty swallowing scores at baseline 3.2 vs. 3.2), although the mean number of streptococcal score features was slightly higher for those from high-recruiting surgeries (1.9 vs. 1.6, respectively). There was some evidence of modest clinical selection bias found in the trial population when it was compared with the previously described diagnostic studies. Although the diagnostic study used to generate FeverPAIN was similar for fever (59%), mean temperature (36.9 °C) and mean sore/difficulty swallowing score (diagnostic 3.1, trial 3.2), the baseline mean number of streptococcal score features was slightly lower in the trial population (diagnostic 2.1 vs. trial 1.7, SD 1.3). Furthermore, the purely observational cohort (MRC DESCARTE) had very similar patient characteristics to the diagnostic study (mean sore/difficulty swallowing score 3.0; mean number of streptococcal features 2.0). Given the similarity of patient-rated severity in all these studies, the slightly lower number of streptococcal features in the trial population may represent some recruitment bias in some practices based on recruiting fewer patients with more florid clinical signs. However, if practices that recruited patients with lower mean streptococcal scores (< 1.33) are excluded (leaving 448 patients), the baseline streptococcal scores for the trial are similar (2.1 diagnostic vs. 2.0 trial), and sore throat and difficulty swallowing symptoms are significantly better for the clinical score group (interaction term –0.60, 95% CI –1.01 to –0.20; p = 0.004), and also better for the rapid test group, but not significantly (interaction term –0.26, –0.67 to 0.16; p = 0.223). If this group of practices is then selected (i.e. excluding practices with low streptococcal features), the mean severity of sore throat and difficulty swallowing is improved by half a point in the clinical score group (–0.51, –0.87 to –0.15; p = 0.005), which is equivalent to one patient in two rating sore throat a slight rather than a moderately bad problem, with a similar effect in the rapid test group (–0.38, –0.75 to –0.01; p = 0.042) (see Table 11 for comparison). Thus, the estimates for the trial are probably conservative.

Strengths and potential limitations

Main results in context of previous literature

Using either RADTs or a clinical score (FeverPAIN) is likely to improve symptom control moderately compared with an empirical management strategy of delayed prescribing. As the score requires two examination components, it will be more difficult to use in telephone triage or out-of-hours contexts; it is likely to be difficult for patients to assess the severity of inflammation, although it might be feasible for patients to report the presence of pus. The effect on symptoms of the score in the first few days after seeing the doctor is to make a difference of one person in three rating sore throat and difficulty swallowing a slight problem rather than a moderately bad problem. Similarly, antibiotic use was lower in both of the intervention groups. These results also suggest there is likely to be little additional benefit either for symptom management or antibiotic use when using a strategy for rapid antigen testing targeted by clinical score when compared with using a clinical scoring method alone. This might be because the diagnostic advantage in RADTs that identify GABHS is in part matched by the disadvantage of not identifying group C and G streptococci, which provide similar symptom burden to GABHS. The additional disadvantages of RADTs are the costs of the test (although most RADTs cost less than £5, and the one chosen for the current study was approximately £3) and the staff time taken to get a result (5 minutes). A previous small trial of RADTs19 demonstrated that using the Centor score with or without rapid tests on its own did not modify antibiotic use, but this trial unfortunately did not report symptomatic outcomes. The difference between these trials may be that the current trial used a slightly more valid score: we have shown that individual items from the Centor score and score 1 did not perform optimally in our two prior diagnostic cohorts in terms of identifying patients with a low likelihood of streptococcal infection.

Based on the results of the previous diagnostic study, the limited performance of score 1 in ruling out streptococcal infection might help explain the differences in antibiotic use, but it is less clear why symptom management should be significantly better with FeverPAIN than with score 1. It may be that the particular combinations of more florid symptoms in FeverPAIN (e.g. fever, pus) are more important in determining symptom burden, and/or better in determining symptom response to antibiotics due to either microbiological or patient factors (e.g. the differential nature of organisms on the surface and in the crypts of the tonsils, or the relation between symptoms, the immune response and prognosis). 1,51 Furthermore, as our qualitative data suggest, doctors prefer clinical intuition to rigid use of clinical scores, it is also possible that FeverPAIN has greater clinical face validity for clinicians or patients. This could perhaps facilitate stronger advocacy and potentially better adherence to the proposed prescribing strategy – although our relatively crude data concerning what health professionals did (but not how they did it) suggest health professionals complied reasonably well with the proposed prescribing strategies.

The rate of antibiotic use reported in the current trial with delayed prescribing (> 40%) is significantly higher than previous research,33 but the number of features associated with streptococcal infection is also higher (e.g. 15% had purulent tonsils in the previous trial compared with 26% in this trial), so it may be that more patients with milder sore throats are now self-caring rather than consulting their GP compared with 15 years ago. This trial also recruited in slightly different settings: the first trial recruited predominantly in deprived inner-city settings, whereas the current trial had a wider range of practices.

We were unable to demonstrate any difference in belief in the need to see the doctor, or reconsultations either within a month or with longer follow-up (i.e. no apparent ‘medicalising’ effect of the RADT strategy). This contrasts with a dramatic medicalising effect of prescribing antibiotics. 23,33 However, the lack of an effect with a RADT in a trial setting in which RADTs are not used routinely may mean that it is difficult to demonstrate medicalisation in the short term. Longer-term implementation studies or possibly international comparison studies may be needed.

Background

The use of clinical scoring methods and RADTs could help target antibiotics at those who are most likely to benefit from them and help reduce antibiotic use, but there has been little exploration of what the key issues might be for patients and health-care practitioners (HCPs).

Objective

The aim of this study is to explore patient and HCPs’ views of clinical scores and RADTs.

Method

This was a qualitative study carried out in the primary-care setting. Semi-structured face-to-face and telephone interviews were conducted with GPs and nurse practitioners (NPs) from general practices across Hampshire, Oxfordshire and the West Midlands.

Results

In total, 51 participants took part in the study. Of these, 42 were HCPs (29 GPs and 13 NPs) and nine were patients. HCPs could see a positive role for RADTs in terms of reassurance, as an educational tool for patients and for aiding inexperienced practitioners, but also had major concerns about RADT use in clinical practice. Particular concerns included the validity of the tests (the role of other bacteria, and carrier states), the tension and possible disconnect with clinical assessment/intuition, the issues of time and resource use, and the potential for medicalisation of self-limiting illness. In contrast, however, experience of using RADTs over time seemed to make some participants more positive about using the tests. Moreover, patients were much more positive about the place of RADTs in providing reassurance and in limiting their antibiotic use.

Conclusions

It is unlikely that RADTs will have a (comfortable) place in clinical practice in the near future until health professionals’ concerns are met, and they have direct experience of using them. The routine use of clinical scoring systems for acute upper respiratory tract illness also faces important barriers related to clinicians’ perceptions of their utility in the face of clinician experience and intuition.

Participants and procedure

Participants were identified and recruited (by JB, JK, KM) from practices across the south and south-east of England, aiming for a mix of inner-city and rural locations (Table 12). HCPs were eligible for inclusion if their practice had consented to take part in the PRISM trial. NPs were eligible for inclusion if they had responsibilities that would involve using a RADT in practice. Patients were eligible for inclusion if they had experienced a RADT during a consultation.

The interviews

Trained interviewers (TL, RC, HH) conducted both face-to-face and telephone interviews (according to participants’ preference) with each lasting approximately half an hour. All interviews were audio recorded and transcribed verbatim in preparation for analysis.

Qualitative interviews provided the best method for gathering insights into participants’ individual experiences of using RADTs, and their views about the RADTs. The interview guide was amended before the final 11 interviews were conducted in order to include a question that asked about experiences of using clinical scoring methods. A semi-structured interview guide included key topic areas, while also providing flexibility to explore unanticipated issues.

Analysis

Inductive thematic analysis54 was conducted on all transcripts to determine HCPs and patients’ experiences and views of RADTs in practice. The first phase involved ‘immersion’ in the data, that is transcripts were read and reread a number of times in order for the researchers to become as familiar as possible with the data. The transcripts were then read again and meaningful sections of each transcript were systematically assigned a code to reflect issues which were represented within them. Each code label referred directly to the operationalisation of the theme discussed. This process was repeated after initial coding was complete and themes that consistently occurred within the data were identified and labelled. The codes and definitions were refined during a continuing process, which involved themes being linked, grouped, moved, relabelled, added and removed to produce a set of themes and subthemes that adequately fitted and thoroughly explained the data. Two qualitative researchers (LM and GL) iteratively developed the themes through discussion and adjustment when appropriate, before reaching ‘inter-rater’ agreement on the final themes and descriptions.

Participants

In total, 51 participants took part in the study. Of these, 42 were HCPs (29 GPs and 13 NPs) and nine were patients. GPs/NPs were recruited from 15 practices across Hampshire, Oxfordshire and the West Midlands. Registered practice population size varied from 3272 to 16,596. Surgeries were recruited from both inner-city and rural locations, with the Index of Multiple Deprivation score ranging from 2.7 to 36.9.

Themes

The analysis identified five main themes, and a total of 18 subthemes (Table 13).

In the following sections we describe each theme in turn, using exemplary quotations for illustrative purposes.

Strengths and limitations

One of the main strengths of the study is its attempt to include both HCPs and patients, to capture a dual view of the RADT in practice. As both groups had used the RADT, the study was able to directly examine the experience of conducting the RADT as a HCP and the experience of receiving the test as a patient. This process was able to reveal differences between HCPs’ perceptions of patient attitudes towards the RADT and actual patient views. The sample size was relatively large for a qualitative study; however, as the patient sample size was small, further work could usefully focus on this group to confirm and provide additional insight to the present findings. The NP sample was smaller than the GP sample, but consensus in most views was clear within the NP sample and NPs reported similar views of the RADTs to the GP sample. The only clear difference between the samples was identified in a view that was widely held by GPs but not NPs: that the RADT may be ‘unnecessary for experienced practitioners’.

Nesting qualitative interviews into the PRISM trial provided in-depth views, which help interpret the results obtained from the main trial. In terms of limitations, three researchers conducted the interviews and the level of exploration varied. Nonetheless, all interviews explored the key questions/topics and this permitted comparative analysis across the data corpus. Transparency of the analysis process and agreement on themes across the team ensured that final themes were robust.

Main findings

Overall, the interviews suggest that taking part in the PRISM trial had influenced the HCPs’ attitudes towards the use of RADTs, in that the tests were now viewed more positively than before participation in the trial. Positive views towards the RADTs related to patient outcomes, such as the test results providing reassurance to the patient that they do not need antibiotics and also providing an educational tool to indicate why antibiotics may not always be necessary. HCPs also felt that the test could provide a useful confirmation of diagnosis, which reassured them and supported their prescribing decisions. In relation to the practicalities of conducting the RADT, HCPs reported that it was easy to use and possibly a useful tool for less experienced staff members and that they felt it had reduced their antibiotic prescribing for sore throat – hence, potentially useful for experienced and less experienced staff.

There were also some negative views and concerns relating to the RADT. HCPs reported concerns regarding the additional costs to the practice and time taken to conduct the RADTs. However, the interviews also suggested that concerns relating to time were most likely to be reduced after the HCP had conducted the tests a few times and had become familiar with the procedure. Concerns were also raised in relation to the accuracy of diagnosis in terms of the possible identification of GABHS carriers who may be unnecessarily prescribed antibiotics, and the possible presence of alternative forms of bacteria that the RADT could not detect. HCPs were concerned with the difficulty in making a prescribing decision when the RADT result differed from their own opinion of diagnosis; however, it was also reported that the RADT could reassure them if they were in doubt over the diagnosis. Finally, there was some concern that the use of RADTs could increase the ‘medicalisation’ of sore throat and increase patient attendance to the GP for this condition. However, patient interviews suggested that tests would be unlikely to increase their GP attendance, and that patients valued the use of the RADT, reporting that they felt reassured by it.

Comparison with existing literature

It was reported that HCPs viewed the RADTs positively as a tool that could provide a confirmation of their own diagnosis and, therefore, support their prescribing decision. However, HCPs reported a negative view of the RADT if it presented evidence for a diagnosis that differed from their own opinion. Similar findings were also reported in a study that used a computer-based clinical intervention to assist with antibiotic prescribing decisions for respiratory tract infection. 56 GPs reported acceptance of use of the system if it was perceived as a tool that could support their own decisions. However, if the system were perceived as a method of enforcing a prescribing decision, the GPs viewed it negatively and did not wish to use it during consultations. In a similar vein, a study exploring the use of severity measures for the diagnosis of depression described the measures as beneficial when they confirmed a GP diagnosis and treatment disposal, but as of limited value when the score was not congruent with the GP’s view founded on clinical experience and intuition. 57

Practitioners also reported concern that use of the RADT could increase patient pressure for tests and GP consultations for sore throat. However, patients reported that, although they felt reassured by the tests, they would be unlikely to increase their GP attendance for the condition or their expectation for antibiotics. These findings are analogous to perceptions about prescriptions: GPs often perceive a pressure from patients to prescribe, investigate and refer despite the fact that patients do not report such expectations. 58 More recent findings from a qualitative study of patients attending with suspected urinary tract infection also identified patients’ reluctance to rely on antibiotics to ameliorate symptoms, whereas GP interviews highlighted belief in patient pressure or desire for antibiotic prescriptions,44,59 thus suggesting an ongoing disconnection between HCPs’ views of patients’ expectations and patients’ expectations.

Implications for clinical practice

Practitioners reported that being in the PRISM trial had resulted in them viewing RADTs more positively than before their participation. It was also reported that GPs who had not taken part in the trial were likely to view RADTs more negatively than their participating counterparts. This suggests that offering GPs the opportunity to pilot or trial the tests before agreeing to implement them in practice could enhance how the RADTs are subsequently embedded into routine primary care. Practitioners who discussed the clinical scoring method widely held the view that it was time-consuming, unnecessary and beneficial only for inexperienced staff. These perceptions may hinder acceptance of the score by experienced clinicians in the future and would need to be addressed in future initiatives aimed at incorporating these tests into practice.

Patients’ views of using the RADTs were generally positive, which suggests that patients would not object to their use in primary care. A larger sample of patients would be useful to further test and confirm the acceptability of the RADTs. HCPs were concerned that the RADT would increase consultations, but patients reported that their likelihood of attending the GP would, in fact, remain the same. This finding could be further explored with patients and, if confirmed, could usefully be presented to GPs to reduce their concerns about using RADTs in relation to the effects on patient attendance.

Aims

The aim of this study was to assess resource use and health-related quality of life (HRQoL) associated with clinical scores and RADTs and to show whether these can represent an efficient use of NHS resources.

Methods

A cost-effectiveness study (cost/change in symptom severity) and a cost–utility study [cost/quality-adjusted life-year (QALY)] were carried out as part of the randomised controlled trial. Resource use data were obtained from GP case notes and from study clinicians. QALYs were estimated by means of EQ5D scores obtained from the 14-day diary.

Results

The cost-effectiveness results presented are for FeverPAIN compared with control and RADT. In total, 498 individuals had both symptom severity and cost data from case notes review. Costs for the initial visit and for the 1-month follow-up were similar at £51, £44 and £49 for the delayed prescribing, FeverPAIN and RADT groups, respectively. FeverPAIN dominated both other groups for the cost/change in symptom severity analyses, being both less costly and more effective. Cost-effectiveness acceptability curves (CEACs) indicated the clinical score group to be the most likely to be cost-effective. The cost–utility analysis showed considerable uncertainty around change in QALYs, but FeverPAIN appeared to be the most likely to be cost-effective, particularly for lower values of QALY.

Conclusion

FeverPAIN is a more efficient use of health-care resources than the other approaches.

Data sources for economic evaluation

A number of sources were used to derive data for the economic evaluation. Information regarding the initial (randomisation) contact was entered directly into the website by study practitioners. This included duration, whether a RADT had been performed and prescription of antibiotics. Information on health-care resource use was obtained by a review of practice notes, carried out by a study researcher who visited each participating practice, using a data collection proforma. Resource use data collection was limited to upper respiratory tract-related contacts. Data collected included the following: GP and NP visits; antibiotics; practice visits for complications of infections and antibiotic complications; and hospital admissions related to infections. Data were collected in three time periods: first, the 12 months prior to recruitment (antibiotics; GP and NP visits); second, the 4 weeks following recruitment (GP and NP visits; antibiotics; visits for complications; secondary-care resource use); and, third, resource use in the period from 4 weeks after recruitment up to the end of data collection.

The final important source of health economics data was the 14-day diary completed by study participants. This included the EQ5D62 completed at baseline and at 14 days. This was used to calculate QALYs. If individuals did not return this diary, then data on symptoms were obtained from a short questionnaire or a telephone interview. However, these sources did not include the EQ5D.

Analysis of costs

Participants

Two clinical scoring algorithms were used in the clinical study. Score 1 was used at the start of the study and was used with 1129 participants. This was replaced as planned by FeverPAIN in the second part of the study (for 613 participants). The results of the clinical trial indicated no statistically significant results for score 1, but did find statistically significant results for FeverPAIN compared with delayed antibiotics (see Chapter 4). We confined our economic analysis to FeverPAIN, as this was the score tested in the clinical trial.

Outcome data used

Outcomes in the economic analysis were the symptom severity score (primary outcome measure in trial) and QALYs based on EQ5D. 62 EQ5D was obtained from the patient-completed diary. Values were obtained at baseline and at 14 days after recruitment. However, as EQ5D scores were obtained from the 14-day diary, we did not have EQ5D values for the end of the 28-day follow-up period. We assumed that health obtained at the end of the 14-day period persisted to the end of the study period, that is the last value obtained was carried forward for 14 days. However, we also calculated QALYs for the 14-day period to test the effect of this assumption on generated results. These were scored using the standard UK tariff (www.euroqol.org). Values obtained were converted into QALYs using ‘area under the curve’, that is values obtained were multiplied by duration spent in those health states. The estimates of QALY used in the analysis presented here are differences from baseline EQ5D, calculated by estimating ‘area under the curve’, then subtracting the area that would be generated if baseline EQ5D had been maintained for 28 days. The same method was used for the 14-day QALY estimates, but the baseline QALY for the 14-day period was subtracted. The estimated QALY gain was compared with baseline values of EQ5D.

This ‘area under the curve’ method assumes a linear relationship between values at different time points; that is, if the EQ5D score changes between baseline and 14 days, a straight line would be drawn between the two points (i.e. HRQoL changes at a uniform rate between the two time points). There is no guarantee that HRQoL would change in this manner, particularly with an acute illness. A number of alternative patterns of recovery would be possible. Two are considered here. First, individuals recover quickly in the first few days, with only minor improvements in health for the rest of the 14-day period. Second, individuals have poor health for the majority of the 14-day period and then recover quickly. Linear interpolation between the two EQ5D scores would understate the QALYs generated in the first case and overstate them in the second. It was not clear at the design stage how individuals’ health would change and what the effect of the profile of health change might be. It was not considered feasible to ask participants to complete the EQ5D daily without dramatically complicating and lengthening the short diary, possibly reducing completion rates. We included a page in the patient diary with a number of visual analogue scores (VAS). These were modified slightly in order to fit VASs for 14 days over two pages in the diary. However, the scales and the description of end points were not changed. Individuals were asked to complete these until they felt better and these scores were used to describe the profile of recovery from acute sore throat.

As VASs were completed only until recovery, we lacked scores for the full 14 days for most respondents. Confining analysis to complete data for each day would mean that only cases that took longer to recover would appear in the estimates for the later days. This would bias estimates downwards, and give a misleading picture of, the profile of potential recovery. Therefore, we assumed that the last value given when the individual had deemed themselves to be ‘fully recovered’ could be carried forward to represent the score they would have given in the remaining time period. These were calculated for individuals for whom we had both baseline and 14-day EQ5D, that is those individuals who we were able to calculate QALY scores for. Examination of the health profiles obtained from this process indicated that change in health did not differ markedly from a straight line recovery (Figure 9), although there did appear to be an increased rate of recovery in the first 8 days compared with the next 6. Linear interpolation between EQ5D scores at 0 and 14 days was used in our analysis.

FIGURE 9.

Daily VASs for the 257 participants used in the cost–utility analysis.

The primary outcome measure was the symptom severity score used in the main clinical study. This was the mean rating of sore throat and difficulty of swallowing for days 2–4. A higher score indicates worse symptoms.

Analysis

We present results for both a cost-effectiveness study using cost per point change in symptom severity score and a cost–utility analysis based on incremental cost per QALY. The time frame of the analysis is for 1 month after randomisation and the perspective that of the NHS. Costs included those associated with the initial visit as well as those in the 1-month follow-up. Data were analysed in PASW v. 18 (SPSS, Quarry Bay, Hong Kong), Stata v. 11.2 (StataCorp LP, College Station, TX, USA) and Microsoft Excel (Microsoft Corporation, Redmond, WA, USA). Mean and 95% CIs were generated for use cost variables. For outcome variables (both symptom score and QALYs), mean values were estimated using regression equations which allowed us to control for baseline characteristics (fever and baseline symptoms). Mean QALY gain at 14 and 28 days (with 95% CI) for each arm are adjusted by baseline sore throat/swallowing clinical score and fever. To do this, we estimated a linear model for the QALY gain controlling for the clinical score and fever, replicating the method used in the clinical study. We then predicted mean values and 95% CIs. Analyses were performed using the regress and adjust commands in Stata v. 11.2. Bootstrapping using 5000 samples was used to generate CEACs. These show the probability that each intervention is cost-effective at different values placed upon the outcome variable. The regression methods previously described were used to adjust for baseline characteristics when producing these bootstrapped estimates. These present results that allow for the uncertainty arising from trial data. Bootstrapping was also used to generate scatterplots on the cost-effectiveness plane. These also show the uncertainty inherent in study data.

Cost-effectiveness analysis

Table 16 shows the results based on the symptom score for the 498 individuals included in the cost-effectiveness study. Means for symptom score were adjusted for differences in baseline symptoms and fever, as detailed earlier. In terms of point estimates, the FeverPAIN group dominated the other two groups, being more clinically effective (having a lower symptom score) and less costly. However, for the comparison of FeverPAIN against RADT, it can be seen that the point estimates of symptoms are very similar with overlapping CIs. Therefore, it is important to consider uncertainty around these results. This can be seen in the CEAC (Figure 10). The value of a point change in the symptom score was varied between £0 and £500 in this analysis. For this entire range, the clinical score group was the most likely to be cost-effective. This was most marked at lower values of a point change on the symptom score, as the FeverPAIN approach was associated with lower costs. Although estimated symptoms were similar between FeverPAIN and RADT, FeverPAIN had a higher probability of being the most cost-effective option at each value of a point change in symptoms because it had lower costs.

FIGURE 10.

Cost-effectiveness acceptability curve for cost-effectiveness study.

Scatterplots for the cost-effectiveness analysis are given in Figures 11–13. In these scatterplots, a negative value of incremental effectiveness indicates a better outcome, as lower symptom scores are preferable. Figure 11 shows that the FeverPAIN group have lower symptom scores than the delayed group for most simulations and, for the vast majority of simulations, the costs of FeverPAIN are lower than those of delayed prescribing. Figure 12 shows that the RADT group appears to generate lower symptom scores than the delayed prescribing group. However, costs were very similar and are distributed fairly evenly over the x-axis. Finally, Figure 13 shows that the RADT appears to show similar effectiveness, but at increased costs, to the clinical score group.

FIGURE 11.

Scatterplot showing FeverPAIN vs. delayed prescribing group for cost per point change in symptom score.

FIGURE 12.

Scatterplot showing RADT vs. delayed prescribing group for cost per point change in symptom score.

FIGURE 13.

Scatterplot showing RADT vs. clinical score group for cost per point change in symptom score.

Cost–utility analysis

Results of cost–utility study

The results presented are for the 257 individuals included in the cost–utility complete case analysis. Table 18 shows the results of the cost per QALY analysis for the 14-day results. The QALY difference from baseline ranged from 0.0057 to 0.0058. There were no clear differences in generated QALYs among the three groups, as can be seen from the wide and overlapping CIs. The costs per patient were very similar to those generated for the larger data set of 498. However, as there was considerable uncertainty in these data, no clear conclusion should be drawn from this. The 28-day results are presented in Table 19 and these are similar (although larger in magnitude) with no clear differences among groups in QALY scores. As the QALY gain is marginally higher in the RADT group than in the FeverPAIN group, RADT generates additional QALYs at between £74,286 to £24,528 per QALY, depending on which QALY measure is used.

To take account of uncertainty, CEACs were generated for these results (Figures 14 and 15). These both show considerable uncertainty. This is largely generated by uncertainty around the QALY estimate. At a value of £30,000 per QALY, the probabilities that the three groups are cost-effective are 25%, 40% and 35%, for the delayed prescribing, FeverPAIN and RADT groups, respectively, for the 14-day QALY gain. For the 28-day QALY gain, the same values are 28%, 38% and 35%. As can be seen from the scatterplots (Figures 16–18), the distribution of estimated incremental QALYs is fairly evenly distributed around the y-axis. These scatterplots are for the 14-day QALY analysis. The 28-day analysis plots are very similar, apart from the magnitude of the QALY difference, so are not shown here.

FIGURE 14.

Cost-effectiveness acceptability curve for 14-day QALY difference.

FIGURE 15.

Cost-effectiveness acceptability curve for 28-day QALY difference.

FIGURE 16.

Scatterplot showing clinical score group vs. delayed prescribing group for cost/QALY analysis for 14-day QALY difference.

FIGURE 17.

Scatterplot showing RADT group vs. delayed prescribing group for cost/QALY analysis for 14-day QALY difference.

FIGURE 18.

Scatterplot showing RADT group vs. clinical score group for cost/QALY analysis.

Statement of main findings

The cost-effectiveness analysis showed that the clinical scoring algorithm used in the FeverPAIN group was effective in reducing symptoms. Costs in all three groups were similar and modest, but the FeverPAIN score group was associated with lower costs. This indicates that the use of the FeverPAIN score may be an efficient use of health-care resources. However, for the cost–utility analysis (cost/QALY) there was a less clear message, as differences in QALYs generated were very small with wide CIs, and therefore there was no statistically significant differences between any groups. The CEACs for the cost–utility study indicate that FeverPAIN is the most likely to be cost-effective over all values of a QALY examined; however, they also indicate considerable uncertainty. The cost/QALY offers weak support for the conclusions of the cost-effectiveness analysis.

Strengths and potential limitations

A strength of the current work is that it used a clinical measure of symptoms as well as the QALY. The former was plausibly more sensitive to changes in participants’ symptoms of sore throat. Its administration ensured it picked up changes day to day. Another strength concerns the costs obtained from a case note review being comprehensive. A weakness of the current study is that the 14-day diary had EQ5D data only at two time points (0 and 14 days) and for 52% of all participants. We did not find any statistically significant differences in terms of QALY differences for participants. There could be a number of reasons for this. First, the smaller data set for which we had QALY score may not be representative of the larger group of individuals in the cost-effectiveness study. Second, it may be that many individuals would have recovered before the 14-day QALY was asked and thus their health may have returned to normal. For this reason, the 14-day EQ5D score and hence the QALY difference at 14 days may not be well correlated with changes in symptom scores. It may also be the case that the EQ5D is not sensitive in assessing HRQoL related to change in sore throat-related symptoms. The true impact of RADTs in practice may not be fully captured in our estimation of costs: we estimated an additional 4.5 minutes in consultation time on average when a test was performed, which may be an underestimate of the time taken to do a RADT – the pack and reagents have to be prepared, a throat swab taken, the antigen extracted from the swab, reagents mixed, the resulting mixture applied to the test pack, and then for a negative result there is a 5- to 10-minute wait (see Appendix 7). The reason for this possible underestimate is unclear, but clinicians may have been able to be more rapid with trial paperwork when they knew they had to do a RADT. The time taken to feed back the results may also not have been captured. Furthermore, the ‘costs’ of the stress of a longer consultation, and the knock-on effect for other consultations, are not included in costs. Thus, it is likely that RADTs are, if anything, less cost-effective than we have estimated.

The estimates of both clinical effectiveness and cost-effectiveness may be modified by alternative algorithms for using RADTs (e.g. using a RADT at lower FeverPAIN scores). However, a more widespread use of RADTs would probably not be acceptable to clinicians in the UK based on our qualitative work, and the targeting of RADTs in the current study was based on modelling to determine the most efficient use of the tests in identifying patients with streptococci, so more widespread use of RADTs is likely to be a less efficient policy.

Relation to existing literature

Webb60 used decision analysis for a paediatric population and compared immediate treatment, culture, RADT and RADT and culture and concluded that empirical treating was most cost-effective. Neuner et al. 61 modelled five strategies (no treatment; immediate treatment; culture and delayed antibiotics until results; RADT with culture for negative results; and RADT alone) and, in contrast to Webb, found empirical treatment to be the least cost-effective, but all others to have a similar QALY gain, and culture to be the least expensive so the most cost-effective. Neither of these studies collected prospective data from a trial and neither performed direct measurement of quality of life or the change in quality of life over time.

Our Research Objectives are:

1. to assess which RADT is the most accurate in predicting the presence of group A streptococcus by throat swab in a clinical sample from primary care;

2. to estimate the error from sampling bias by performing parallel standardised in vitro studies;

3. to assess the validity of a scoring system based on the throat swab as the reference standard (such as the Centor criteria) in a UK population;

4. to assess the effectiveness and cost-effectiveness of rapid tests when compared to clinical scoring rules and delayed antibiotic prescription;

5. to explore the effect of additional benefit from the RADT use on GP diagnostic prediction accuracy and treatment decisions.

Existing research.

Research methods

Methods

Phase 2