New technologies for diagnosing active TB: the VANTDET diagnostic accuracy study

Article history

The research reported in this issue of the journal was funded by the EME programme as project number 12/65/27. The contractual start date was in March 2014. The final report began editorial review in April 2018 and was accepted for publication in December 2018. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The EME editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

© Queen’s Printer and Controller of HMSO 2021. This work was produced by Halliday et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2021 Queen’s Printer and Controller of HMSO

Background: tuberculosis

Tuberculosis (TB) is a devastating disease, with approximately 10 million cases worldwide each year. 1 Globally, it is one of the top 10 causes of death and is the leading cause of death from a single infectious agent, ahead of human immunodeficiency virus (HIV)/acquired immunodeficiency syndrome (AIDS). 1 Important characteristics of the current global epidemic are the interaction with the HIV/AIDS epidemic (with HIV/TB co-infection driving increased mortality) and the increasing rates of multidrug-resistant TB cases. The areas with the greatest TB burden are in south-east Asia, Africa and the Western Pacific World Health Organization (WHO) regions. England has among the highest rates of TB in Western Europe, with 5664 notifications and a rate of 10.2 cases per 100,000 people in 2016. 2 The majority of TB cases in Europe occur in large cities (especially London, Birmingham and Leicester in England), and the disease disproportionately affects immigrants and socially marginalised and socioeconomically disadvantaged groups.

Tuberculosis is caused by the bacterium Mycobacterium tuberculosis (Mtb), and is transmitted from person to person via the aerosol route by droplet particles that are produced when an individual with the pulmonary form of TB coughs. On exposure to an uninfected individual, a number of eventualities can occur. Approximately 50% of exposed individuals resist the infection; the remainder become infected, as identified by the detection of peripheral cellular immune responses to Mtb antigens. The majority of infected individuals will develop prolonged but controlled latent infection of Mtb [i.e. latent tuberculosis infection (LTBI)]. It is estimated that between one-quarter and one-third of the world’s population has LTBI. 3 In a small proportion of infected individuals, the immune response can lose control of Mtb replication, and in such cases TB disease will develop. In most cases this development to active TB occurs within the first 6 months after initial infection,4 although progression from LTBI to active TB (also known as reactivation) can occur at any time during an individual’s lifetime. Active TB can manifest in many forms and can infect any organ of the body. Owing to the mode of transmission, pulmonary tuberculosis (PTB) is the typical form of TB; however, extrapulmonary tuberculosis (EPTB) is becoming increasingly common, particularly in cases of reactivation from LTBI.

Reduction of TB cases and working towards elimination of TB as a public health issue are nationally and internationally agreed as public health priorities at the level of the UK, European Union and WHO. 1,5 Although global TB incidence is declining by around 2% per year, improved reduction rates are required to progress towards elimination. Internationally agreed targets for improved global TB control, such as in the sustainable development goals or the WHO’s End TB Strategy,6 require that TB incidence rates fall by up to 5% per year, which is not achievable with existing tools. New innovative methods for TB prevention, diagnosis and treatment are urgently needed, and the WHO’s global plan calls for researchers and industry to deliver an improved, fit-for-purpose anti-TB toolkit, with major emphasis on the need for new diagnostic tests. 7 Prompt and accurate diagnosis and treatment of TB are essential to improve individual patient outcomes, as well as to prevent onward transmission in the community with its attendant health and social care costs. However, the currently available diagnostic toolkit lacks the ability to provide results with clinically acceptable speed and diagnostic accuracy.

Clinical unmet needs

Globally, there is a need for improved diagnostic tests for both active TB and LTBI. As identified by the New Diagnostic Working Group (WHO and Stop TB Partnership), there is a need for a rapid non-sputum-based biomarker test for detecting active TB. 16 A fast, highly sensitive test for all types of active TB, with improved sensitivity over the current microbiological tests (which provide high specificity but low sensitivity), would improve triage of TB suspects and increase the speed at which patients start appropriate treatment.

In high-income, low-incidence settings, such as in the UK, a high proportion of new cases of active TB are a result of reactivation from LTBI. For this reason, a major priority for TB control programmes in these settings is to identify individuals with LTBI and to offer prophylactic treatment to those at highest risk of progression to active TB. 17 Although IGRAs and TST provide high sensitivity for these individuals, an independent or follow-on test with improved specificity would enable us to reduce the number needed to treat to prevent subsequent cases of progression to active TB.

Therefore, the major clinical unmet needs for improved TB diagnostics in the UK are as follows:

• a triage rule-out test for active TB

• a rapid, accurate test for identifying ‘highly probable’ TB, where a clinical diagnosis cannot be confirmed by culture

• a sensitive test to predict those at increased risk of progressing to active TB from exposure and/or positive test for LTBI.

Research objectives

Study management structure

Overview of study design

The original protocol specified the link between the IDEA study and the VANTDET study, as well as the processes and samples for each VANTDET substudy (Figure 1). In the IDEA study, additional blood was taken to allow for long-term storage of PAXGene^® (Beckton, Dickinson Biosciences, NJ, USA), serum and peripheral blood mononuclear cell (PBMC) samples. These samples, coupled with clinical data, provided the biobank for the VANTDET study.

FIGURE 1.

Proposed VANTDET study design and link with the IDEA study. ATB, active tuberculosis.

Patient and public engagement and involvement

The TB research centre where the laboratory team of the VANTDET study is based is associated with the Health Protection Research Unit (HPRU) in Respiratory Infections, and both are led by the VANTDET study’s chief investigator, Professor Ajit Lalvani. The HRPU conducts an extensive patient and public involvement (PPI) and engagement programme, which has been recognised within the Imperial College London as an exemplar of good practice (see our case studies in the Imperial Patient Experience Research Centre30,31). The TB research centre team, including the VANTDET study researchers AH, MTW, LH, TM and RP, are frequently requested to run public engagement events.

For the IDEA study, Ms Nisha Karnani (a lay member) was appointed as the patient representative for the duration of the study. She was consulted at key points during the study and was invited to Study Steering Committee meetings and the IDEA study presentation at the end of the study.

The HPRU conducts quarterly symposium series, in which the VANTDET study team has disseminated and discussed project findings to a wide audience, including the HPRU in Respiratory Infections, the ISAB, patient representatives, collaborators and colleagues. Members of the management board have an excellent track record in active engagement with patients and public:

• Professor Ajit Lalvani was a key member on the panel at the Engagement to End TB event at the University of East London. This was part of the Voices for London Festival, which was attended by community members of East London.

• The VANTDET study team has participated in several public engagement events to showcase the TB research we are involved in. We also, along with other TB research centre members, received specific public engagement funding to develop and evaluate an animation film and LTBI diagnostics and treatment. This film was evaluated with screening and focus groups events for community groups, members of the public and other institutions involved in public engagement. These included the Imperial Festival (May 2017), Bromley by Bow Unity Festival (East London, July 2017), screening of the LTBI film at the Midaye Somali Women’s Group (White City, December 2017) and Let’s Talk about Tuberculosis (White City, December 2017).

The team has also published their research work in various online articles for a wider public audience:

• Tuberculosis in England: How Research at Imperial is Supporting the National Strategy. 32

• Increasing Awareness About Latent Tuberculosis Infection. 33

• Multilingual White City Residents Help Imperial Raise Awareness of TB Research. 34

• Case Study #6: Evaluating a TB Awareness Animation. 35

The HPRU in Respiratory Infections and TB research centre have established their first patient and public panel that comprises seven members of the public directly or indirectly affected by, or interested in, our research areas and both with and without previous experience of PPI. The role of the panel is to support the delivery of effective PPI and engagement approaches and activities within the HPRU. The panel advises on different PPI avenues available for disseminating our findings (e.g. charity/voluntary organisation websites and blogs), and also help to distribute findings within their informal networks.

Introduction

Several studies have demonstrated that whole-blood gene expression profiles have the potential to be used for diagnostic tests for active TB, in adults and children. In 2010, Berry et al. 18 demonstrated that differential transcriptional patterns in peripheral blood can be harnessed to differentiate TB from other conditions and a whole-blood 86-transcript signature distinguished active TB from other inflammatory and infectious diseases. A major milestone was reached by a study that showed that a 44-blood transcript signature was able to differentiate active TB from ODs using a cohort from a high-burden setting. 20 Anderson et al. 36 used a similar approach to identify a 51-gene signature that could distinguish TB from ODs in a cohort of children, all of whom had suspected TB at recruitment. This study has particular strengths in the approach to study design, as highly probable TB cases were also included and the OD group included a wide variety of differential diagnoses that present in a real-life clinical setting. Such a study in adults has yet to be used for validation of transcriptomic signatures for TB diagnosis. Maertzdorf et al. 37 identified a four-gene signature (guanylate-binding protein 1, inhibitor of DNA binding 3, interferon-induced transmembrane protein 3 and purinergic receptor, P2RY14) that distinguished subjects with active TB from healthy controls (HCs) and those with LTBIs with high accuracy. Similarly, a combination of four genes [cluster of differentiation 177 (CD177), Charcot–Leyden crystal galectin, haptoglobin (HP) and immunoglobulin J chain] was identified using support vector machine learning (SVML) and was shown to accurately classify active TB (including EPTB) from non-TB febrile patients. 38

Most recently, Sweeney et al. 39 used a meta-analysis approach to derive a TB risk score from three genes (dual-specificity phosphatase 3, guanylate-binding protein 5 and Krüppel-like factor 2), which was able to distinguish active TB from ODs with high accuracy. The important characteristics and diagnostic classification of TB are summarised in Table 29 for the five studies and the meta-analysis.

The aim of this substudy is to define the role of previously identified whole-blood gene-expression signatures in the diagnostic evaluation of active TB using host transcriptomic microarray.

Methods

Results

Final cohort

The study flow chart in Figure 2 shows the final cohort selected from the total IDEA study cohort. In the IDEA study, 288 patients did not give consent for a PAXGene sample to be taken or for genetic analysis. Of the 799 patients who were eligible for the transcriptomic substudy, 628 patients were selected for RNA extraction, had sufficient RNA, were used in microarray analysis and had passed the RNA and micorarray QC checks.

FIGURE 2.

Study flow chart for validation of transcriptomics signatures. a, 101 samples from OD patients who were HIV+ were excluded. From the full cohort of 1162 patients recruited into the IDEA study, 628 were selected for RNA extraction. Of note, the rate of active TB among the HIV+ cohort of patients in IDEA study was low (≈ 14%). Therefore, all of the eligible HIV+ active TB cases (n = 20) were selected for cohort inclusion, with a similar number of HIV+ OD participants (n = 25) randomly selected from the 146 HIV+ participants available.

The main demographic characteristics of the transcriptomics validation cohort are presented in Table 3. Of all TB cases included, 29.6% were culture unconfirmed (highly probable TB/Dosanjh category 2), which reflects the proportion of this group in the full IDEA study cohort of TB patients (in which 28.1% of TB cases were highly probable). As in the overall IDEA study cohort, significant differences were observed between patients groups, with TB patients being younger, weighing less and having lower body mass index (BMI), and more likely to be of Indian subcontinent ethnicity (includes Indian, Bangladeshi, Pakistani and Sri Lankan).

TABLE 3 - Demographic characteristics of transcriptomic validation cohort (n = 628) of patients selected from the IDEA study cohort

Where the data were categorical, a chi-squared test was used to compare characteristics between groups; where the data were continuous, the groups were compared using a Kruskal–Wallis test.

The full cohort is divided into the three diagnostic categories as outlined in the Dosanjh reference standard [i.e. culture-confirmed TB, highly probable TB and active TB excluded (referred to here as ‘OD’)].

Demographic characteristic	Culture-confirmed TB (Dosanjh category 1) (N = 212)	Highly probable TB (Dosanjh category 2) (N = 89)	OD (Dosanjh category 4) (N = 327)	p-value overalla
Age (years), median (range)	32.0 (16.0–81.0)	36.0 (18.0–76.0)	43.0 (17.0–87.0)	< 0.001
Male, n (%)	149 (70.3)	45 (50.6)	189 (57.8)	0.001
Ethnic origin, n (%)				< 0.001
Asian	12 (5.66)	3 (3.37)	12 (3.67)
Black	36 (17.00)	17 (19.10)	62 (19.00)
Hispanic	1 (0.47)	0 (0.00)	1 (0.31)
Indian subcontinent	139 (65.60)	61 (68.50)	150 (45.90)
Middle Eastern	3 (1.42)	0 (0.00)	11 (3.36)
Mixed	2 (0.94)	0 (0.00)	7 (2.14)
Unable/unwilling to respond	0 (0.00)	0 (0.00)	1 (0.31)
White	19 (8.96)	8 (8.99)	83 (25.4)
Height (m), median (range)	1.69 (1.37–1.96)	1.67 (1.47–1.98)	1.70 (1.32–1.96)	0.305
Height missing, n (%)	94 (44.3)	33 (37.1)	106 (32.4)
Weight (kg), median (range)	63.1 (34.7–1270)	63.0 (40.4–1160)	68.7 (41.4–1400)	< 0.001
Weight missing, n (%)	6 (2.8)	4 (4.5)	24 (7.3)
BMI (kg/m2), median (range)	22.6 (15.7–48.5)	22.5 (15.6–42.2)	24.4 (14.8–47.2)	0.005
BMI missing, n (%)	96 (45.3)	36 (40.4)	114 (34.9)

The main clinical features of the transcriptomic signatures validation cohort are presented in Table 4. There was a higher proportion of HIV+ individuals in the highly probable TB group than in the culture-confirmed TB and OD groups. Because the number of HIV+ OD cases was higher in the overall IDEA study cohort, the HIV+ rate we reported within this subcohort does not reflect the parent population. However, all HIV+ TB patients were included. The higher proportion of HIV-infected individuals in the highly probable TB group than in the culture-confirmed TB group reflects the comparatively paucibacillary nature of active TB and HIV co-infected cases.

TABLE 4 - Clinical characteristics of transcriptomics validation cohort, stratified by patient final diagnosis

Where the data were categorical, a chi-squared test was used to compare characteristics between groups; where the data were continuous, the groups were compared using a Kruskal–Wallis test.

Clinical characteristic	Culture-confirmed TB (Dosanjh category 1) (N = 212)	Highly probable TB (Dosanjh category 2) (N = 89)	OD (Dosanjh category 4) (N = 327)	p-value overalla
Clinical setting, n (%)				0.74
Unknown	1 (0.47)	1 (1.12)	1 (0.31)
Inpatient	68 (32.1)	31 (34.8)	103 (31.5)
Outpatient	143 (67.5)	57 (64.0)	223 (68.2)
BCG vaccinated yes, n (%)	126 (59.4)	46 (51.7)	191 (58.4)	0.439
BCG scar visible, n (%)				0.632
Missing	49 (23.1)	19 (21.3)	75 (22.9)
No	13 (6.13)	4 (4.49)	16 (4.89)
Unsure	10 (4.72)	8 (8.99)	30 (9.17)
Yes	140 (66.0)	58 (65.2)	206 (63.0)
HIV+, n (%)	9 (4.2)	11 (12.4)	25 (7.6)	0.04
TB contact no, n (%)	160 (75.5)	72 (80.9)	257 (78.6)	0.527

The distribution of type and severity of TB, and distribution of alternative diagnoses in the OD group, are presented in Appendix 1.

Diagnostic accuracy analysis

There were no missing data for any of the genes for the validation of published signatures.

Performance of ‘score’-based signatures in diagnosing active tuberculosis

Three of the published signatures presented a score [referred to in the publications as either disease risk score (DRS) or tuberculosis score (TBS), as previously described20,36,39], which is calculated using the expression levels of signature genes. We were able to apply the same score-based approach to testing these gene signatures within the signatures in our cohort. Cross-tabulation of the score value distributions against diagnostic categories according to the reference standard (Dosanjh criteria) is shown in Table 26 in Appendix 3.

Analysis of performance of score-based signatures in the full cohort

The distribution of score values in TB and OD cases in the overall cohort, as well as within either the HIV+ or HIV– cohort, is shown in Figure 16 in Appendix 3 (the ROC curves showing diagnostic performance are also shown).

The TB group had significantly higher score values than the OD group for all three signatures when assessed within the full transcriptomic cohort, with the difference being lowest for the Anderson et al. 36 DRS. The Kaforou et al. 20 44-gene signature and the Sweeney et al. 39 three-gene signature performed comparably, and both performed better than the Anderson et al. 36 signature in the overall and HIV+ cohorts, with area under the curve (AUC) values between 0.82 (95% CI 0.78 to 0.85) and 0.84 (95% CI 0.80 to 0.86).

Interestingly, although the scores were all significantly higher in the TB groups than in the OD group within the HIV– population, there were no significant differences between scores in these groups within the HIV+ population (Table 5 and Figure 16c in Appendix 3). This was reflected in a poor diagnostic performance (AUC = 0.50–0.69) for all three signatures in the HIV+ population, and the observation that, when the HIV+ population was excluded from the overall cohort, the AUC for each signature improved slightly (see Table 5).

TABLE 5 - Performance of published gene expression score signatures in full cohort, stratified by HIV infection status

–, negative; –/+, negative and positive; +, positive.

To provide sensitivity > 95%.

To provide specificity > 90%.

To provide the optimal Youden index.

When two cut-off values gave the optimal Youden index, both are given.

Signature (reference)	HIV	TB (n)	OD (n)	AUC, %(95% CI)	Cut-off point	Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV	NPV
DRS (Anderson et al.36)	–	281	302	0.70 (0.66 to 0.75)	> 86.0a	95.0 (91.8 to 97.3)	23.8 (19.2 to 29.1)	0.48	0.86
					> 100.6b	30.3 (24.9 to 36.0)	90.0 (86.1 to 93.2)	0.70	0.63
					> 91.4c	75.4 (70.0 to 80.4)	53.6 (47.8 to 59.4)	0.55	0.74
	+	20	25	0.50 (0.33 to 0.68)	< 105.6a	95.0 (75.1 to 99.9)	8.0 (0.98 to 26.0)	0.44	0.68
					< 84.2b	5.0 (0.13 to 24.9)	92.0 (74.0 to 99.0)	0.32	0.56
					> 97.9c	70.0 (45.7 to 88.1)	44.0 (24.4 to 65.0)	0.49	0.66
	–/+	301	327	0.68 (0.64 to 0.72)	> 86.0a	95.0 (91.9 to 97.2)	22.9 (18.5 to 27.9)	0.48	0.86
					> 100.8b	28.2 (23.2 to 33.7)	90.2 (86.5 to 93.2)	0.68	0.62
					> 91.4c	75.4 (70.2 to 80.2)	52.0 (46.4 to 57.5)	0.54	0.74
DRS (Kaforou et al.20)	–	281	302	0.83 (0.79 to 0.86)	> 101.1a	95.0 (91.8 to 97.3)	32.8 (27.5 to 38.3)	0.52	0.90
					> 113.4b	57.0 (51.0 to 62.8)	90.4 (86.5 to 93.5)	0.82	0.74
					> 108.6c	74.7 (69.2 to 79.7)	78.8 (73.8 to 83.3)	0.73	0.81
	+	20	25	0.69 (0.53 to 0.85)	> 104.3a	95.0 (75.1 to 99.9)	36.0 (18 to 57.5)	0.53	0.91
					> 124.2b	35.0 (15.4 to 59.2)	92.0 (74.0 to 99.0)	0.77	0.65
					> 120.2d	45.0 (23.0 to 68.5)	88.0 (68.8 to 97.5)	0.74	0.68
					> 113.2d	65.0 (40.8 to 84.6)	68.0 (46.5 to 85.05)	0.61	0.72
	–/+	301	327	0.82 (0.78 to 0.85)	> 101.1a	95.0 (91.9 to 97.2)	31.8 (26.8 to 37.2)	0.51	0.89
					> 114.7b	53.2 (47.4 to 58.9)	90.2 (86.5 to 93.2)	0.80	0.72
					> 108.6c	74.4 (69.1 to 79.3)	76.8 (71.8 to 81.2)	0.71	0.80
TBS (Sweeney et al.39)	–	281	302	0.84 (0.81 to 0.87)	> –4.05a	95.0 (91.8 to 97.3)	34.4 (29.1 to 40.1)	0.52	0.90
					> –2.60b	56.2 (50.2 to 62.1)	90.0 (86.1 to 93.2)	0.81	0.7
					> –3.2c	79.4 (74.2 to 83.9)	76.8 (71.7 to 81.5)	0.72	0.83
	+	20	25	0.64 (0.47 to 0.81)	> –4.51a	95.0 (75.1 to 99.9)	0.0 (0.0 to 13.7)	0.42	0
					> –1.69b	15.0 (3.2 to 37.9)	92.0 (74.0 to 99.0)	0.59	0.59
					> –2.49c	55.0 (31.5 to 76.9)	80.0 (59.3 to 93.1)	0.67	0.70
	–/+	301	327	0.83 (0.80 to 0.86)	> –4.1a	95.0 (91.9 to 97.2)	29.4 (24.5 to 34.6)	0.50	0.89
					> –2.5b	55.2 (49.3 to 60.9)	90.2 (86.5 to 93.2)	0.81	0.73
					> –3.2c	78.4 (73.3 to 82.9)	75.5 (70.5 to 80.1)	0.71	0.83

The full diagnostic performance of the ‘score’-based signatures in full, HIV+ and HIV– populations is also presented in Table 5, including a range of sensitivity and specificity values selected by varying the cut-off values for each score-based test based on predefined criteria. The PPVs and NPVs are also given.

The poor performance of these signatures within the HIV+ population suggests that these signatures are unable to detect TB in this population. However, the variability in the diagnostic accuracy scores demonstrates uncertainty in the estimates and reflects the small numbers of patients.

The performance of score signatures by Dosanjh classification

By comparing the ‘score’ signature results between Dosanjh categories, we can ascertain the effect of culture status within the TB group, and the likelihood of LTBI within the OD group (Figure 3).

FIGURE 3.

The performance of previously published score signatures, stratified by Dosanjh category. (a) Anderson et al. ;36 (b) Kaforou et al. ;20 and (c) Sweeney et al. 39 Statistical differences between score values between groups were determined using the Kruskal–Wallis with Dunn’s post hoc test for multiple comparisons. *p < 0.05; **p < 0.005; ***p < 0.0005; ****p < 0.0001.

In all cases, the score signatures were higher (significantly so in the case of the Kaforou et al. 20 and Sweeney et al. 39 signatures) in the culture-confirmed TB cases (Dosanjh category 1) than the highly probable TB cases (Dosanjh category 2) and, as a result, the diagnostic accuracy (AUC values) for culture-confirmed TB was slightly better than for highly probable TB for all score signatures (see Appendix 4). The maximum AUC for the detection of culture-confirmed TB was 0.86 (95% CI 0.83 to 0.89) using the Sweeney et al. 39 TBS, [the Kaforou et al. 20 DRS gave similar performance AUC of 0.85 (95% CI 0.81 to 0.88)] (see Table 27). For distinguishing highly probable TB from OD, the highest AUC achieved was 0.75 (95% CI 0.70 to 0.81), again with the Sweeney et al. 39 TBS.

No differences were observed in score signatures between the subgroups of OD patients in Dosanjh category 4, suggesting that neither previous history of TB or inactive TB (not to be confused with diagnosed LTBI) nor stratified risk of LTBI within the OD population affected the outcome of these score signatures. In fact, the highest AUCs were reported when the 4A and 4B groups (highest risk of having LTBI) were classified from all TB cases (see Figure 17b), the opposite of what was expected. The full diagnostic accuracy data are shown in Table 27 in Appendix 3, including additional analyses with the HIV+ patients excluded from the cohort. As expected, given the poor diagnostic performance within the HIV+ population, removing this group of patients resulted in slightly improved diagnostic performance (see Figure 17).

The performance of score signatures in detecting pulmonary or extrapulmonary tuberculosis

The clinical presentation, management and outcome of PTB and EPTB are different, and the latter is typically harder to diagnose as a result of several factors (e.g. difficulty of obtaining biopsy samples). Therefore, we wished to compare the score values in these groups and determine the diagnostic performance in either of these subclassifications of TB. As with the previous analyses, the Kaforou et al. 20 and Sweeney et al. 39 score signatures performed similarly when classifying either PTB or EPTB, whereas the Anderson et al. 36 signature had lower AUC values (see Figure 18 in Appendix 3). The optimal performance for diagnosing either PTB or EPTB was comparable, with highest AUCs being 0.85 (95% CI 0.81 to 0.89; achieved by both the Kaforou et al. 20 DRS and the Sweeney et al. 39 TBS) and 0.84 (95% CI 0.80 to 0.87; achieved by the Sweeney et al. 39 TBS), respectively.

In addition, we stratified the OD group by the most prevalent disease types, so that we could identify any differences within this heterogeneous population of TB suspects with a wide array of ODs. Interestingly, sarcoidosis was the only condition within the OD group that consistently did not differ significantly from either type of TB (PTB or EPTB) across all three signatures (see Figure 18b and 18c), suggesting that this specific differential diagnosis is a major confounding factor for the optimal performance of these score-based signatures in a real-life clinical setting.

Performance of score signatures in detecting tuberculosis in smear-negative tuberculosis suspects

Among those in the IDEA study population tested for acid-fast bacilli (76% of IDEA study cohort), using smear microscopy, the test provided high specificity (96.7%) and poor sensitivity (23%) for all TB. Therefore, smear microscopy functions as a reliable cheap and fast rule-in test for (mostly) PTB in the UK setting, and a positive result indicates high bacterial burden. This allows clinicians to start treatment for this important infectious subgroup of TB patients very quickly. However, there is currently no rapid and reliable rule-in or rule-out test with which to identify TB in the ≈77% of smear-negative (paucibacillary) TB suspects who present in secondary care, as culture confirmation can take several weeks. We therefore assessed the performance of the score signatures within this smear-negative population of TB suspects and, using the prevalence data from the IDEA study as the pre-test probability, calculated the PPVs and NPVs for these tests. In Table 6 we present the performance of these scores in the smear-negative population of TB suspects.

TABLE 6 - Performance of score signatures in the detection of subgroups of TB cases within the smear-negative population of TB suspects

The optimal score cut-off values for providing optimal sensitivity and specificity were calculated using the Youden index.

Performance characteristic	Signature
	Anderson et al.36	Kaforou et al.20	Sweeney et al.39
Performance in smear-negative TB (Dosanjh categories 1 and 2) (n = 198 TB vs. n = 244 OD) Prevalence of TB in all smear-negative TB suspects = 46% (from overall IDEA study population)
AUC, % (95% CI)	0.67 (0.61 to 0.71)	0.79 (0.75 to 0.83)	0.80 (0.76 to 0.84)
Cut-off point (Youden index)a	> 91.78	> 107.5	> –3.241
Sensitivity, % (95% CI)	72.2 (65.43 to 78.34)	77.3 (70.8 to 82.9)	78.3 (71.9 to 83.8)
Specificity, % (95% CI)	50.8 (44.4 to 57.3)	71.3 (65.2 to 76.0)	74.2 (68.2 to 79.55)
PPV/NPV	0.56/0.68	0.69/0.79	0.72/0.80
Performance in smear-negative culture-confirmed (Dosanjh category 1) TB (n = 139 TB vs. n = 244 OD) Prevalence of culture-confirmed TB = 33%
AUC, % (95% CI)	0.66 (0.61 to 0.72)	0.81 (0.77 to 0.86)	0.82 (0.78 to 0.87)
Cut-off point (Youden index)a	> 91.78	> 107.5	> –3.233
Sensitivity, % (95% CI)	72.7 (64.5 to 79.9)	82.0 (74.6 to 88.0)	82.7 (75.4 to 88.6)
Specificity, % (95% CI)	50.8 (44.4 to 57.3)	71.3 (65.2 to 76.9)	74.2 (68.2 to 79.6)
PPV/NPV	0.42/0.79	0.58/0.89	0.61/0.90
Performance in detecting smear-negative and culture-negative (Dosanjh category 2) TB (n = 59 TB vs. n = 244 OD) Prevalence of culture-unconfirmed TB in smear-negative TB suspects = 13%
AUC, % (95% CI)	0.65 (0.58 to 0.73)	0.74 (0.67 to 0.81)	0.75 (0.68 to 0.82)
Cut-off point (Youden index)a	> 92.41	> 109.5	> –3.467
Sensitivity, % (95% CI)	71.2 (57.9 to 82.2)	62.7 (49.2 to 75.0)	74.6 (61.7 to 85.0)
Specificity, % (95% CI)	52.9 (46.4 to 59.3)	76.2 (70.4 to 81.4)	68.9 (62.6 to 74.6)
PPV/NPV	0.18/0.92	0.28/0.93	0.26/0.95

These results demonstrate that the performance of the Sweeney et al. 39 and Kaforou et al. 20 transcriptomic signatures in the smear-negative population of TB suspects is sufficient to provide a high NPV (89–90%) for culture-positive TB in a UK-like setting, in which the prevalence of culture-positive TB in smear-negative TB suspects was 33%.

Analysis of Roe et al. signatures using published method

We obtained the SVML model from Roe et al. 38 and validated the signature using the approach proposed in their publication. The performance of this model for discriminating TB cases from patients with ODs in our cohort was very poor with an AUC of 0.59 (95% CI 0.54 to 0.63). Although sensitivity for TB was high using this model [296 of 301 cases correctly classified, sensitivity of 98.3% (95% CI 96.2% to 99.3%)], specificity was very low [10/327 cases correctly classified as fever; specificity of 3.1% (95% CI 1.7% to 5.5%)].

Analysis of all previously reported signatures using linear discriminant analysis

To allow for a fair and comparable validation of all six transcriptomic signatures in our cohort (i.e. including those in which the published method of diagnostic assessment was not reproducible),18,37 we applied a LDA approach to assess the accuracy of each signature to each of the predefined comparisons we selected for validation analyses. Furthermore, this approach determined the optimal performance of each gene set from the selected signatures within our cohort, without the introduction of learning bias. Thus, the findings using this approach should be reproducible in a new, similarly designed cohort.

Assessment in the full cohort

For the distribution of these signatures by probability as calculated by LDA in the full cohort, and diagnostic performance for detecting all TB cases (including stratification by HIV infection status), see Figure 19 in Appendix 5.

Interestingly, in contrast to the score values, where the Anderson et al. 36 DRS underperformed compared with the Kaforou et al. 20 and Sweeney et al. 39 signatures (although no formal statistical comparison was made), the six signatures performed with similar accuracy when assessed using the LDA approach, with AUCs between 0.81 and 0.84 for detecting all TB cases in the full cohort. The accuracy marginally increased to AUC = 0.85 when HIV+ cases were excluded. The performance in the HIV+ population could be assessed only for these signatures using this method,37–39 as the other signatures used a larger set of genes and, in all cases, the performance was poor and highly variable, with no significant differences between probability scores.

Performance in detecting culture-confirmed or highly probable tuberculosis

The performance of all six signatures at detecting either culture-confirmed (Dosanjh category 1) or highly probable (Dosanjh category 2) TB was assessed using the LDA approach (see Table 30 in Appendix 5). For detecting culture-confirmed TB, the performance of all six signatures was comparable, with AUCs between 0.84 and 0.87. However, for detecting highly probable TB, there was a wide range in performance, with the Sweeney et al. 39 signature providing the highest test accuracy (AUC = 0.80).

Performance in detecting either pulmonary tuberculosis or extrapulmonary tuberculosis

When the six signatures were compared using LDA for the detection of either PTB or EPTB, comparable test accuracies were found for all signatures, with detection of PTB being marginally more accurate (maximum AUC = 0.86) than detection of EPTB (maximum AUC = 0.85) (see Table 32 in Appendix 5).

Derivation of novel signatures

To address our secondary objective to derive novel transcriptomic signatures for the detection of TB, we plan to split the validation cohort into a training set and a test set (80% : 20% split), and to use a variable selection method to identify our candidate genes within the training set, train them and finally test their performance on the remaining 20%. We are in the process of carrying out this analysis, but it did not fit within the scope of this report and will therefore be published elsewhere.

Discussion

In this study, we have shown that none of the previously identified transcriptomic signatures for detecting active TB is sufficiently accurate for the detection of all TB in this clinically relevant cohort. Indeed, the signatures are not sufficiently accurate to detect any of the major subgroups of TB, even when considering the culture-confirmed TB cases alone, which are typically the most straightforward to identify. Their performance was much lower than previously reported, which probably reflects differences in study cohorts, as most of the previous studies did not include the full spectrum of TB types and their differential diagnoses.

Assessment of the previously proposed signatures in an unbiased and reproducible way by using summary scores20,36,39 or a SVML model as previously reported,38 or by applying a LDA approach, allowed us to assess performance in key patient groups in our validation cohort. Although diagnostic performance among culture-confirmed TB cases was greater than for highly probable TB cases, neither group could be accurately distinguished from the OD group. For highly probable TB, the Sweeney et al. 39 signature performed better than other published signatures, although only marginally so, and no statistical comparison was made. It is of note that the Anderson et al. 36 TBS underperformed when compared with the Kaforou et al. 20 DRS, whereas when the signatures were both assessed with LDA their performance was comparable. This probably reflects the nature of the derivation cohorts on which the score values were derived, with Anderson et al. 36 deriving the signature from a paediatric cohort. 36

However, as there is currently no test for rapidly ruling in the highly probable TB group, it is still possible that a whole-blood transcriptomic signature test may be useful for assessment of TB in those cases where a negative microbiological test result is obtained. The prevalence of highly probable TB within the culture-unconfirmed cohort from the IDEA study was found to be 17.5%. Thus, we are able to assess the predictive performance of transcriptomic tests for highly probable TB in those with a negative culture test for TB.

The best-performing, previously identified signature was the Sweeney et al. 39 signature, which gave an AUC of 0.80 (95% CI 0.74 to 0.85) for detecting highly probable TB cases when assessed using LDA on the full cohort, with a sensitivity of 82.1% and specificity of 73.8% when a balanced cut-off point was selected. The scenario in which such a test would be useful is in a TB suspect for whom the culture (or other microbiological test, e.g. Xpert^® MTB/RIF Ultra) is negative or not available. The prevalence of highly probable TB in the culture-unconfirmed TB suspects recruited into the IDEA study was 17.5%. Using this prevalence value, the Sweeney et al. 39 gene signature for highly probable TB would provide a poor PPV of 0.4, but a very high NPV of 0.95. Therefore, it could be that such a transcriptomic signature, if developed into a simple and cheap rapid blood test and validated in new cohorts, could provide a sufficiently high NPV to be considered as a useful rule-out test in the microbiologically unconfirmed population. Furthermore, in settings with higher prevalence rates of highly probable TB, the PPV of these tests would be higher.

A rapid test with high diagnostic accuracy would be particularly helpful in the context of patients who are smear negative on initial assessment for TB, as culture takes a long time to give a definitive result. We present data on the performance of the score signatures20,36,39 in this subcohort of patients; both the Sweeney et al. 39 and the Kaforou et al. 20 signatures provided reasonably high NPV (89–90%). If this result is validated in other studies, this may provide clinicians with sufficient reason to withhold treatment (or at least delay it until culture results are available) in smear-negative TB patients who test negative with these signatures.

The performance of all signatures was very poor within the HIV+ population in this cohort. This suggests that HIV co-infection masks the transcriptomic signal that differentiates TB from OD in the wider population. However, as the number of HIV+ participants in our cohort was small, this study was likely to be underpowered to validate performance within this population. Nevertheless, it is intriguing that others have shown that their identified signatures can perform with comparable test accuracy for detecting TB from OD in HIV+ and HIV– populations. 20,39

Our data suggest that the performance of signatures for distinguishing TB from OD was compromised by sarcoidosis, as well as by HIV co-infection. If sarcoidosis could be excluded early in the diagnostic work-up of active TB, then the performance of transcriptomic signatures would improve and may potentially be of use in specific patient subgroups.

Our transcriptomic validation substudy has some limitations. We used microarray to determine the transcriptional profile of the whole-blood samples used in this cohort, and this technology has now more or less been replaced by RNA sequencing. Nevertheless, as the previously published signatures were all generated by microarray, this was the appropriate platform to use. It is generally accepted that clinical diagnostic scores overdiagnose TB in children, but to what extent this occurs in adult TB is unknown. 44 It is possible that the presence of patients without TB in the highly probable group negatively affects the transcriptomic signatures’ performance. Thus, these signatures could still be clinically useful for paucibacillary (i.e. smear negative and Xpert negative) if these cases are removed from the analysis. In future studies, we aim to use this novel data set to derive and test new signatures, with a particular focus on identifying signatures with high diagnostic performance for smear-negative, culture-negative and EPTB cases of TB.

The suboptimal performance of the published signatures and the newly derived signatures highlights the unmet need in TB diagnosis, which is a rapid, accurate and generalisable test of the full spectrum of TB, including culture-negative TB. Although a similar cohort has been used to derive signatures of paediatric TB,36 our study is the first one to address it in a large clinical adult TB cohort.

Introduction

Serum protein biomarkers have the potential to be rapid, cheap and easy-to-use tests to rule in or rule out active TB disease. The ‘proteomic signatures’ measure the changes in individual proteins found in the serum as a result of the host’s response to disease. Many of the changes observed are not unique to one condition, as they reflect overlapping strategies to combat or adapt to disease processes. However, certain proteins, when used in combination, can provide a high level of diagnostic accuracy when applied in the correct clinical situation. Therefore, the cohort used for selection and testing of proteomic signatures needs to be appropriate to the clinical question and condition for which it is being developed.

Methods

Results

A cohort of 171 samples was selected randomly from the eligible HIV individuals within the full IDEA study cohort. SELDI-TOF mass spectrometry was performed on a total of 90 serum samples. This was the maximum number achievable given the limited number of SELDI-TOF chips available (see Chapter 2, Issues and problems). LC–MS analysis was performed on the full cohort of 171 samples; the 90 samples used in SELDI-TOF were a direct subset of the 171 individuals eligible for analysis, allowing for a bridging assessment between the two technologies (Figure 4). During experimentation and analysis, samples were excluded if consent was withdrawn, if the patient was found to have not been followed up by clinical assessment, or if errors occurred in experimental processing. The final cohorts with complete SELDI-TOF and LC–MS data were n = 86 for SELDI-TOF, and n = 86 and n = 166 for LC–MS (see Figure 4).

FIGURE 4.

Flow diagram for the proteomics validation substudy.

We also performed an additional LC–MS experiment on a larger number of samples (n = 279) using crude sera, which included samples from HIV+ patients. This experiment was sent to an external contractor (FingerPrints Proteomics, URL: http://proteomics.lifesci.dundee.ac.uk/). This data set was restricted to 218 proteins and we were unable to consistently identify and quantitate the protein signatures found in the studies reviewed (see Table 36 in Appendix 6), apart from SERPINA1. This data set was therefore not utilised in biomarker validation results presented below.

Performance of previously published signatures using SELDI-TOF

We selected two SELDI-TOF signatures for validation using this technology in the SELDI-TOF VANTDET study cohort: those described by Sandhu et al. 49 and Hamilton and Levin (unpublished) (see Table 33 in Appendix 6). To allow interstudy comparisons to be made, all SELDI-TOF profiles were acquired for the surfaces utilised in these studies.

The Sandhu et al. 49 signature consisted of two sets of m/z SELDI-TOF peaks, one set (n = 98) derived from crude sera on the CM10 and the second set from 54 m/z peaks from fractionated sera. We acquired SELDI-TOF data on the CM10 surface at pH 4.0 and pH 6.0 in line with these signatures; however, we found that only 2 out of 54 and 2 out of 98 peaks matched between the profiles of Sandhu et al. 49 and our spectra. Based on this, we were unable to validate the Sandhu et al. 49 signatures for diagnosis of TB in our cohort.

The signature proposed by Hamilton and Levin (unpublished) is based on the signature of four to six m/z peaks, corresponding to four unique proteins, and working closely with Hamilton and Levin (unpublished) we could detect the same peaks in our SELDI-TOF profiles (see Table 33). With these data we assessed the classification accuracy in the VANTDET study cohort (n = 86) using the DRS methodology they propose (see Methods). This resulted in an AUC = 0.66 (95% CI 0.54 to 0.77) in the full cohort when comparing all TB (Dosanjh categories 1 and 2, n = 45) to OD (Dosanjh category 4, n = 41). The AUC improved if only culture-confirmed TB (Dosanjh category 1, n = 33) was included in the TB disease group, providing an AUC = 0.74 (95% CI 0.63 to 0.86), whereas no classification was observed for detection of highly probable TB (Dosanjh category 2, n = 12) from OD patients (AUC = 0.58, 95% CI 0.4 to 0.76) (Figure 5).

FIGURE 5.

Performance of the four-protein signature (Hamilton and Levin, unpublished) on SELDI-TOF, for the classification of TB in the VANTDET SELDI-TOF substudy cohort (n = 86). The DRS for the four target proteins was calculated and assessed to diagnostic performance using ROC curve analysis. The performance for classifying all TB (Dosanjh categories 1 and 2 combined, n = 45, black line), culture-confirmed TB (Dosanjh category 1, n = 33, green line) and highly probable TB (Dosanjh category 2, n = 12, blue line) against all OD patients (n = 41) is displayed.

Performance of previously published signatures using liquid chromatography–mass spectrometry

Next, we asked if any of the key published and unpublished signatures [i.e. the signatures identified by Hamilton and Levin (unpublished), Song et al. ,51 Xu et al. ,52 Achkar et al. 21 and De Groote et al. 19] could provide accurate disease classification for active TB using the LC–MS platform. To do this, we utilised the quantitative LC–MS data set generated on immunodepleted sera from 166 individuals, extracting specific quantitative data for the protein signatures within each signature. The data set consisted of spectra collected from immunodepleted sera digested with trypsin and labelled by TMT-10 plex labelling (see Methods). Samples were fractionated by strong anion exchange chromatography into six fractions and each fraction was analysed by LC–MS.

Using the software MaxQuant we identified 8773 peptide identifications mapping to 720 proteins, each with a quantitative value in at least one patient, and 380 proteins with a quantitative value for all patients. Imputation was required for 18% of the proteins used for the discovery of novel signatures (42/228). Overall, the imputation rate for OD samples (n = 84) was 0.3% of all of the reported protein abundances. Similarly, for TB samples (n = 82) the proportion of imputed values was 0.1%.

Detection of key proteins from published signatures

All proteins from the four key publications [Hamilton and Levin (unpublished), Song et al. ,51 Xu et al. ,52 Achkar et al. 21] were detected in our depleted LC–MS data set and could therefore be validated within our cohort of 166 patients. Unfortunately, we could not validate the De Groote et al. 19 signature within this data set, as only four of the six proteins could be detected, and two of these had > 50% missing values. For those signatures we could validate, we first assessed the abundance of the proteins in the key patient groups. The average of the relative abundance (log₂ ratio) and statistical analysis of the proteins is shown in Table 35 in Appendix 6 for the key patient groups. When comparing all TB cases with ODs, the majority of proteins (10/18) were found to be significantly different between these groups, this number fell to 5 out of 18 if only highly probable TB (Dosanjh category 2) cases were considered [P08294 from the Xu et al. 52 signature, Levin 1 and Levin 2 from the Hamilton and Levin (unpublished) signature, and P02654 and P49908 from the Achkar et al. 21 signature] (see Table 35).

Validation of diagnostic performance of serum protein signatures using liquid chromatography–mass spectrometry

To assess whether or not these protein signatures could be used to classify all TB cases from OD, we used LDA (Figure 6) to test the performance of the signatures in separate diagnostic groups, with the exception of the Song et al. 51 signature, for which only one protein (P01009 or SERPINA1) is used51 and the raw abundance data can be used for validation purposes (see Figure 21).

FIGURE 6.

The diagnostic performance of the Xu et al.,52 Hamilton and Levin (unpublished) and Achkar et al. 21 protein signatures on the LC–MS depleted data set (n = 166) using LDA. The diagnostic performance is presented for discrimination of TB vs. OD, discrimination of Dosanjh category 1 vs. OD (Dosanjh category 4) and discrimination of Dosanjh category 2 vs. OD (Dosanjh category 4).

All four signatures performed similarly, providing an AUC = 0.62–0.67 for detection of all TB cases within the cohort. The AUC values, as well as the sensitivity and specificity scores, can be found in Appendix 6 (see Table 37) and Appendix 7 (see Table 38). We further assessed their performance for the classification of either culture-confirmed or highly probable active TB (i.e. Dosanjh categories 1 and 2, respectively) against all OD for culture-confirmed TB; the best classification accuracy was achieved with the Xu et al. 52 and Achkar et al. 21 signatures, providing AUCs = 0.70 (see Table 39). All of the signatures gave a poor performance for detecting highly probable TB, with no significant discriminatory ability (AUC = 0.53–0.59). The performance of all signatures for diagnosing PTB was greater than that for EPTB, for which no discriminatory ability was achieved by any of the published signatures (see Table 37 in Appendix 6 and Table 39 in Appendix 7). Thus, the subgroup in which all signatures performed optimally in VANTDET was the culture-confirmed TB group.

Removal of sarcoidosis from other disease

It is known from our own studies, and those by others, that the serum proteomic profiles of patients with sarcoidosis are very similar to those of patients with active TB. We therefore carried out a LDA on the detection of all TB, as well as stratifying by Dosanjh category, when patients with sarcoidosis were removed from the OD cohort. The results from these analyses are shown in Appendix 8 (see Tables 40 and 41). For the detection of all TB, the performance of all of the published signatures improved slightly. However, when the performance in key subgroups was assessed, the performances of the signatures changed very little, or even reduced in diagnostic accuracy.

Discussion

The focus of this substudy of VANTDET was to test the diagnostic accuracy of previously identified protein signatures that occur in the serum of patients with active TB. To do this we reviewed and tested previously published proteomic signatures using the two major technological platforms, SELDI-TOF and LC–MS.

Our results indicate that diagnosis of TB cases in UK hospitals is not possible using currently identified proteomic signatures in SELDI-TOF or LC–MS technology. We used ROC curve analysis to indicate diagnostic accuracy for the binary classification of TB or OD. Applying this, none of the signatures tested met the GO/NO-GO threshold (AUC > 0.85) outlined in the management plan for this proteomics substudy of the VANTDET study. The best-performing signature was described by Achkar et al. 21 and resulted in an AUC = 0.68 when measured by LC–MS and analysed by LDA. Using SELDI-TOF, the Hamilton and Levin (unpublished) signature performed with an AUC = 0.66 when analysed by calculating the DRS. The classification accuracy was reduced compared with that reported by our collaborators in their cohort of TB and OD patients (AUC = 0.85). The other SELDI-TOF signature developed by Sandhu et al. 49 was not measurable because of poor interlaboratory reproducibility. All signatures improved their classification accuracy if only culture-confirmed active TB cases were considered for diagnosis. Herein, the Hamilton and Levin (unpublished) SELDI-TOF signature gave an AUC = 0.74, which was the optimal diagnostic performance we identified using any proteomic technique; the Xu et al. 52 and Achkar et al. 21 signatures gave a test accuracy performance of AUC = 0.70 using LC–MS as assessed using LDA.

We utilised the Dosanjh categorisation of active TB as our reference standard, in which TB diagnosis is based on both (1) gold standard microbiological culture and (2) clinical diagnosis of culture-negative TB. Approximately 28% of all TB cases recruited were culture negative, and this was driven by the relatively high prevalence of EPTB and other forms of paucibacillary TB cases among the UK cohort. Here, the development of new diagnostics is vital, as standard methods, such as culture or polymerase chain reaction (PCR), perform poorly. Our data show that the host response signatures found in the serum proteome for TB also perform poorly in culture-unconfirmed TB. The markers tested here perform better in culture-conformed (mainly pulmonary) TB potentially as a result of the robust acute-phase response in this type of disease that tracks with increased symptoms and more extensive immunopathology or necrotic disease. A limitation of the previous signatures for TB is their development in cohorts with a high prevalence of pulmonary TB, in which severity of disease at presentation is probably greater. The signature in the Achkar et al. 21 study was the only signature developed from patients in different hospitals, including both culture-negative TB and OD with TB-like symptoms, very similar to the IDEA study cohort utilised in the VANTDET study. The IDEA study12 also recruited prospectively, resulting in a cohort of diverse OD diagnoses (> 100), all classified as one group (Dosanjh category 4). The high diversity among this group, and similarity of symptoms reported by patients from all groups, reflects the real clinical problem, but it hampers any biomarker that is driven by a generalised mechanism in the host response to disease. For example, in diseases consisting of both neutrophil-driven necrotic disease and macrophage recruitment it is likely that the same proteins that are altered will be the same as found in active TB (e.g. S100A8/A9, MMP9). In developing their signatures, both Xu et al. 52 and Song et al. 51 limit selection of control groups to specific diseases, and controls often did not have similar symptoms in the Song et al. 51 study. This artificial selection leads to the development of tests that work well between diseases that are specifically different in the host response orchestrated, but also leads to the selection of markers that reflect only the difference between healthy and sick people. In utilising the IDEA study cohort, the VANTDET study improved on this by creating a design that tested the utility of such markers in a complex environment in which symptomatic people present on a case-by-case basis.

Our study was limited by technological problems in the platforms utilised and the cross-platform compatibility available for protein detection and quantitation. The discontinuation of SELDI-TOF limited the numbers of samples that could be analysed and the poor interlaboratory reproducibility limited us to validating only the four-protein signature developed in-house [developed by Hamilton and Levin (unpublished)]. The transfer of these biomarkers to new methods, LC–MS and ELISA, is troublesome. First, the identification of the proteins was not made until late 2017 and remains unpublished. Second, markers detected by SELDI-TOF do not always correlate accurately to antigenic regions raised during in vivo immune responses, making the development of ELISA difficult. LC–MS detects tryptic peptides for proteins and, therefore, can also not fully recapitulate what is observed in SELDI-TOF, which may be large fragments of proteins. Our goal was to analyse 200 samples from the IDEA study cohort, but this was limited to 90 by SELDI-TOF and 171 by LC–MS, as a result of low levels of consumables left in the UK and limited access to LC–MS equipment internally. Furthermore, the larger data set of crude sera resulted in poor protein coverage and an inability to detect the published signatures. To combat the limited power of our initial investigations, we analysed only HIV– samples. HIV status is known to affect proteomic profile in the serum of people with TB; however, as our cohort consisted of limited numbers of HIV+ (15%) co-infected patients, we decided to test signatures in HIV– patients initially and expand into HIV+ individuals if we achieved an AUC > 0.85. This AUC was never achieved using our approach by LC–MS or SELDI-TOF.

Liquid chromatography–mass spectrometry and SELDI-TOF failed to achieve the minimum requirements set in the VANTDET study for a clinically useful signature. Both technologies are limited by the depth of proteomic coverage and the ease of cross-platform development to a point-of-care test; and this was apparent in our study. Recently, a new technology termed SOMAscan offers an attractive method able to overcome these limitations. De Groote et al. 19 used this technology to discover and test a novel eight-protein signature for TB in the serum of people with PTB compared with other related non-TB diseases. This study was multicentre and prospective, but was limited to culture-confirmed PTB, for which it gave an AUC = 0.87 (95% CI 0.81 to 0.91). Testing of this signature in the VANTDET study using SOMAscan was not possible owing to the limited accessibility to this technology in the public domain at the time of project conception. Furthermore, the proteins comprising this signature were not all detected in either of our LC–MS data sets.

To address our secondary objective of using these proteomic data sets to derive novel signatures that may offer improved accuracy for diagnosing either all TB cases or specific subgroups of TB, we will be applying variable selection methods to identify the proteins providing the best classification accuracy (in both crude and depleted sera samples). We intend to train a prediction model on 60% of the cohort and test the accuracy of the model on the remaining 40% (test set). This work was not within the scope of this report, but will be the logical next step in utilising this novel proteomic data set to derive a clinically useful proteomic signature with potential to be used in the diagnostic work-up of TB.

Our study provided two important conclusions. The previously reported proteomic signatures for detecting TB were not found to be clinically useful in this cohort, probably because of the differences in the complexity of TB/OD patients in the respective cohorts. Therefore, our data from the IDEA study cohort, which does reflect real-life clinical practice, provide a springboard for future studies and an opportunity to derive new signatures that are robust and can pave the way for future validation in similar cohorts.

Introduction

The IGRA tests measure the presence of circulating memory T lymphocytes producing IFNγ in response to the Mtb region of difference 1 (RD1) antigens. In addition to being the gold standard test for LTBI, they are able to detect between 55% and 87% of all active TB cases, depending on the population and test used. 12 However, as IGRAs cannot discriminate between these two states of infection, they lack the specificity to be used as a rule-in test for active TB. To improve on the specificity of such an immune-based test, various groups have explored the Mtb-specific T-cell response with greater complexity than simply measuring production of IFNγ, and have found that, by using flow cytometry to measure phenotypic and functional markers of Mtb-specific T cells, cellular immune signatures have the potential to be used as immune-based tests to discriminate between active and LTBI.

The first group to identify a signature that differed between disease states sufficiently to allow discrimination between active TB and LTBI was Harari et al. ,24 who, in 2011, showed that the functionality of CD4-positive T cells could discriminate between active and LTBI. Specifically, the proportion of Mtb-specific T cells with a polyfunctional profile [IFNγ positive, interleukin 2 (IL-2), positive, tumour necrosis factor alpha (TNF-α) positive] was higher in those with LTBI, whereas the proportion secreting TNF-α only (IFNγ negative, IL-2 negative, TNF-α positive) was higher in those with active TB and could discriminate between the two groups with high accuracy in a training and test cohort. 24

Work from our laboratory exploited the potential diagnostic value of TNFα-only-secreting Mtb-specific T cells while simultaneously improving diagnostic accuracy by combining measurement of T-cell memory/differentiation phenotype (based on cell surface markers) in parallel. Indeed, Pollock et al. 25 demonstrated that measurement of the proportion of TNF-α-only-secreting purified protein derivative (PPD)-specific cluster of differentiation 3 (CD3)- and CD4-positive cells that were CD45RA^–CCR7^–CD127^– (i.e. highly differentiated late-effector T cells) was highly discriminatory between active TB and LTBI (see Table 42 in Appendix 9). The signature had 100% sensitivity and 93% specificity in a test cohort of 34 patients (n = 13, active TB; n = 21, LTBI), of whom 50% were HIV+. 25 Since then, additional cellular immune signatures have been identified that also show promise in their ability to discriminate between active TB and LTBI, including one that combined the proportion of TNF-α-only population (among all cytokine responders as in Harari et al. 24) with the presence of a detectable cluster of differentiation 8 (CD8)- and IFNγ-positive T-cell response into a SCORE value. 58

However, in recent years, the research focus has returned to the CD4-positive IFNγ-producing Mtb-specific population, and several groups have reported using the memory marker CD27 to differntatiate TB from LTBI patients. Different approaches to incorporating this marker into a cellular immune signature have been taken (i.e. by measuring the levels of CD27 as a ratio in relation to the parent population27 or measuring the presence/absence of expression in combination with the memory marker CD45RA). 28 The discrimination between groups was high in these studies, with a specificity of > 90% for active TB for the CD27^–CD45RA^– signature. 28 More recently, the incorporation of activation markers, such as human leucocyte antigen – antigen D related (HLA-DR), in cellular immune signatures has been shown in several studies to provide improved disrimination between these two disease states;26 a finding that has been validated by external groups, including those with/without co-infection with HIV. 59,60 For reference throughout this chapter, the signatures are referred to using either the numerical value or short description, as follows: (1) %TNF-α only,24 (2) % differentiated T effector cells (T_EFF),25 (3) SCORE,58 (4) CD27 MFI (median fluorescence intensity) ratio,27 (5) %CD27–/+CD45RA–28 and (6) %HLA-DR. 26

Flow cytometry is already established in many NHS diagnostic laboratories throughout the UK and is suitable for routine diagnostic use. 61 These signatures therefore warrant prospective validation in a large-scale independent cohort in routine practice. As yet, none of the studies have tested the performance of such signatures in key patient groups that reflect those to present in a clinical setting, such as those with ODs and LTBI, and TB patients with hard-to-diagnose forms of the disease (i.e. culture-unconfirmed TB and/or EPTB). Indeed, patients were not recruited as part of routine clinical practice in any of the previous studies that have been conducted.

Methods

Results

Cohort selection

The nested case–control approach was used to select the final cohort for validation of cellular immune signatures, resulting in a cohort of 184 participants samples being defrosted for the stimulation and flow cytometry staining (Figure 7), with equal numbers of TB and OD/LTBI participants (n = 92).

FIGURE 7.

Study flow chart of patients included in the validation of cellular immune signatures substudy. A nested case–control approach was used to select the patients from the full cohort of IGRA-positive participants who were eligible for analysis in the IDEA study, resulting in a total of 92 patients per group (with the TB patients being randomly selected and all the HIV+ and IGRA-positive patients being selected). Note: this cohort was larger than originally anticipated (i.e. n = 164), as additional IGRA-positive patients were identified since the October 2016 protocol was written. ATB, active tuberculosis.

Diagnostic performance of signatures in full cohort

Here we present the diagnostic performance of six previously identified cellular immune signatures for the detection of TB from OD/LTBI in the IGRA-positive population of TB suspects.

Distribution of signature values in tuberculosis and other disease/latent tuberculosis infection groups

The distribution of signature measurements in the TB and OD/LTBI groups after stimulation with PPD is displayed in Figure 8a and the values after RD1 antigen stimulation are displayed in Appendix 12. There was significant and strong correlation in four of the six signature scores between PPD and RD1 stimulations in those individuals who had both stimulations (see Figure 27; shown only for %HLA-DR).

FIGURE 8.

Performance of published signatures in full cohort, stratified by HIV infection status. (a) Dot plots scoring the signature values after PPD stimulation in TB suspects, stratified by diagnosis (TB or OD) and HIV infection status (positive or negative); and (b) ROC curves demonstrating diagnostic performance of signatures in the detection of active TB cases within the combined HIV+/– population, HIV+ alone and HIV– alone.

Performance characteristics

The performance of each of the signatures for discriminating all TB from those with OD/LTBI is presented in Figure 8 and Table 12 for the PPD stimulation, stratified by HIV infection status.

TABLE 12 - Performance of published cellular immune signature in diagnosing all TB, stratified by HIV infection status

–, negative; –/+, negative and positive; +, positive.

Point at which sensitivity = specificity.

Setting specificity to > 90%.

Note

For calculation of PPV and NPV, the following prevalence values were used: for the HIV– and +/– populations, the prevalence of TB in the T-SPOT. TB-positive population was 76.7%; for the HIV+ population, the prevalence of TB in the T-SPOT. TB-positive population was 43.6%.

Signature, reference	HIV	TB, n	OD/LTBI, n	AUC, %(95% CI)	Cut-off point	Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV	NPV
%TNF-α-only24	–	72	72	0.57 (0.47 to 0.66)	< 29.3a	54.2 (42 to 66)	54.2 (42 to 66)	0.80	0.26
					< 18.7b	23.6 (14.4 to 35.1)	90.3 (81 to 96)	0.89	0.26
	+	9	3	0.83 (0.59 to 1.07)	< 49.0a	77.8 (40 to 97.2)	66.7 (9.4 to 99.2)	0.64	0.80
					< 44.2b	66.7 (29.9 to 92.5)	100 (29.4 to 100)	1.0	0.80
	–/+	81	75	0.56 (0.47 to 0.65)	< 29.8a	55.6 (44.1 to 66.6)	56 (44.1 to 67.5)	0.81	0.28
					< 18.7b	23.46 (14.8 to 34.2)	90.7 (81.7 to 96.2)	0.89	0.26
%TEFF25	–	72	66	0.86 (0.79 to 0.92)	> 27.6a	76.4 (64.9 to 85.6)	77.3 (65.3 to 86.7)	0.92	0.50
					> 37.2b	56.9 (44.73 to 68.6)	90.9 (81.3 to 96.6)	0.95	0.39
	+	8	3	0.58 (0.25 to 0.92)	< 59.3a	50 (15.7 to 84.3)	66.7 (9.4 to 99.2)	0.54	0.63
					< 53.7b	50 (15.7 to 84.3)	100 (29.4 to 100)	1.0	0.72
	–/+	80	69	0.83 (0.76 to 0.90)	> 28.1a	77.5 (66.8 to 86.1)	76.8 (65.1 to 86.1)	0.92	0.51
					> 49.4b	36.3 (25.8 to 47.8)	91.3 (82.0 to 96.7)	0.93	0.30
SCORE58	–	72	72	0.58 (0.49 to 0.68)	< 3.43a	52.8 (40.7 to 64.7)	52.8 (40.7 to 64.7)	0.79	0.25
					< 1.7b	23.6 (14.4 to 35.1)	90.3 (81 to 96)	0.89	0.26
	+	9	3	0.91 (0.73 to 1.1)	< 5.2a	77.8 (40 to 97.2)	66.7 (9.4 to 99.2)	0.64	0.85
					< 5.0b	77.8 (40 to 97.2)	100 (29.2 to 100)	1.0	0.85
	–/+	81	75	0.58 (0.49 to 0.67)	< 3.5a	51.9 (40.5 to 63.1)	52 (40.2 to 63.7)	0.78	0.25
					< 1.7b	23.5 (14.8 to 34.2)	90.7 (81.7 to 96.2)	0.89	0.26
CD27 MFI ratio27	–	71	69	0.78 (0.70 to 0.86)	> 9.5a	74.7 (62.9 to 84.2)	75.4 (63.5 to 85)	0.91	0.47
					> 25b	32.4 (21.8 to 44.6)	91.3 (82 to 96.7)	0.92	0.29
	+	9	5	0.53 (0.19 to 0.88)	< 7.2a	55.6 (21.2 to 86.3)	60 (14.7 to 94.7)	0.52	0.64
					< 4.3b	44.4 (13.7 to 78.9)	80 (28.4 to 99.5)	0.63	0.65
	–/+	80	74	0.76 (0.68 to 0.84)	> 9.3a	72.5 (61.4 to 81.9)	73 (61.4 to 82.7)	0.90	0.45
					> 25.0b	31.3 (21.4 to 42.6)	90.5 (81.5 to 96.1)	0.92	0.29
%CD27–CD45RA–28	–	71	69	0.82 (0.75 to 0.89)	> 60.1a	73.2 (61.4 to 83.1)	72.5 (60.4 to 82.5)	0.90	0.45
					> 76.5b	39.4 (28 to 51.8)	91.3 (82 to 96.7)	0.94	0.31
	+	9	5	0.56 (0.19 to 0.92)	> 69.1a	44.4 (13.7 to 78.8)	40 (5.3 to 85.3)	0.36	0.48
					> 86.8b	11.1 (0.28 to 48.3)	100 (47.8 to 100)	1.0	0.59
	–/+	80	74	0.80 (0.73 to 0.87)	> 60.7a	72.5 (61.4 to 81.9)	73 (61.4 to 82.7)	0.90	0.45
					> 76.6b	38.8 (28.1 to 50.3)	90.5 (81.5 to 96.1)	0.93	0.31
%HLA-DR26	–	71	69	0.90 (0.84 to 0.95)	> 41.5a	80.3 (69.1 to 88.8)	81.2 (70 to 89.6)	0.93	0.56
					> 49.6b	70.4 (58.4 to 80.7)	91.3 (82 to 96.7)	0.96	0.48
	+	9	5	0.91 (0.76 to 1.1)	> 75.0a	77.8 (40 to 97.2)	80 (28.4 to 99.5)	0.75	0.82
					> 81.3b	77.8 (40 to 97.2)	100 (47.8 to 100)	1.0	0.85
	–/+	80	74	0.89 (0.84 to 0.94)	> 42.2a	80.3 (69.9 to 88.3)	79.2 (68.5 to 87.6)	0.93	0.55
					> 54.3b	64.2 (52.8 to 74.6)	90.9 (82.2 to 96.3)	0.96	0.44

The diagnostic accuracy of the discriminatory signatures was optimal for the culture-confirmed TB group (with the %HLA-DR signature achieving the highest AUC of 0.92, 95% CI 0.88 to 0.97), although the performance for the culture-unconfirmed group (Dosanjh category 2) was still reasonable for this hard-to diagnose group of patients (Table 13). To evaluate if the Dosanjh category 4 subcategory influences the diagnostic accuracy of the signatures, we evaluated the performance of detecting all TB using only the Dosanjh subcategories 4B or 4C patients in the TB-excluded (OD/LTBI) group. Interestingly, the performance of the cellular immune signatures was comparable, suggesting that likelihood and risk of LTBI in the OD/LTBI group does not influence test accuracy of cellular immune signatures (see Table 52 in Appendix 13).

TABLE 13 - Performance of cellular immune signatures in detecting TB, stratified by Dosanjh criteria

Point at which sensitivity = specificity.

Setting specificity to > 90%.

Note

For calculating PPV and NPV, the prevalence of Dosanjh 1 was 57.1%, and the prevalence of Dosanjh 2 was 19.6%.

Comparison	Signature	TB, n	OD/LTBI, n	AUC, %(95% CI)	Cut-off point	Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV	NPV
Dosanjh category 1 vs. Dosanjh category 4	%TNF-α-only24	47	72	0.58 (0.47 to 0.68)	< 28.90a	55.3 (40.1 to 69.8)	55.6 (43.4 to 67.3)	0.62	0.48
					< 18.70b	23.4 (12.3 to 38.0)	90.3 (81.0 to 96.0)	0.76	0.47
	%TEFF25	48	66	0.88 (0.82 to 0.94)	> 29.30a	81.3 (67.4 to 91.1)	81.8 (70.4 to 90.2)	0.86	0.77
					> 42.20b	47.9 (33.3 to 62.8)	90.9 (81.3 to 96.6)	0.88	0.57
	SCORE58	47	72	0.58 (0.48 to 0.69)	< 3.46a	51.1 (36.1 to 65.9)	51.4 (39.3 to 63.4)	0.58	0.44
					< 1.70b	23.4 (12.3 to 38.0)	90.3 (81.0 to 96.0)	0.76	0.47
	CD27 MFI ratio27	46	69	0.81 (0.73 to 0.89)	> 10.4a	73.9 (58.9 to 85.7)	75.4 (63.5 to 85.0)	0.80	0.68
					> 25.0b	39.1 (25.1 to 54.6)	91.3 (82.0 to 96.7)	0.86	0.53
	%CD27–CD45RA–28	46	69	0.84 (0.77 to 0.92)	> 61.3a	78.3 (63.6 to 89.1)	78.3 (66.7 to 87.3)	0.83	0.73
					> 76.5b	47.8 (32.9 to 63.1)	91.3 (82.0 to 96.7)	0.88	0.57
	%HLA-DR26	46	69	0.92 (0.88 to 0.97)	> 44.2a	82.6 (68.6 to 92.2)	82.6 (71.6 to 90.7)	0.86	0.78
					> 49.6b	78.3 (63.6 to 89.1)	91.3 (82.0 to 96.7)	0.92	0.76
Dosanjh category 2 vs. Dosanjh category 4	%TNF-α-only24	25	72	0.55 (0.41 to 0.69)	< 30.3a	52.0 (31.3 to 72.2)	52.8 (40.7 to 64.7)	0.21	0.82
					< 18.7b	24.0 (9.4 to 45.1)	90.3 (81.0 to 96.0)	0.38	0.83
	%TEFF25	24	66	0.81 (0.71 to 0.90)	> 23.0a	70.8 (48.9 to 87.4)	72.7 (60.4 to 83.0)	0.39	0.91
					> 37.2b	45.8 (25.6 to 67.2)	90.9 (81.3 to 96.6)	0.55	0.87
	SCORE58	25	72	0.58 (0.45 to 0.72)	< 3.3a	56.0 (34.9 to 75.6)	56.9 (44.7 to 68.6)	0.24	0.84
					< 1.6b	24.0 (9.4 to 45.1)	90.3 (81.0 to 96.0)	0.38	0.83
	CD27 MFI ratio27	25	69	0.73 (0.62 to 0.84)	> 7.5a	68.0 (46.5 to 85.1)	68.1 (55.8 to 78.8)	0.34	0.90
					> 25.1b	20.0 (6.8 to 40.7)	91.3 (82.0 to 96.7)	0.36	0.82
	%CD27–CD45RA–28	25	69	0.77 (0.67 to 0.87)	> 57.4a	72.0 (50.6 to 87.9)	71.0 (58.8 to 81.3)	0.37	0.91
					> 76.5b	24.0 (9.4 to 45.1)	91.3 (82.0 to 86.7)	0.40	0.83
	%HLA-DR26	25	69	0.84 (0.76 to 0.93)	> 38.1a	76.0 (54.9 to 90.6)	76.8 (65.1 to 86.1)	0.44	0.93
					> 49.6b	56.0 (34.9 to 75.6)	91.3 (82.0 to 96.7)	0.61	0.89

The next best-performing signatures in the HIV– and HIV–/+ combined subgroups were, in order of decreasing accuracy, the %T_EFF (of CD4-positive TNF-α-only cells) proposed by Pollock et al. 25 (AUC = 0.86 in HIV–, 95% CI 0.79 to 0.92), %CD27–CD45RA– proposed by Petruccioli et al. 28 (AUC = 0.82 in HIV–, 95% CI 0.75 to 0.89) and CD27 MFI ratio proposed by Portevin et al. 27 (AUC = 0.78, 95% CI 0.70 to 0.86), all of which showed highly significant discrimination between TB and OD/LTBI above the level of AUC = 0.5 (p < 0.0001). However, none of these signatures performed well in the HIV+ population, with AUCs ≤ 0.58 in all cases (see Table 12). No statistical analysis to compare diagnostic accuracy between the signatures was performed.

The two signatures that measured proportions of functional Mtb-specific subsets only, and not phenotype of Mtb-specific cells, were the %TNF-α-only (of CD4-positive and cytokine-positive) signature proposed by Harari et al. 24 and the SCORE signature proposed by Rozot et al. 58 These did not provide significant discrimination between TB groups in the HIV– or HIV+/– populations, with AUCs of ≤ 0.57 that were not significant (p > 0.05). Owing to the small numbers of responders in the HIV– cohort, this population was excluded from subsequent subcohort analyses.

Discussion

Here we present the first validation of previously identified cellular immune signatures for the detection of active TB cases in a cohort of TB suspects recruited as part of routine clinical practice. The immunological nature of these assays, with the signatures being measurable only when a detectable Mtb-specific CD4-positive response can be characterised, confines the eligible patient population to those who respond to Mtb antigens (i.e. the IGRA-positive individuals) and, therefore, the challenge for these assays is to discriminate between active TB and OD/LTBI. We found that, of the six signatures validated in this cohort, only those that measured incorporated phenotypic markers in the flow cytometry signature (i.e. %T_EFF, CD27 expression or %HLA-DR25–27) showed discriminatory ability, whereas the signatures measuring functionality did not differentiate between TB and OD/LTBI.

The best-performing signature was found to be the %HLA-DR signature,26 which has recently been validated in several cohorts and shown to have high test accuracy for differentiating TB from healthy LTBI. 26,59,60 This is the first study to demonstrate its diagnostic performance in a cohort in which the LTBI group also had OD and in which all patients were recruited with suspected TB. The performance we present here (AUC = 0.90) is not as high as previously found (AUCs 0.92–1.0). However, when only culture-confirmed TB among HIV– patients was considered, the diagnostic accuracy of this signature was comparable to that reported by Riou et al. 60 (also AUC = 0.92). Therefore, the overall lower performance we present here for detection of all TB using previously described cellular immune signatures probably reflects the inclusion of paucibacillary and hard-to-diagnose TB patients, particularly those with culture-unconfirmed TB.

The %T_EFF25 signature also performed with reasonably high accuracy for detection of all TB, providing an AUC of 0.86 for detecting all TB in the HIV– population. Unlike in the original paper, in which performance was high in a mixed cohort of HIV+ and HIV– patients, in this study the signature could not discriminate between TB and OD/LTBI in the HIV+ population.

Both the %HLA-DR26 and %T_EFF25 signatures could be useful in the setting of IGRA-positive TB suspects, as, in this scenario, they are able to provide a high PPV for all TB (0.95 and 0.95, respectively) and for culture-confirmed TB (0.92 and 0.88, respectively). Therefore, on initial screening of an IGRA-positive suspect, a positive result in these tests could provide clinicians with sufficient evidence (i.e. ≈90% probability) to start treatment immediately, without any need to wait for culture confirmation. This is the testing algorithm in which these assays could be useful. In future studies, we wish to determine these signatures in the smear-negative group of TB patients. Importantly, we also assessed the performance of these two signatures in the smear-negative subgroup within the cohort. These signatures were found to provide very high PPVs for all TB and culture-confirmed TB within the T-SPOT. TB-positive smear-negative population of TB suspects. These patients usually require several more investigations to inform treatment decision, or else treatment will be started without confidence in the correct diagnosis having been made. For culture-confirmed TB, the prevalence of which in the smear-negative T-SPOT. TB-positive population was 59%, the PPVs of 0.87 and 0.88 raised the pre-test probability of TB by 28–29% (i.e. from 59% pre-test probability to 87–88% post-test probability). The increase in the pre-test probability for all TB cases in this specific population was 13% (from 79% to a PPV of 94%), demonstrating that a positive test result with either of these assays would provide a rapid rule-in test, facilitating an accelerated decision to treat. A positive result in this setting would thus provide strong support for a diagnosis of TB and a decision to treat.

We found in this study that the signatures measuring polyfunctional profiles (%TNF-α-only24 and SCORE58) do not appear to be able to discriminate between TB and OD/LTBI groups. Several research groups have measured similar profiles, and it appears that these signatures are not reproducible across different studies. Day et al. 63 found that there were differences between LTBI and smear-negative and smear-positive TB, not with TNF-α-only CD4-positive responses, but with IFNγ+TNF-α+IL-2– and IL-2 responses.

Incorporating measurement of CD27 showed reasonable test accuracy in our cohort, but only in the HIV– population, and to a lesser extent than the %HLA-DR26 and%T_EFF25 signatures (although statistical analyses to compare tests were not performed).

There are some limitations. Some of the PBMC stimulations did not result in detectable cytokine responses, meaning that they were excluded from the diagnostic performance analysis. Therefore, the performance may not be fully reflected in the analysis we present and, in practice, these tests may result in a number of failed or indeterminate results.

This is the largest validation of cellular immune signatures in a cohort of TB and LTBI patients and the first to test their performance in a cohort of TB suspects. We have demonstrated that cellular immune signatures can provide high test accuracy and may be useful for a rapid blood-based test for TB in the IGRA-positive population.

Introduction

Interferon gamma release assays are the recommended immunological assays to detect Mtb infection. There are two commercially available IGRAs, both of which measure IFNγ production in response to RD1 antigens (the T-SPOT. TB, which uses the T-SPOT. TB platform, and the QFT-GIT, which uses ELISA). IGRAs are unable to discriminate between active disease and LTBI and, as such, are unsuitable as tests to rule in active TB (because of poor specificity). Their sensitivity for detecting active TB cases is reasonably high, leading to the possibility that these tests could be used as a rule-out test for active TB. Ideally, such a test would provide > 95% sensitivity for all TB cases for it to be clinically useful. The IDEA study demonstrated that the T-SPOT. TB tests provided optimal accuracy (as compared with QFT-GIT), but that neither test in their current format provides sufficient sensitivity (between 67.3% and 82.3%) for detecting all TB12 for this important clinical utility. 11,12 In this substudy we explored two candidate approaches to improve on the sensitivity of immunological tests for active TB.

Methods

Results

Cohort characteristics

The details of the cohort eligible for analysis and the stratification between the full eligible cohort and GO/NO-GO cohort are given in Figure 9.

FIGURE 9.

Study flow diagram for the validation of the candidate rule-out tests. ATB, active tuberculosis.

From the 100 individuals selected to be part of the GO/NO-GO cohort, three were not found in the biobank and an additional three samples were found, subsequently, to be ineligible (having a previous history of TB), leaving a total of 94 in the full cohort of assay individuals.

The demographic and clinical characteristics of the final cohort are shown in Table 15. Further details on the severity and type of TB disease, or OD, within the full cohort are given in Appendix 14 (see Tables 54 and 55).

TABLE 15 - Demographic and clinical characteristics for the GO/NO-GO cohort for the assessment of a candidate molecular rule-out test for active TB

N/A, not applicable.

Note

Characteristics were compared across groups using either a Kruskal–Wallis test or a chi-squared test.

Demographic/clinical characteristic	Active TB		OD		p-value
	T-SPOT.TB positive (N = 24)	T-SPOT.TB negative (N = 22)	T-SPOT.TB positive (N = 22)	T-SPOT.TB negative (N = 26)
Age (years), median (range)	33 (19–57)	48.5 (27–81)	43.95 (19–74)	49.19 (21–84)	0.0011
Female, n (%)	12 (50)	7 (31.8)	8 (36)	10 (38)
Ethnic origin, n (%)					N/A
Asian	2 (8.3)	2 (9.1)	1 (4.5)	2 (7.7)
Black	4 (16.6)	0 (0)	3 (13.6)	7 (26.9)
Indian subcontinent	17 (70.8)	13 (59.1)	10 (45.5)	9 (34.6)
Mixed	0 (0)	2 (9.1)	0 (0)	0 (0)
White	1 (4.2)	5 (22.7)	8 (36.4)	8 (30.1)
HIV+, n (%)	1 (4.2)	2 (9.1)	6 (27.3)	1 (3.8)	0.032
TST, n (%)					< 0.0001
Positive	5 (20.8)	9 (40.9)	0 (0)	1 (3.8)
Negative	0 (0)	0 (0)	1 (4.5)	8 (30.1)
Not tested	19 (79.2)	13 (59.1)	21 (95.5)	19 (73.1)
BCG, n (%)
Yes	18 (75)	17 (77.3)	19 (86.4)	22 (84.6)
No	6 (25)	5 (22.7)	3 (13.6)	4 (15.4)
Height (m), median (range)	1.69 (1.52–1.85)	1.69 (1.61–1.8)	1.65 (1.55–1.88)	1.68 (1.52–1.86)	0.3981
Height missing, n (%)	4	10 (45.5)	2	7
Weight (kg), median (range)	61 (42–123.5)	61.5 (47.4–96)	64.5 (45–82.6)	72.3 (48.3–140)	0.2815
Weight missing, n (%)	1	0	0	4
BMI (kg/m2), median (range)	23.1 (17.6–36.9)	22.6 (18.9–31.1)	22.4	24.4 (15.9–43.2)	0.6108
BMI missing, n (%)	4	10	2	8

Validation of molecular rule-out tests

To measure gene expression of all analytes (CXCL10, CXCL9, BATF2 and IFNγ) using qRT-PCR, a minimum of 50 ng of total RNA was needed. Of the 94 samples that were assayed, 13 samples yielded low amounts of RNA (around ≈40 ng); therefore, not all analytes could be measured. In these cases, BATF2 expression was not measured.

Gene expression of target analytes in response to the positive control

As all of the target genes (CXCL9, CXCL10 and BATF2) are known to be up-regulated in response to IFNγ, this was used as the positive control in the assay. Previously, phytohaemagglutinin (PHA) was used in our laboratory for these assays (as is used in T-SPOT. TB and QFT-GIT). However, preliminary optimisation experiments in the VANTDET study revealed that the levels of the target genes were not reliably increased using PHA. This is probably a result of the reduced viability of PBMCs following PHA stimulation, which we confirmed by flow cytometry (data not shown). By plotting the normalised relative expression of target genes in response to ECRR or IFNγ, we were able to select a cut-off point for an acceptable response in the positive control condition. The cut-off points selected were as follows: > 100 for CXCL10, > 100 for CXCL9 and > 10 for BATF2 (as shown in Figure 31 in Appendix 15).

Gene expression in response to Mycobacterium tuberculosis antigens

The results of the PCR detection of target genes in response to the antigen conditions (normalised to unstimulated) are shown in Table 16.

TABLE 16 - Quantitative reverse transcription-polymerase chain reaction detection of CXCL10, CXCL9 and BATF2 in response to Mtb antigens or positive control in the GO/NO-GO cohort of patients selected for validation of candidate molecular rule-out tests

Analyte/measurement	Active TB		OD
	T-SPOT.TB positive (N = 24)	T-SPOT.TB negative (N = 22)	T-SPOT.TB positive (N = 22)	T-SPOT.TB negative (N = 26)
CXCL10
Total for analysis, n (%)	19 (75)	19 (87)	12 (55)	21 (81)
Total failed, n (%)	5 (25)	3 (13)	10 (45)	5 (19)
PCR indeterminate	0	1	0	1
HPRT1 fail	3	2	6	5
Low levels in positive control	2	1	4	0
Fold change ECRR/unstimulated, mean (range)	23,604.50 (116.68–14,0596.57)	9362.95 (8.11–46,681.24)	115,353 (1.45–56,513.26)	1408.53 (1.5–18,053.68)
IFNγ/unstimulated, mean (range)	50,483.2 (224.9–325,447.0)	126,019.7 (829.6–432,419.1)	104,350.1(156.8–384,828.4)	55,180.4 (361.6–381,780.3)
CXCL9
Total for analysis, n (%)	18 (76)	18 (83)	13 (59)	21 (81)
Total failed, n (%)	6 (24)	4 (17)	9 (40.9)	5 (19)
PCR indeterminate	0	1	1	1
HPRT1 fail	3	2	6	5
Low levels in positive control	3	2	2	0
Fold change ECRR/unstimulated, mean (range)	33,280.75 (350.84–120,140.97)	15,049.99 (0.99–74,914.48)	1030.38 (0.19–5146.35)	788.49 (0.23–7952.44)
IFNγ/unstimulated, mean (range)	97,456.2 (301.1–605,633.7)	245,973.5 (1025.6–804,235.3)	171,387.5 (220.2–792,079.3)	123,803.4 (1033.6–795,414.8)
BATF2
Total for analysis, n (%)	19 (80)	14 (66.7)	10 (45.5)	18 (69.2)
Total failed, n (%)	5 (20)	8 (33.3)	12 (54.5)	8 (30.8)
Low RNA concentration	1	4	4	4
PCR indeterminate	0	1	1	0
HPRT1 fail	3	2	6	4
Low levels in positive control	1	1	1	0
Fold change ECRR/unstimulated, mean (range)	33,280.75 (350.84–120,140.97)	15,049.99 (0.99–74,914.48)	42,919.01 (0.19–545,582.52)	788.49 (0.23–7952.44)
IFNγ/unstimulated, mean (range)	97,456.2 (301.1–605,633.7)	245,973.5 (1025.6–804,235.3)	171,387.5 (220.2–792,079.3)	123,803.4 (1033.6–795,414.8)

Of note, a high proportion (15–27%) of the assays resulted in a ‘fail’ result, as a result of a number of factors. The most prevalent reason for assay failure was unreliable detection of the housekeeping gene HPRT1. Individuals with HIV co-infection were responsible for many of the assay failures, particularly for detection of Mtb-specific CXCL10. We also, for comparison, measured the levels of IFNγ in response to the antigens using the same approach. The findings are shown in Table 56 in Appendix 15. The gene expression data for individuals within each group who passed test inclusion criteria for passing the assay are presented in Figure 30 in Appendix 15.

The T-SPOT. TB-positive TB group displayed the highest Mtb-specific expression of all target genes when compared with other groups, with the exception of CXCL10, for which levels were also very high in the T-SPOT. TB-positive OD (i.e. OD/LTBI) group.

Diagnostic performance for detection of active tuberculosis

The diagnostic performance of the target genes for detecting all TB cases compared with all OD is presented in Figure 10. Although obtaining > 95% sensitivity for all active TB is achievable when the cut-off point for fold/change over unstimulated is set at around 10 for CXCL9 and CXCL10, this was coupled with very low specificity (≈ 5%). For BATF2 and IFNγ, the cut-off point corresponding to > 95% sensitivity was < 1, suggesting that it is not achievable to use these analytes to detect this proportion of all TB cases.

FIGURE 10.

Diagnostic performance of candidate qRT-PCR detection of CXCL10, CXCL9, BATF2 or IFNγ in the candidate molecular rule out, as assessed in the GO/NO-GO cohort. The classification of all TB from all OD (either T-SPOT. TB positive/negative for both) using these readouts is shown (n = 2–48 per group depending on the test). The performance of the T-SPOT. TB assay in this cohort is also shown. ATB, active tuberculosis.

When compared with the T-SPOT. TB test within the same cohort, which showed no diagnostic accuracy in this setting owing to the artificial way the participants were selected (AUC of 0.53), all of the analytes showed improved diagnostic accuracy. This suggests that this method of qRT-PCR measurement of cytokines may provide greater sensitivity than the T-SPOT. TB platform, as even IFNγ detection by qRT-PCR performed better than the T-SPOT. TB in detecting active TB patients. However, these ROC curve analyses do not include those individuals for whom the qRT-PCR assay failed.

The best-performing gene for detection of all active TB was CXCL9, which provided an AUC of 0.84 (95% CI 0.74 to 0.94) for detecting all TB cases in this cohort. However, none of the new assays was able to provide 95% sensitivity with an acceptable corresponding specificity (i.e. > 65%) (see Appendix 15, Table 57). In fact, in all cases, the specificity for either TB group, or indeed LTBI, was unexpectedly low, with a wide range of gene expression values in all groups.

As the proportion of T-SPOT. TB-positive OD patients within this artificial cohort was inflated, the true specificity of these tests is likely to be greater in a cohort that better reflects the levels of T-SPOT. TB positivity in the TB and OD groups. To test this, we compared the diagnostic performance of the tests for detecting all TB cases (T-SPOT. TB positive/negative) with their performance for detecting T-SPOT. TB-negative OD cases only (see Figure 32 in Appendix 15). In this setting, the performance of target genes was more like that seen in the T-SPOT. TB test (AUC = 0.76), with CXCL9 again providing the best test accuracy (AUC = 0.86).

We also compared the performance of target genes when only T-SPOT. TB-positive TB and T-SPOT. TB-negative OD were considered. In this setting, none of the tests provided 100% accuracy and, therefore, performed suboptimally to the T-SPOT. TB (see Figure 33 in Appendix 15). The full performance data for these calculations, including sensitivity and specificity values, are shown in Appendix 15.

Discriminating active tuberculosis from latent tuberculosis infection/other disease

To address our secondary objective of assessing the performance of our target genes in discriminating between active TB and LTBI in our cohort, we assessed their performance using ROC analysis comparing TB (T-SPOT. TB positive) and OD (T-SPOT. TB positive) (see Figure 34). In this setting, the T-SPOT. TB provides no discrimination (as results used are either ‘positive’ or ‘negative’ rather than numerical). All of the target genes, including IFNγ, provided some discrimination between action TB and OD/LTBI in this subcohort of patients, with CXCL9 providing the highest AUC of 0.97 (95% CI 0.91 to 1.0).

Discussion

In this substudy we have evaluated two candidate approaches for improving the sensitivity of immune-based tests for active TB: (1) the molecular rule-out test for measuring chemokines from stimulated PBMCs using qRT-PCR; and (2) the measurement of IFNγ and CXCL10 in QFT-GIT supernatants using MSD. Unlike other studies that used a case–control approach and in which patients were often pre-selected for immune assay responsiveness, in this study we have designed a cohort that is deliberately enriched for patients in whom current immune-based tests fail (i.e. those in the TB group who are IGRA negative and those in the OD group who are IGRA positive). This artificial cohort selection approach was used to provide a GO/NO-GO point for the validation of these candidate tests.

We found that CXCL9, CXCL10 and BATF-2 gene expression was upregulated in response to RD1 antigens (ECRR) in TB patients, including in T-SPOT. TB-negative TB patients. We confirmed previous findings23,67 that the levels of CXCL9 and CXCL10 produced by stimulated blood cells are detected at amplified levels compared with those of IFNγ. However, these genes were also upregulated in a large proportion of OD patients in our cohort, which compromised diagnostic performance. This phenomenon was also observed by others, where IGRA-negative controls also displayed production of CXCL10 in response to Mtb antigens,23,67 suggesting potential non-specific production of this chemokine in some tests. Although reasonably high diagnostic accuracy was found (AUC = 0.84 for CXCL9) for all TB in our GO/NO-GO cohort, this did not correspond to a test which could provide > 95% sensitivity and 65% specificity at a given threshold. Therefore, this approach is unlikely to be clinically useful as a rule-out test for all TB. Of all the analytes measured in the molecular rule-out assay approach, CXCL9 provided best diagnostic accuracy for detection of all TB cases from all OD, and for discrimination of TB from OD/LTBI in T-SPOT. TB-positive patients. To our knowledge, this is the first study to test the performance of these chemokines in discriminating between TB and LTBI in a cohort in which the LTBI patients had OD and were suspected of active TB at the point of recruitment. It appears that CXCL9 may be a useful marker for differentiating between these groups. However, it should be noted that statistical analyses to compare diagnostic accuracy (i.e. AUC, sensitivity or specificity values) between analytes were not performed.

We reported a high proportion of assay failures for the candidate molecular rule-out test assay, which were probably a result of poor recovery of some PBMC samples from cryopreservation and into culture. Using either fresh blood or fresh isolated PBMCs for this candidate assay would have been optimal, but we did not have this material available within the VANTDET study. Kasprowicz et al. 23 have already demonstrated that detection of Mtb-specific CXCL9 and CXCL10 using a similar qRT-PCR approach at as little as 50 µl of fresh blood per condition is achievable. Their findings now warrant validation in a cohort of TB suspects, similar to the IDEA study cohort, but using fresh blood samples. We anticipate that fresh samples would result in fewer assays failures and a better performance for detecting all TB cases.

Using supernatants stored from the QFT-GIT tests conducted as part of the IDEA study, we were able to assess the use of the sensitive MSD platform for detection of target analytes and compare the performance of CXCL10 and IFNγ between the two candidate platforms. This way, we could compare the performance (although somewhat indirectly) in detection of these analytes by using either qRT-PCR (for the detection of the transcribed genes) or ELISA (for detection of the proteins). We found comparable performance for detection of the analytes and very similar diagnostic accuracy for detecting TB. However, neither approach was able to provide the required diagnostic accuracy to be used as a rule-out test for all TB and, furthermore, not all IGRA-positive individuals were detected using the new assays. Nevertheless, we were able to demonstrate that MSD is able to detect analytes, such as CXCL10 in plasma, including a wide range of concentrations. Although this is not required for a simple positive/negative immune test, a test for discriminating between disease states, in which differences in quantity of analytes may provide discriminatory power, may benefit from using a MSD platform for quantitation. We plan to investigate this in our future experiments using MSD.

Both of the candidate rule-out test experimental approaches we have evaluated are complex, with multiple steps required (which can introduce significant variability), and requiring technical expertise as well as expensive equipment to perform. This is not in keeping with the WHO’s target product profile for a triage test for TB, which stipulates that such a triage test should take between 5 and 30 minutes to get a result, cost ≈US$1–2 and be achievable using a single device that requires no calibration. 16 However, we propose that in high-income settings, where TB incidence is generally low, an immune-based test providing > 95% sensitivity for all active TB would be clinically useful if available at low cost. Further work is needed to determine the price at which such an assay would be cost-effective.

Introduction

We present methods for assessment of the cost-effectiveness of several novel tests as rule-out and rule-in tests for active TB. That is, in the case of a rule-out test, we consider using novel tests as an initial test, with a negative result indicating that a patient does not have TB, thus accelerating diagnosis of the actual cause of disease in such patients. In the case of a rule-in test, we consider using novel tests as an initial test, with a positive result indicating that a patient does have TB, averting the need for other tests for TB. The use of several tests was compared against current practice, as determined by analysis of patient records. We considered which diagnostic tests were performed, their costs and the time taken between decision points involving each test. The time taken to diagnose, rule in or rule out, TB is a key consideration. Our report adheres to the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. 72

Methods

Decision tree model

We developed a decision tree model to calculate the incremental costs and incremental QALYs of changing from current practice to using a novel test as an initial rule-in or rule-out test. Current practice was determined by analysis of patient records. Adding a rule-out test to the diagnostic pathway introduces additional delays in the diagnosis of active TB in those patients who have the disease, as it introduces an additional step in the pathway. Patients who were not initially diagnosed with active TB have a follow-up consultation after approximately 2 months; those who had a false-negative rule-out test result (i.e. they had TB incorrectly ruled out) can have TB identified at this point. The final diagnostic outcomes were the four categories described in Dosanjh et al.,11 herein referred to as ‘Dosanjh categories’.

The health economic analysis was undertaken from an NHS perspective. No discounting was required as the diagnostic process occurs over a relatively short time period. The time horizon was up to final diagnosis of TB, which was assumed to be < 1 year.

The model contained two levels of uncertainty:

1. individual-level uncertainty – patient records revealed variation in the number and type of tests used for TB diagnosis and time to diagnosis

2. parameter uncertainty – uncertainty in the costs of tests and procedures, and the sensitivity and specificity of the novel tests.

Uncertainty distributions for each are estimated for either uniform or gamma distributions, depending on the form of the data available.

A (balanced) bootstrap sample of TB and non-TB patients was created for each simulation of the decision tree, which retained the TB and non-TB subsample sizes. Individual patient costs and time to diagnosis were jointly sampled to preserve the dependency between them. We sampled across the model parameter distributions to obtain a set of realisations. This was then repeated in a Monte Carlo framework to obtain a sample of 10,000 model runs, which encapsulated the first-order (patient variability) and second-order (parameter) uncertainty in the model outputs. Table 17 summarises the key model parameters.

TABLE 17 - Key model parameters

ATB, active tuberculosis.

Parameter	Symbol	Main model value
Follow-up time for those not diagnosed with TB (days)	T followup	Direct estimate from clinical data set
Cohort ATB prevalence	n+n	Direct estimate from clinical data set, also stratified by HIV status
Active TB QALY loss	q	Gamma(5.427,0.0154)73
Hepatotoxicity as a result of treatment QALY loss	q h	Gamma(65.753,7e-5)73
Nausea as a result of treatment QALY loss	q n	Gamma(109.67,3e-6)73
Current time to diagnosis by TB status	t¯+, t¯−	Direct estimate from clinical data set
Current combined cost of diagnosis by TB status	c¯+, c¯−	Direct estimate from clinical data set

The model was implemented in the statistical programming language R. The key model parameters are presented in Table 17.

An example decision tree representation is given in Figure 11. Equivalent figures for alternative scenarios are given in Appendix 16, Figures 36 and 37.

FIGURE 11.

Decision tree comparing current practice (‘no rule-out test’) with a diagnostic pathway incorporating an initial rule-out test (‘rule-out test’). The ‘standard pathway’ branch represents the range of variation observed in the patient cohort. Probabilities are shown below branches following a circular chance node and costs are below branches following a square decision node. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Results

For decision tree diagrams, please see Appendix 16. The results of the cost-effectiveness analysis for the Sweeney et al. 39 transcriptomic signatures used as a rule-out test are given in Figure 12 and Table 19, and the transcriptomic signatures used as a rule-in test are given in Figure 13 and Table 20.

FIGURE 12.

Cost-effectiveness planes for status quo against transcriptomic rule-out test scenario, with contour lines.

FIGURE 13.

Cost-effectiveness planes for status quo against transcriptomic rule-in test scenario, with contour lines.

Flow cytometry rule-in test

The results of the cost-effectiveness analysis for the %HLA-DR flow cymometry signatures26 to be used as a rule-in test are presented in Figure 14 and Table 21.

FIGURE 14.

Cost-effectiveness planes for status quo against flow cytometry %HLA-DR rule-in test scenario, with contour lines.

TABLE 21 - Cost-effectiveness statistics for flow cytometry rule-in scenario

WTP, willingness to pay.

Scenario	Incremental		Probability cost-effective
	QALY gain	Cost incurred (£)	WTP £20,000/QALY	WTP £30,000/QALY
Total	–0.00014	185	0	0

Molecular rule-out test

The results of the cost-effectiveness analysis for the molecular rule-out test are presented in Figure 15 and Table 22.

FIGURE 15.

Cost-effectiveness planes for status quo against molecular rule-out test scenario, with contour lines.

TABLE 22 - Cost-effectiveness statistics for molecular rule-out scenarios

WTP, willingness to pay.

Scenario	Incremental		Probability cost-effective
	QALY gain	Cost incurred (£)	WTP £20,000/QALY	WTP £30,000/QALY
Total	–0.0010	408	0	0

Discussion

This health economic analysis has demonstrated that the use of those tests that performed best in classifying all TB cases in our VANTDET substudies (i.e. the Sweeney et al. 39 four-gene transcriptomic signature, the %HLA-DR cellular immune signature26 and the CXCL9 qRT-PCR test for ruling out active TB) cannot be considered cost-effective, given the assumption of the modelling. This is due in part to the significant cost of these tests, which, in their current proposed format, are very expensive. Furthermore, the sensitivity and specificity provided by these tests were suboptimal, meaning that they did not improve effectiveness when introduced into the current diagnostic pathway. All tests incurred extra costs and there was a minor reduction in QALYs, probably because of the increased delay in diagnosis resulting from the introduction of new tests into the standard pathway. This was the case even when tests were considered as either rule-in or rule-out tests.

It should be noted that, as all of the technologies and signatures are being investigated in research laboratory settings, and have not begun to be commercialised in any way, the cost estimates had a high degree of uncertainty, which affected the cost-effectiveness analysis. The effectiveness for the %HLA-DR cellular immune assay, for which we propose using in the context of IGRA-positive TB suspects only, the cost-effectiveness plane demonstrated reasonable effectiveness, but with a very high cost differential. If the test accuracy of this signature could improve slightly, alongside a significant reduction in cost, it might be that this approach could be cost-effective for diagnosing TB in the IGRA-positive population. It is feasible that the cost of some of these technologies will become more economical in the coming years. Furthermore, it is likely that the costs of a commercial assay using simpler but comparable technologies (i.e. a simple flow cytometry method rather than complex multicolour analysis) would be lower.

This health economic analysis has some limitations. Given the lack of accurate data on novel test costs, we have relied on estimates based on current laboratory research costs and timings, which may not reflect the likely costs of these tests after they have been developed commercially. Therefore, there is a high level of uncertainly in our analyses. The uncertainty about the test sensitivities and specificities is represented as marginal distributions when there is a correlation between the two values that may be represented in a joint distribution if the appropriate information is available. Furthermore, we highlight that one of the main assumptions of the analysis is that we consider a relatively short time frame up to TB diagnosis and, therefore, cannot extend the conclusions beyond this.

Principal findings

In this large study to validate new technologies for the diagnostic evaluation of active TB, we successfully evaluated four promising approaches for the diagnosis of active TB, all of which used blood samples: transcriptomics, proteomics, flow cytometry and immunological rule-out tests. In each substudy, we were able to evaluate multiple previously identified signatures and assess their diagnostic performance.

For the transcriptomic signature substudy, we report that diagnostic performances range from 0.81 to 0.84 for all previously reported gene expression signatures, for the detection of active TB in the full cohort (using LDA). The performance for all TB and for culture-confirmed TB was not found to be clinically useful. Although performance for either culture-confirmed TB or for PTB was highest, the scenario in which a transcriptomic test is likely to be useful is where current tests fail. For this reason, despite a maximum AUC of 0.80 for identifying highly probable TB, the Sweeney et al. 39 four-gene signature may be a useful clinical test for patients who test negative with microbiological tests. For culture-negative TB, this signature could provide a high NPV (0.95), which may be clinically useful.

None of the previously identified proteomic signatures was able to provide high diagnostic accuracy for detecting active TB cases, with the maximum test accuracy being for the Hamilton and Levin (unpublished) four-protein signature, detected by SELDI-TOF, which provided an AUC of 0.74 for culture-confirmed TB, using the DRS method. We were able to compare this signature across two proteomic technology platforms, SELDI-TOF and LC–MS, and found relatively poor corroboration between the two platforms. We were able to assess the performance of four published signatures19,21,51,52 using LC–MS, none of which performed well for detecting TB (not culture-confirmed or PTB groups).

Of the six previously identified cellular immune signatures for active TB, the %HLA-DR signature26 was found to perform the best, providing an AUC of 0.90 for all TB in the HIV– population. The %HLA-DR and %T_EFF tests25,26 provide high PPV for all TB and for culture-confirmed TB in IGRA-positive TB suspects and, therefore, may be clinically useful. Only the signatures that measured the phenotype (either memory/differentiation phenotype, or activation status) provided discriminatory ability. Those signatures measuring function alone did not perform well.

We tested two novel approaches for the development of a rule-out test. We found that the detection of CXCL9 by qRT-PCR of stimulated PBMCs provided a sensitive assay that could detect a high proportion of TB cases. However, the assay has a high failure rate and was also hampered by poor specificity. The use of CXCL9 for discriminating between TB and OD/LTBI in immune assays should be considered.

None of the signatures validated was found to be cost-effective when evaluated for use to rule in or rule out active TB, relative to the current standard diagnostic pathway in the UK. The signature that performed best was the %HLA-DR flow cytometry signature26 for the ruling in of all TB in the IGRA-positive population. However, this signature resulted in significant cost incurred and a slight reduction in QALY.

Strengths and limitations

In the IDEA study, we consecutively recruited a unique, large, multicentre, prospective cohort of individuals who were suspected of having active TB, as part of routine clinical practice. The cohorts we used in these validation studies (with the exception of the rule-out test, for which an artificial cohort was selected) represent the full spectrum of clinical disease reflecting heterogeneity of all clinical phenotypes and other differential diagnoses. The study participants, patients with suspected TB symptoms but unknown diagnosis at point of enrolment, represent the uncertainty faced by clinicians. The high-resource setting provides a high-quality diagnostic facility and reliable and accurate diagnosis for culture-unconfirmed, highly probable TB.

Ultimately, our study provided a real-life road test for the published signatures and evaluated their clinical utility, a crucial test before they enter the next phases of biomarker development. We present the performance of the previously published signatures in our hands. The previously proposed thresholds for each signature were not reported here, as, overall, these thresholds provided suboptimal performance in our cohort. In our experience with diagnostic tests in which the outcome is a continuous, the threshold for optimal diagnostic performance does change between laboratories and cohorts, for a number of reasons, particularly when these tests are yet to be standardised and when the cohorts are so variable. We therefore took several approaches to define optimal thresholds and their associated performance characteristics. Our findings suggest that the performance of all signatures and technologies in detecting TB in the HIV+ population was very poor, much worse than in the HIV– population. This is not in line with other studies, in which performance of signatures in the HIV+ population has either been comparable to that in the HIV– population, or marginally worse. 20,39,60 We were likely under powered to fully assess diagnostic performance in the HIV+ population, owing to the small number of these patients in the parent IDEA study cohort. As such, further validation is required in this population, in particular for the proteomic and cellular immune signature validations. Within the IDEA study cohort, there is a small number of patients who are classified as ‘clinically indeterminate’ (Dosanjh category 3), who were not included within this study, as assessing the performance of new tests within this population would give no known assessment of performance against gold standard/current practice. Nevertheless, they represent a subset of TB suspects who present in real-life clinical practice. We did, however, included this population when all calculations of pre-test probability for TB subsets were made.

In addition to providing detailed statistics on the diagnostic accuracy of the previously published signatures, we herein present complex health economic analyses, in which the most promising signatures were evaluated for cost-effectiveness for either the rule-in or rule-out of active TB. However, we did not present health economic analysis for all subgroups (i.e. culture-confirmed TB alone or PTB alone), and it is likely that effectiveness in these scenarios will improve as the cohort of patients becomes smaller and the clinical question narrower.

We are lacking some analyses. In particular, we do not compare different signatures with each other using a formal statistical comparison. Furthermore, there is a need for further subgroup analyses using these data, for example to perform health economic analysis in particular clinical scenarios (i.e. within the smear-negative groups), as a conclusion of this study is that these new tests are likely to be useful in specific types of TB suspects.

Recommendations for clinical practice

Given the poor performance of each of the new technologies for detecting the full spectrum of active TB cases in this study, and their lack of cost-effectiveness for this purpose, it is unlikely that the findings of this project will immediately have implications for changing current practice when it comes to the diagnostic pathways for TB suspects in the UK. None of the tests evaluated had sufficiently high diagnostic accuracy to provide a clinically useful rule-in or rule-out function over and above the clinical utility of the currently available diagnostic tests. However, diagnostic assessment of certain key groups of patients could potentially be improved by the flow cytometric and the transcriptomic tests. Smear microscopy results are available on the same day that the patient presents to secondary care and IGRA results are usually available by the next day. The role of new tests need to be considered in the context of patients with suspected TB for whom both these test results are known. In patients with suspected TB who are IGRA positive and smear negative, there is commonly a high index of suspicion for TB but a definitive diagnosis is not possible until culture results become available, typically after 2–3 weeks. A test that could rule in or rule out TB in these patients is thus an important unmet clinical need.

The %HLA-DR and %T_EFF flow-cytometry tests25,26 gave 94% PPV in the smear-negative IGRA-positive group, providing what is, effectively, a rule-in test and facilitating a decision to start treatment for presumptive TB. The test could thus form part of the diagnostic assessment pathway, forming part of an algorithm alongside smear microscopy and IGRA. Interestingly, two transcriptomic signatures provided high NPVs of 89–90% for culture-positive TB in the smear-negative population (unstratified by IGRA result). These NPVs may thus provide clinicians with sufficient basis for withholding treatment (or at least delaying it until culture results are available) in smear-negative TB suspects who test negative with these signatures.

Recommendations for future research

Smear still remains a quick, cheap and specific method for quickly ruling in TB in cases with high bacterial load. Therefore, a new test is not required for this subpopulation. However, it remains the case that most research into new TB diagnostics includes only individuals with ‘typical’ TB with high bacterial load, who are positive with either culture or smear tests, or both. The clinical need is in fact in the smear-negative patients, for whom there is currently not a quick test available for either rule-in or rule-out of active TB. This is particularly the case in low-prevalence settings, such as in the UK, where a high proportion of those diagnosed with TB are paucibacillary and/or have EPTB.

We have demonstrated that the transcriptomic test is able to provide a reasonably high NPV in those TB suspects who have a negative smear result. These tests are likely to be cheap and rapid if developed commercially and, if found to be reproducible using a rapid qRT-PCR approach, could potentially be developed into near-patient or point-of-care tests. For the reasons already discussed, we propose that future research into the use of these tests focuses on the smear-negative TB suspects and includes those with clinically diagnosed (i.e. culture-negative) TB. The high accuracy for diagnosing TB using the Adekambi et al. 26 and Pollock et al. 25 flow cytometry assays reported in Chapter 5 suggests a clinically useful role for these tests to help rule in a diagnosis of TB in smear-negative IGRA-positive patients, for whom this is currently a clear unmet clinical need. Accordingly, further development, standardisation and simplification of these assays should be prioritised in order to deliver an in vitro diagnostic for routine clinical use. An automated flow cytometry set-up that allows for simple assay preparation and automated gating and analysis would help to make the tests suitable for routine diagnostic laboratories. It is unlikely that these tests would be applicable to low-income high-burden settings, given their complexity and costs. We also propose that the tests are further validated within the specific subpopulations for which they would be useful (i.e. smear-negative TB suspects).

An overall conclusion from this study is that blood-based tests for TB can perform best when specific subtypes of TB are considered and when specific subsets of OD can be excluded. As such, we recommend that future studies exploring these technologies focus on specific subtypes of TB, most usefully those groups where a diagnostic test is most needed (i.e. EPTB, culture/smear-negative TB).

It appears that, although the ‘omic’ technologies are hampered by specific differential diagnoses (e.g. sarcoidosis) for the cellular immune signatures, for which immune cells are stimulated with Mtb antigens before the test is performed, there is no subgroup within the OD population which hampered diagnostic performance. As such, we conclude that immune-based tests are more useful when all TB is considered (with the exception, perhaps, of those infected with HIV).

Conclusion

In conclusion, in this report we presented the validation of new technologies for the diagnostic evaluation of TB and have fully evaluated their use in a real-life clinical setting. Overall, the diagnostic performance of all previously identified ‘signatures’ of TB was lower than previously reported. This probably reflects the nature of the cohort we used, which includes the harder-to-diagnose groups, such as those with culture-unconfirmed TB and EPTB, which were under-represented in previous cohorts.

We have addressed all of our primary objectives. Some analyses are outstanding; importantly, we are yet to fully address our secondary objective of deriving novel signatures for active TB using the data we generated in the substudies. This is an area of ongoing investigation by our group and was beyond the scope of this report. This study provided a real-life road test for these technologies and highlights the need for new diagnostic tests and signatures for TB to be derived and evaluated in cohorts that included the full spectrum of TB cases, and where the hard-to-diagnose subgroups of TB are considered.

Independent Scientific Advisory Board

Professor Sanjeev Krishna (Chairperson for ISAB meetings), Molecular Parasitology and Medicine, St George’s University of London, London, UK.

Dr Mahdad Noursadeghi (Clinical Senior Lecturer), Infection and Immunity, University College London, London, UK.

Dr David Connell (Respiratory Registrar), The Royal Brompton Hospital, London, UK.

Study Management Group

Professor Ajit Lalvani, Professor Onn Min Kon, Professor Peter Beverly, Dr Alice Halliday, Dr Long Hoang, Dr Pooja Jain, Dr Robert Parker, Mrs Mica Tolosa-Wright and Miss Tereza Masonou.

Wider Study Management Group

Professor Ajit Lalvani, Professor Onn Min Kon, Professor Peter Beverly, Dr Yemisi Takwoingi, Professor Jon Deeks, Dr Alice Halliday, Dr Long Hoang, Dr Robert Parker, Mica Tolosa-Wright and Miss Aime Boakye.

Contributions of authors

Alice Halliday (Research Associate) led the cellular immune signature validation substudy, oversaw the candidate rule-out validation studies, managed the overall VANTDET project co-ordination, co-ordinated the write up of the report and contributed to writing all sections.

Pooja Jain (Bioinformatician) led the statistical analysis for the transcriptomics and proteomics substudies, conducted the main statistical analyses for these two chapters (see Chapters 3 and 4, respectively), and supported the writing of methods, results and discussion sections in these chapters (see Chapters 3 and 4).

Long Hoang (Research Associate) led the validation of transcriptomic signature substudies, oversaw the validation of the molecular rule-out test, conducted statistical analysis and supported the write up of Chapters 3 and 6.

Robert Parker (Research Associate) led the validation of proteomic signatures substudy and led the writing of Chapter 4.

Mica Tolosa-Wright (Research Technician), with the support of Alice Halliday and Long Hoang, led the validation of the molecular rule-out test substudy and led the writing of Chapter 6.

Tereza Masonou (MSc student and Research Technician) supported the validation of cellular immune signature experimental work and analysis and interpretation of the data. She also supported the report write up by writing the methodology section for Chapter 5 and production of results tables.

Nathan Green (Health Economics Statistician) led the health economic analysis for the project and led the write up of the chapter (see Chapter 7).

Aime Boakye (IDEA study administrator) managed and co-ordinated the clinical database for the IDEA study patients and co-ordinated the cohort selection for the separate substudies within the VANTDET study. She ensured that the research staff remained blind to clinical data throughout the study period, until analysis phase. She also supported and led many of the public engagement and involvement activities.

Yemisi Takwoingi (Senior Research Fellow in Biostatistics) provided statistical analysis oversight, support and advice on study design throughout the project.

Shea Hamilton (Research Associate) performed the experimental work for the validation of the four-protein signature by SELDI-TOF technology.

Vinay Mandagere (Medical Student) designed and conducted the QuantiFERON supernatant project in Chapter 6.

Anastasia Fries (Junior Doctor) designed and conducted the QuantiFERON supernatant project in Chapter 6.

Lachlan Coin (Associate Professor) analysed the SELDI-TOF signature validation data.

Jon Deeks (Professor of Biostatistics) was a co-investigator on the project, contributed to the study design and support of statistical analysis approaches.

Peter J White (Reader in Public Health Modelling) was a co-investigator on the project, contributed to the study design and led the statistical analysis of diagnostic pathways and the economic analysis.

Michael Levin (Chairperson in Paediatrics and International Child Health) was a co-investigator on the project and oversaw the SELDI-TOF proteomic signature validation.

Peter Beverley (Emeritus Professor) stood in for Ajit Lalvani during his leave of absence (May 2015–16), oversaw the project, provided clinical expertise, and contributed to the study design and data interpretation.

Onn Min Kon (Professor of Respiratory Medicine) was a co-investigator on the project, provided clinical expertise, and contributed to the study design, data interpretations, execution and the writing of the report.

Ajit Lalvani (Professor and Consultant Physician Imperial College Healthcare) was chief investigator, provided clinical expertise and contributed to the study design, data interpretations, execution and the writing of the report.

Publication

Long TH, Jain P, Pillay TD, Tolosa-Wright M, Niaz U, Takwoingi Y, et al. Transcriptomic signatures for diagnosing tuberculosis in clinical practice Lancet Infect Dis 2021;21:366–75.

Data-sharing statement

All data requests should be submitted to the corresponding author for consideration. Please note exclusive use will be retained until the publication of major outputs. Access to anonymised data may be granted following review.

Patient data

This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives You can find out more about the background to this citation here: https://understandingpatientdata.org.uk/data-citation.

Disclaimers

This report presents independent research. The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the MRC, NETSCC, the EME programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the EME programme or the Department of Health and Social Care.

Analysis for bridging study

FIGURE 20.

Comparison of three proteins from the Hamilton and Levin (unpublished) signature with LC–MS data. (a) Correlation of protein 1 SELDI-TOF intensity for protein 1 with LC–MS ratio; and (b–d) dot plots for proteins 1–3 in LC–MS and SELDI-TOF data. Abundance was compared between groups using a Mann–Whitney U-test. P1, protein 1; P2, protein 2; P3, protein 3. **p < 0.005.

Abundance of signature proteins

FIGURE 21.

Abundance of SERPINA1 (alpha-1-antitrypsin or P01009). (a) Box plot depicts the median values and distribution of data in Dosanjh categories 1, 2 and 4; (b) ROC curves present the ability of the levels of SERPINA1 protein to discriminate between all TB vs. OD; (c) either culture-confirmed TB (Dosanjh category 1) or highly probable TB (Dosanjh category 2) (1/2) vs. OD (Dosanjh category 4); and (d) PTB vs. OD or EPTB vs. OD. Medians were compared between groups using the Kruskal–Wallis test. **p < 0.005.

Missing and imputed data

Spreadsheets detailing the missing data for each signature are available on request.