Artificial intelligence software for analysing chest X-ray images to identify suspected lung cancer: an evidence synthesis early value assessment

Search strategy

An iterative approach was taken to develop the search strategy, making use of relevant records identified during initial scoping searches and from relevant reviews. 12,13 The strategy was developed by an information specialist, with input from team members, aiming for a reasonable balance of sensitivity and specificity. Based on scoping work already undertaken, a series of complementary, targeted searches were favoured over a single search to retrieve a manageable number of records to screen (Appendix 1). Searches were run in a range of relevant bibliographic databases covering the fields of medicine and computer science. Searches were limited to studies published in English because studies published in other languages were likely to be difficult to assess in the timescale of this EVA. Non-human studies, letters, editorials, communications and conference abstracts were removed during the searches. No date limit was applied to the searches, but only records published in or after 2012 were screened. Database search strings were developed for MEDLINE and appropriately translated for each of the other databases, considering differences in thesaurus terms and syntax. The following bibliographic databases were searched: MEDLINE All (via Ovid), EMBASE (via Ovid), Cochrane Database of Systematic Reviews (via Wiley), Cochrane CENTRAL (via Wiley), Epistemonikos and ACM Digital Library.

A search for ongoing trials was conducted in the World Health Organization International Clinical Trials Registry Platform (WHO ICTRP). A search for ongoing systematic reviews was undertaken in the PROSPERO database.

The full record of searches is provided in Appendix 1.

Records were exported into EndNote X9.3 (Clarivate Analytics, Philadelphia, PA, USA), where duplicates were systematically identified and removed. Reference lists of included studies and a selection of relevant reviews were checked. Experts and team members were consulted and encouraged to share relevant studies.

Company submissions

As part of the Diagnostics Assessment Programme (DAP) process, 14 companies producing AI software were identified by NICE and invited to participate in the EVA and to submit evidence. 9 The External Assessment Group (EAG) assessed company submissions in exactly the same way as it assessed published evidence, and references lists were examined for relevant studies.

Eligibility criteria

The eligibility criteria for the test accuracy, practical implications and clinical effectiveness questions are presented in Table 1.

Review strategy

Titles and abstracts of records identified by the searches were screened by one reviewer, with a random 20% assessed independently by a second reviewer. Records considered potentially relevant by either reviewer were retrieved for further assessment. Full-text articles were assessed against the full inclusion/exclusion criteria by one reviewer. A random 20% sample was assessed independently by a second reviewer. Disagreements were resolved by consensus or through discussion with a third reviewer. Records rejected at full-text stage (including reasons for exclusion) are reported in Report Supplementary Material 2.

Data extraction

We planned to extract data into a piloted electronic data collection form. Data were to be extracted by one reviewer, with a random 20% checked by a second reviewer, and disagreements resolved by consensus or discussion with a third reviewer. However, no studies met the inclusion criteria.

Risk of bias

We planned to assess the risk of bias of included studies using tools appropriate to the study design, such as those produced by the Joanna Briggs Institute. 14 Risk of bias was to be assessed by one reviewer, with a random 20% assessed by a second reviewer and disagreements resolved through consensus or discussion with a third reviewer. As no studies met the inclusion criteria, no formal risk-of-bias assessment was undertaken.

Analysis and synthesis

Methods of analysis and synthesis were described a priori in the research protocol. 1 However, no studies met the inclusion criteria, so no data synthesis was undertaken.

Post hoc methods

No studies meeting the inclusion criteria were identified. Following discussions with the NICE technical team for this project, we examined the list of excluded studies that were closest to the review inclusion criteria (see Table 1), that is:

• Interventions: CXRs interpreted by radiology specialist in conjunction with eligible AI software versus radiologists alone and/or reference standard.

• Population: no details provided on the referral status or symptom status [studies that had an explicitly excluded population, for example health screening, preoperative CXRs, inpatients, accident and emergency (A&E), were not selected].

• Outcomes: as defined in Table 1.

Selected studies were tabulated using the approach described in Review strategy and key biases were noted. Results were summarised narratively.

Literature review

A pragmatic search of the literature was used to identify existing methods of cost-effectiveness modelling for AI software in CXRs and inform parameterisation of the conceptual model. It was not intended as a substitute for a systematic literature review or to provide a definitive summary of evidence gaps. This will be required for any future development of an executable cost-effectiveness model.

Following initial scoping searches, we did not expect to find any full economic evaluations of AI software as an adjunct to radiology specialist review of CXRs, particularly in the primary care population. For this reason, a broad search strategy was used across two databases (MEDLINE and Tufts CEA), and broad screening criteria were applied. The primary inclusion criterion was ‘lung cancer studies’, but, following this, any study that could inform the structure or parameters of a conceptual model was identified at title/abstract level. Full-text assessment of these papers was used to refine screening criteria further into studies that satisfied (1) the primary care referral population, (2) those with specific intention of diagnosis or screening and (3) those most relevant to the UK setting. Reference lists of these studies and publication lists of authorship groups were also screened for any further potentially relevant papers. Studies identified in these targeted reviews were not subject to a formal assessment but discussed narratively. This focused on the methods used, assumptions made, availability of evidence to support evidence linkage approaches and considerations for future modelling and research.

Clinical guidelines

The structure of the decision-analytic model is intrinsically linked to current clinical pathways. Key points throughout the clinical pathway for the detection and management of lung cancer, and the positioning of AI software within this pathway (for adults referred for CXR from primary care), were identified with reference to Figure 1 in the final NICE scope for this topic,9 existing guidelines on the diagnostic and care pathway10,11,16,17 and close collaboration with clinical experts.

Company and clinical expert involvement

Information on the relevant AI technologies under review was obtained from company submissions, with requests for additional information sent to companies that registered as stakeholders (Annalise AI, Behold AI, Infervision, Lunit Inc. and Siemens Healthcare).

Using the information gathered from these sources, an iterative process was used to achieve a model structure that is pragmatic in its representation of the complex clinical pathways that adults from primary care populations may follow to arrive at a diagnosis of lung cancer.

Given the time available to conduct this EVA, the primary focus of this report was the diagnostic component of the model. Priority was given to the following:

• input parameters to populate the model – including consideration of the type of evidence required, sources available and gaps in the evidence

• relevant outcome measures to compare the cost-effectiveness and clinical effectiveness of AI software in the detection of lung cancer

• identification of potential value drivers of the model – with recommendations of how these can be measured for inclusion in a cost-effectiveness model.

Once diagnosis is achieved in the model, evidence linkage between intermediate outcomes and long-term outcomes is required to assess cost-effectiveness over a clinically appropriate time horizon. These mainly relate to the mapping of the disease state (i.e. lung cancer) and are not specific to the diagnostic technology being assessed (e.g. utilities, costs and effects of current treatments). Potential sources for the main longer-term outcomes were identified during the literature search, with a focus on those relevant to the UK setting and in line with the requirements of the NICE reference case. 18 An overview is presented in this report as an example of current practices in modelling lung cancer.

Methods to assess potential budget impact

Estimates of the potential budget impact of introducing AI software as an adjunct to radiology specialist review of CXRs were calculated based on methods for a budget impact analysis (BIA) outlined in the NICE evidence standards framework for digital health technologies19 and International Society for Pharmacoeconomics and Outcomes Research Task Force recommendations. 20 These identify six key elements that require inputs for the modelling framework of a BIA:

1. size and characteristics of affected population

2. current intervention mix without the new intervention

3. costs of current intervention mix

4. new intervention mix with the new intervention

5. cost of the new intervention mix

6. use and cost of other health conditions and treatment-related healthcare services. 20

Given the limitation of time and scope, a fully comprehensive BIA was not attempted as this would have required data on any changes in resource use and associated cost. The intended outcome of this report was a conceptual model where no outcome data were run and no results were produced. Therefore, estimates on this element were not included. The aim was to approximate the budget impact at an individual institution level, with information sourced from the literature and supplemental information provided by representatives of the institution used as an example.

Company submissions to NICE as part of the DAP request for information were screened for cost data. Clarifying questions were sent to all companies (whether or not costings were already submitted) to obtain more granular detail for the purpose of BIA.

Records retrieved from the broad cost search were screened at title/abstract level by one reviewer (MJ) to identify any studies that may have been applicable. These were then retrieved as full text and their suitability for use was assessed. Studies that yielded relevant information were retained, data extracted and authors contacted to obtain further context-specific information.

Bajre et al.32

Bajre et al. 32 used a decision tree structured model to assess the cost-effectiveness of trained radiographers compared with radiologists for the reporting of CXR in people suspected of having lung cancer. The model simulated a pathway for a hypothetical cohort of 1000 people undergoing CXR for suspected lung cancer, with cost-effectiveness calculations concluding at 5 years. The model started with a cohort of people receiving either a radiologist-reported CXR or a radiographer-reported CXR. The pathway for both strategies was the same. The proportion of those with true disease status was known, characterised by the prevalence of lung cancer. People with lung cancer who had a positive CXR result received a confirmatory test of a CT scan, which also provided staging. The authors included stages I, II, III and IV lung cancers. People with a false-negative result presented later to the A&E department, where they were diagnosed with lung cancer and staged. People who had a false-positive result following CXR received a CT scan that confirmed no lung cancer was present. People who had no lung cancer and had been correctly identified as negative by the CXR received no further testing/imaging.

Information required to populate the model was obtained from the literature and NHS reference costs. The model required information about the prevalence of lung cancer, sensitivity and specificity of radiologist-reported and radiographer-reported CXR to identify lung cancer, and sensitivity and specificity for radiologist-reported CT scan to confirm lung cancer diagnosis and probabilities. Although not explicitly stated, a confirmatory diagnosis was made by the radiologist. The proportion of people diagnosed at first presentation was obtained from statistics published by Cancer Research UK in 2013 (Bajre et al. 32). Additionally, information was required about the probability of lung cancer by stage at second presentation following misdiagnosis. All costs included in the model were reported at 2014–5 prices. Costs were required for radiologist and radiographer reading of CXRs, cost of CT scans and total costs of treatment by stage. The authors were not explicit about which treatment people received. The benefit of the strategies was reported in terms of cases detected at first presentation and quality-adjusted life-years (QALYs) yielded. Utility values by stage of diagnosis were obtained from Naik et al. 40

Several simplifying assumptions were made to give a workable model structure:32

• Time taken to report CXR is 2 minutes for both radiographers and radiologists.

• False negatives present at A&E at a later date, at which point disease may have advanced a stage (for patients at stages I to III).

• Sensitivity and specificity of radiographer reporting of CXR and radiologist reporting of both CXR and CT scan are independent of disease stage or other patient characteristics such as age.

• Quality of life (QoL) in the year following diagnosis (according to stage at diagnosis) is maintained in subsequent years.

• There is no QoL impact arising from false-positive reporting.

• Findings for non-small-cell lung cancer are representative of lung cancers in general.

The perspective and setting of the economic analysis were not clearly defined, but the analysis appears to be from the NHS and Personal Social Services (PSS) in a secondary care setting, based on the cost inputs. The results of the analysis were presented in terms of an incremental cost-effectiveness ratio (ICER), expressed as cost per QALY. The authors undertook a probabilistic sensitivity analysis (PSA) to assess the joint uncertainty in key model input parameters: prevalence of lung cancer, sensitivity and specificity of radiologist and radiographer reporting of CXRs, lung cancer stage distribution at initial CXR and stage progression following misdiagnosis. The authors stated the sampling distributions for the parameters included in the PSA but have not reported their parameters. The authors undertook a threshold analysis but not a one-way sensitivity analysis.

The authors reported disaggregated results for both strategies. Results were reported on the number of people expected to be diagnosed with lung cancer, QALYs yielded and treatment costs, all by stage. The QALYs yielded appeared to be high, with stage IV expected to yield more QALYs than stage III, and both of these higher than with stage II. There were modest QALY gains by strategy and by stage, with stage I having the greatest expected gain of 2.4 QALYs, favouring radiographer reporting. Radiographer reporting yielded more overall QALYs, but it was unclear with the inputs reported why QALYs yielded by radiologist reporting was greater for stages II and stage IV. Radiographer reporting diagnostic and treatment costs were lower than radiologist reporting costs. Overall results showed that radiographer reporting of CXR dominated radiologist reporting. The PSA results showed that radiographer reporting continued to dominate radiologist reporting in 98% of the iterations. Based on the model structure, its inputs and assumptions, the authors concluded that the use of trained radiographers to report CXR is cost-effective and an increased role for radiographers in the diagnostic pathway would be beneficial to meet hospital waiting time targets for lung cancer diagnosis.

Foley et al.35

Foley et al. 35 conducted a retrospective review of trust audit data (Royal United Hospitals Bath NHS Foundation Trust) to analyse the use of CXR as the first-line investigation in primary care patients with suspected lung cancer. In total, 1488 of the 16,495 primary care referrals received between 1 June 2018 and 31 May 2019 were for suspected lung cancer. CXRs were coded by result as CX1, normal but a CT scan is recommended to exclude malignancy; CX2, alternative diagnosis; or CX3, suspicious for cancer. Outcomes for the study cohort were stratified by CX code and included patient characteristics, number undergoing CT scan, number of lung cancers diagnosed, stage at diagnosis, time from initial CXR to CT scan, time from CT request to CT scan, time to diagnosis, treatment strategy taken and mortality (over an average follow-up period of 322 days in the total cohort). Table 5 shows the results of key outcomes.

Based on these findings, the authors concluded that there was significant delay in lung cancer diagnosis in patients who received a CX1 ‘normal’ initial CXR result (p < 0.001) and the majority of patients with a ‘normal’ or an ‘abnormal’ CXR are diagnosed at an advanced disease stage (p = 0.26) with no difference in survival outcomes based on the CXR findings (p = 0.42). 35

Bradley et al.36

Bradley et al. 36 undertook a retrospective observational study using routinely collected healthcare data from Leeds Teaching Hospitals NHS Trust.

All patients diagnosed with primary lung cancer between January 2008 and December 2015 with a GP-requested CXR in the year before diagnosis were coded based on the result of the earliest CXR in that period. CXR report codes were assigned: (1) suspicion of lung cancer identified/urgent investigation needed; (2) abnormality identified/non-urgent investigation indicated, including diagnoses of pneumonia or consolidation even if repeat imaging was not explicitly suggested; (3) abnormality identified but no further investigation/assessment indicated; and (4) normal CXR, no abnormalities identified.

The sensitivity of CXR was calculated and analyses were performed on time to diagnosis, stage at diagnosis and survival outcomes. Statistical analysis on these outcomes was performed by combining CXR codes 1 and 2 to form a ‘positive’ result group and codes 3 and 4 to form a ‘negative’ result group. However, the authors present numerical outcome data for all codes separately as well as for combined groups. Table 6 shows a summary of the key data by individual codes.

Data were also presented on the number of people who had further CXRs requested by their GPs, with median time to second CXR and median times to diagnosis from initial CXR. Of 376 patients with an initial CXR that was ‘negative’ (codes 3 and 4), 98 (26.1%) had at least one further CXR. Sensitivity calculated based on initial CXR (codes 1 and 2) was 82.3% (95% CI 80.6% to 84.1%).

The authors concluded that the sensitivity results supported previous systematic review findings,41 and while those with a ‘positive’ initial CXR finding had a median of 43 days to diagnosis compared with 204 days for those with ‘negative’ findings, no direct association with time to diagnosis was found between stage at diagnosis and survival in this study.

Woznitza et al. (2018)

Woznitza et al. 37 conducted a 4-month feasibility study (November 2016 to March 2017) at a single radiology department at an acute general hospital (Homerton University Hospital, London). The primary outcome was to establish the feasibility of an immediate reporting service for CXRs. Comparison between CXR referrals from general practice that received an immediate and routine report was made to determine the number of lung cancers diagnosed, time to diagnosis, time to CT and number of urgent referrals to respiratory medicine.

From 1687 CXRs of people referred from general practice over the 26-week study period, 36 patients (22 immediate CXR report, 14 routine CXR report) had a CT scan arranged by radiology following a suspicious CXR. This equated to less than one additional unplanned patient per week (mean 0.8 scans per week) accommodated by the CT department. Time from CXR to CT was shorter in the immediate report group, with a mean of 0.9 (SD 2.3) days, than in the routine reporting group, at 10.6 (SD 4.5) days (p > 0.0001). No apparent difference was found in time to discussion at multidisciplinary team (MDT).

The study also gave a detailed description of the radiology department demographics and processes for reporting and referral. The results of all CXRs included in the study and pathways taken were explained, including 17 patients with a normal or non-cancer diagnosis at CXR who were subsequently diagnosed with lung cancer.

The authors concluded that it was feasible to introduce a radiographer-led immediate CXR reporting service, but a definitive study assessing outcomes would be needed to determine whether this would have an impact on patient mortality and morbidity.

Woznitza et al. (2022)

Woznitza et al. 38 conducted a prospective, block-randomised controlled trial (RadioX) at a single acute district general hospital in London (Homerton University Hospital). People referred for CXR from primary care attended sessions that were pre-randomised to either immediate radiographer (IR) reporting or standard radiographer (SR) reporting within 24 hours. Those who received SR reporting were the control group, as this was usual practice in the department. In the intervention group, CXRs were reported while the patient was still in the department, with all patients with CXR findings suggestive of lung cancer offered a same-day CT scan. Those who declined were scheduled for another day.

In total, 8682 CXRs were performed between 21 June 2017 and 4 August 2018, 4096 (47.2%) for IR and 4586 (52.8%) for SR. Lung cancer was diagnosed in 49 patients. Table 7 shows the summary outcome data from trial reporting arms.

The authors stated that a health economic evaluation based on their RadioX trial was to be reported separately. 38 The corresponding author was contacted and confirmed that analysis of the data was still under way and they were unable to share any usable information at that time (Nicholas Woznitza, consultant radiographer, University College Hospital London NHS Foundation Trust, 29 November 2022, personal communication).

Dwyer-Hemmings and Fairhead39

The authors performed a systematic review of evidence to inform the diagnostic accuracy of CXR to detect lung malignancy in symptomatic patients presenting to primary care. Nine databases were searched, and data from included studies were extracted to calculate the sensitivity and specificity of CXR where possible. Risk of bias was assessed using the QUADAS-2 tool, with analyses conducted by means of random-effects meta-analyses. Ten studies were included in this review. Summary sensitivity of five studies (those not at high risk of bias) was 81% (95% CI 74% to 87%). Specificity of five studies was 68% (95% CI 49% to 87%). The authors concluded that there was good evidence regarding sensitivity because they included only those studies that were of similar design and were not at high risk of bias. By contrast, they considered the evidence on specificity to be weaker due to differences in study designs and variability in reported outcomes. 39

Pathway A

When CXR findings are suggestive of malignancy, a referral for urgent CT on the suspected lung cancer pathway is made. There is a variation in practice across trusts, but in many institutions highly suspicious CXR findings are flagged to secondary care lung cancer teams who request the CT scan and await referral to the suspected lung cancer clinic from the GP. Once reported, CT scans are triaged by lung cancer team consultants. If they suggest probable lung cancer, an urgent lung cancer team appointment is arranged with appropriate tests for example spirometry, planned biopsy (endobronchial ultrasound) (Alberto Alonso, consultant radiologist, Manchester Hospital NHS Foundation Trust, 19 November 2022, personal communication) for histopathological staging and to inform treatment options at the fast-track lung cancer clinic. 16

If the CT scan appears reasonably normal (despite CXR appearances), then the lung cancer team writes to the patient to inform them of their relatively normal CT appearances and arrange a non-urgent general respiratory clinic (not lung cancer clinic) outpatient appointment (Vidan Masani, consultant respiratory physician and lead for lung cancer, Royal United Hospitals Bath NHS Foundation Trust, 1 February 2023, personal communication). This also includes those who require investigation and management of pulmonary nodules in accordance with British Thoracic Society guidelines. 16,17

Pathways B and C

If CXR results are reported as ‘abnormal’ where findings are indeterminate or suggestive of an alternative diagnosis, people may follow pathway B or C. Here, findings are not sufficient to warrant further urgent investigation, but additional clinical enquiry is required.

Pathway B is taken when an alternative diagnosis is suspected and referral is made by the GP to a secondary care outpatient clinic with relevant expertise for that clinical finding, for example a non-urgent respiratory clinic.

Pathway C is followed when a 6-week repeat CXR is advised in the report. The referral for repeat CXR is made by the GP, and a radiologist or reporting radiographer compares the new image with the previous one. If the abnormality is resolved, then no further action or follow-up is required. If abnormal and suspicious, these cases are ‘red-alerted’ or ‘upgraded’ and the lung cancer team and referring GP are notified as per pathway A. The 6-week repeat CXR is used in cases where there is need to exclude infection, try a course of treatment and reassess before considering CT (Jonathan Rodrigues, consultant radiologist, Royal United Hospitals Bath NHS Foundation Trust, 13 November 2022, personal communication).

Pathways D and E

Where CXRs are reported as ‘normal’, findings may be unremarkable, but several trusts (including Royal United Bath NHS Foundation Trust and Manchester University NHS Foundation Trust) include the following automatic caveat in the report: ‘please note that a normal CXR does not exclude malignancy. If there is still a strong suspicion of malignancy (weight loss/unresolved cough/significant or unresolved haemoptysis), referral for a CT scan is advised’. This is to counter false reassurance in a case where clinical suspicion remains high.

People with normal results may, therefore, proceed along pathway D, where their GP considers them at high risk of lung cancer despite nothing being detected on CXR and refers the patient for CT scan and specialist review.

Pathway E is taken when the GP has no further concerns, no further diagnostic testing is requested and management is continued under primary care.

Intermediate measures for consideration

• Accuracy in detecting lung cancer

No eligible studies were found in the clinical effectiveness review, but one of the six ineligible studies summarised examined the test accuracy of AI software in detecting lung cancer on CXR. 22 In this UK study of Red Dot (Behold.ai), sensitivity was significantly higher for the interpretation of CXR with AI (77% 95% CI 75% to 80%) than without AI (66%, 95% CI 59% to 71%). No difference was observed for specificity (with AI: 75%, 71% to 77%; without AI: 81%, 77% to 85%) (Table 3).

A systematic review and meta-analysis identified in the cost-effectiveness literature review39 provided evidence on the test accuracy of CXR to detect lung cancer in symptomatic patients presenting to primary care. In this population, specifically relevant to this review, summary sensitivity of 81% (95% CI 74% to 87%) was calculated from five studies not at high risk of bias. Summary specificity of 68% (95% CI 49% to 87%) was also obtained from five studies, but evidence was weaker due to their heterogeneous design and variation in reported outcomes. 39 Findings of this systematic review were supported by two other studies from the cost-effectiveness search. 36,41 A retrospective database study of all primary care referrals for CXR conducted by Bradley et al. reported sensitivity of 82.3% (95% CI 80.6% to 84.1%). This was calculated based on an initial CXR coding system that included results suggestive of lung cancer and those with an abnormality identified but no urgent investigation indicated as a ‘positive’ result for CXRs.

• Turnaround time (TAT; time from start of image review to radiology report)

Turnaround time was identified in the final scope9 as a potentially useful outcome measure in this assessment. From a modelling perspective, the review time occurs on the pathway prior to the diagnostic decision outcome. This would be captured in a model as a resource use parameter used to calculate the cost per image, where the rate of radiology specialist’s pay is multiplied by the length of time to review scan.

A reduction in cost may be expected where TAT is decreased. However, the direction and magnitude of this relationship are highly uncertain given the lack of evidence found on TAT with AI software assistance and the variation of estimates given for TAT without AI from the literature and clinical expert feedback.

Estimated TAT for CXR varies considerably. As discussed in What are the practical implications of adjunct artificial intelligence software to detect lung cancer on chest X-rays?, of the ineligible studies reported on from the clinical search, two24,25 presented information on reading times. No statistically significant differences were observed in average image reading times between readers with and readers without AI: Siemens Healthineers AI-Rad Companion 22.5 (SD 40.3) seconds with AI, 24.3 (SD 27.4) seconds without AI, per image;24 Lunit Insight 171 (SD 33.8) minutes with AI, 211.25 (SD 38.4) minutes without AI, to read 434 CXRs,25 which equates to an average of 23.6 seconds per image with AI, and 29.2 seconds without AI (calculated by the EAG).

No information was given on the methods used for timing. With regard to context, timings were recorded during specified reading sessions under study conditions, so how this would translate to reading times in clinical practice is unknown.

Methods by the Royal College of Radiologists (RCR) to derive guidance on reporting output figures are described comprehensively. 43 Eighty reports for plain CXRs per hour (45 seconds per image) is the figure expected on average, over a 6-month minimum period, per in-hours, on-site, non-acute 4-hour reporting session in the NHS. 43

Specialist committee members advised average reading times of < 1 to 5 minutes, with an assumption of 2 minutes used in the economic evaluation by Bajre et al. 32

Many factors have an impact on reporting output and are well outlined by the RCR. 43 Therefore, focusing on this as an outcome measure, without appreciation of real-world context, is of little use unless a reduction in TAT can be shown to have an impact on efficiency of workflow over a sustained period in the NHS environment. This needs to be considered when designing future studies.

Another anticipated benefit of reducing TAT is an increase in the output of radiology specialists performing CXR reviews, thereby addressing the high demand for image reading and inherent limitations on workforce capacity. This is a potential value driver of AI software but would not be captured within the conceptual cost-effectiveness model. The potential value here would be recognised at a system level rather than at the patient level represented in the conceptual model.

• Technical failure rate

Technical failure rate was identified in the final scope9 as a potential measure of interest. None of the six studies summarised in the clinical effectiveness review reported any information on technical failure rate in CXR.

• Impact of software output on clinical decision-making

Impact of software on clinical decision-making is the primary measure of importance as the final CXR result is determined by a radiology specialist whether or not AI software is used. Even if the diagnostic accuracy of AI software alone is higher, the outcomes are mediated by human input. The results then determine which clinical pathway a patient will proceed down, affecting the quantity and type of further tests.

No evidence was found on this, and the only extrapolated data were in the form of two studies22,24 that provided information on hypothetical referrals to CT. No statistically significant differences were observed in the number of people who might be recommended for CT follow-up between readers with and readers without AI: Red Dot (Behold.ai) 144 out of 400 (36%) (95% CI 119 to 172) potential referrals with AI and 117 out of 400 (29%) (95% CI 93 to 147) potential referrals without AI;22 Lunit INSIGHT 96 out of 351 (27%, 95% CI 22.8% to 32.3%, calculated by the EAG) patients with AI and 80 out of 351 (23%, 95% CI 18.5% to 27.5%, calculated by the EAG) patients without AI. 24 It is important to note that these are hypothetical referrals, as CXRs were retrospectively selected from databases in these studies. We found no evidence of the impact of AI on the readers’ behaviour in real-world clinical practice.

• Number of people referred for a CT scan

The number of people referred for a CT scan depends on test accuracy and referral decision based on CXR result. As highlighted in the clinical review, evidence to inform these parameters that fall earlier in the clinical pathway was not available in the primary care population for CXR review with adjunct AI.

Computed tomography scans may be requested as a result of initial investigations, usually CXRs, undertaken in any of pathways A, B, C and D (see Clinical pathway for representation in model for a detailed description). Therefore, the proportion of people referred from each pathway for CXR would be needed for a model.

Only two studies identified in our literature search35,37 mentioned the number of people referred for a CT scan. Woznitza et al. 37 reported that a total of 36 patients out of the 1687 referred for CXR from primary care underwent a CT scan. This included both suspected lung cancer and non-suspected lung cancer populations. The study by Foley et al. 35 provided much more detailed information (Table 5) and was specific to the GP-referred population with suspected lung cancer. The number and percentage of CT scans requested by the three CXR result codes were reported: CX3 (suspicious for malignancy), 92% (66/72) had a CT scan; CX2 (abnormal, alternative diagnosis), 37% (107/288) had a CT scan; and CX1 (normal), 10% (107/1056) had a CT scan.

Although limited to only three potential pathways, the data and reporting format from this paper35 are useful to inform conceptual model parameters for current practice with no AI software. Future studies to identify the number of CT referrals made after CXR review with and without AI software assistance, stratified and reported by clinical pathway for both symptomatic (suspected lung cancer) and incidental (no lung cancer suspected) primary care population, are required. Ideally these would be of prospective study design, but hospital-reported data could be used to retrieve this information retrospectively in the incidental primary care population.

• Number of people referred for follow-up CXR

Two studies36,37 reported information on follow-up CXR following initial CXR results. In the study by Woznitza et al.,37 all patients with CXRs reported as showing pneumonia had a follow-up CXR suggested in 4 to 6 weeks to ensure resolution (17/522, 3%). Where a follow-up CXR was suggested, four (22%) were performed with a mean time from initial to follow-up CXR of 33.8 days (range 10–49 days). In the 13 other cases, follow-up CXRs were not done at the same institution and the authors assumed that they had not been undertaken as no reminders were sent.

Bradley et al. 36 reported follow-up CXRs performed based on result codes of 2129 initial GP-requested CXRs. Of the 376 patients who had an initial ‘negative’ result (codes 3 and 4), 98 (26.1%) had at least one further CXR. Of the 370 patients with an initial abnormal finding where non-urgent further review or investigation was advised (code 2), 191 (56.1%) had a second CXR. The median duration to second CXR was 42 days [interquartile range (IQR) 28–57 days]. In total, 324 (15.2%) patients across all CXR result codes (1–4) had at least two CXRs before diagnosis. 36

These studies36,37 are informative of CXR resource use across multiple pathways, which is useful to consider in future modelling. While Woznitza et al. 37 had only a relatively small sample size in their feasibility study, it was a prospective design and for this measure reported specifically for those on a clinical pathway following ‘abnormal’ (other diagnosis) CXR results. This would be pathway B (see Clinical pathway for representation in model) in the conceptual model. It illustrates that the number of people referred for follow-up CXR does not necessarily equate to resource use, as patient uptake rate is also a factor.

• Number of cancers missed/detected

One (ineligible) study22 from the clinical effectiveness review reported the mean number of cancers detected and found no significant differences with and without AI software (54 cancers, 95% CI 42 to 59 cancers, and 46 cancers, 95% CI 38 to 51 cancers, respectively).

Among the 1687 CXR referrals in the study by Woznitza et al.,37 17 patients were missed who were subsequently diagnosed with lung cancer: 15 were given normal CXR results and 2 were given abnormal (alternative diagnosis) results.

Among 8682 CXR referrals in the Woznitza et al. 38 study, 48 of the 49 lung cancers diagnosed were detected. The single case that was missed was diagnosed on a subsequent emergency attendance for upper limb deep-vein thrombosis. 38

Foley et al. 35 reported the number of cancers diagnosed by CX code: CX1, 10/1056 (1%); CX2, 29/288 (10%); and CX3, 49/72 (68%). Ten people with lung cancer were given false-negative ‘normal’ results but were still referred for CT and received diagnosis. Data on the other 949 patients with negative CXR results who were not referred for CT would be informative (although difficult to obtain) to give the total number of false-negative results by CX1 code for use in modelling. Similar information would be required for the CX2 result patients.

Future studies with extended follow-up and use of patient-level hospital-reported data linked to cancer registries would facilitate access to and reporting of this information. These may also provide data on stage at diagnosis.

Both numbers of false negatives (i.e. lung cancers missed) and stage at diagnosis may be important outcome measures for use in evidence linkage. This could be used in modelling to inform any association between time to diagnosis and stage shift and to assign appropriate costs and QoL outcomes by stage at diagnosis.

• Stage of cancer at detection

Bradley et al. 36 found that 1490 (70%) of the 2129 patients in their study were diagnosed with lung cancer at stage III/IV. Across the four CXR codes used to stratify results of initial CXR, these were reported as (1) 981 (70.9%), (2) 259 (70%), (3) 147 (63.9%) and (4) 103 (70.5%). There was no evidence of a statistically significant association between CXR result and stage at diagnosis. 36

Foley et al. 35 also found no statistical difference between CXR result and stage at diagnosis. Those with advanced stage (IIIc/IV) at diagnosis were reported as CX1, 5 (50%); CX2, 11 (38%); and CX3, 28 (57%) (p = 0.26). This was a much smaller sample size than in the Bradley et al. 36 study, and advanced stage was defined as IIIc/IV35 rather than III/IV. 36

Findings from both studies35,36 showed that a majority of patients with normal or abnormal CXR results have advanced stage disease at diagnosis.

• Time to CXR report

Time to CXR report was highly dependent on trust and service provided. Most had a same-day reporting facility for GP-requested plain CXR films. Woznitza et al. 38 reported on the RadioX trial that compared immediate reporting and standard reporting to find median report time to CXR report (termed TAT in this paper).

This may be a more informative measure than TAT per scan as it has a more direct impact on the speed at which a CT scan is requested.

• Time to CT scan

Time from CXR to CT scan was reported in two studies retrieved from the cost-effectiveness search. 35,37 Foley et al. 35 found a significant difference in the number of days from CXR to CT by CX result code. The reported mean days were 34.6 for those with CX1, normal but a CT scan is recommended to exclude malignancy; 19.6 for CX2, alternative diagnosis; and 1.9 for CX3, suggestive of cancer.

By contrast, the feasibility study by Woznitza et al. 37 looked at the time from CXR to CT scan by reporting strategy for those with a CXR result suggestive of lung cancer. Those whose CXR image was reported immediately had a mean of 0.9 days (n = 22, SD 2.3 days) until CT scan compared with routine reporting (mean 10.6 days, SD 4.5 days; p > 0.0001).

Although these cannot be compared directly, the results of Woznitza et al. 37 are for the equivalent result population of the CX3 in the Foley et al. 35 study. This shows significant variation in time from CXR to CT scan as a result of department reporting practices alone. In the Foley et al. 35 study there were GP reporting sessions for consultants on most days (Jonathan Rodrigues, personal communication), suggesting that this was more in line with the standard reporting process in the study by Woznitza et al. 37 However, many other procedural variables between the two radiology departments are likely to have an impact on these times.

This highlights the need for the real-world clinical context to be taken into consideration when generating future evidence to inform these measures. This is relevant for studies of outcomes after CXR both with and without AI, as there are only limited data even in current practice, which is difficult to generalise because of variation both within and between NHS trusts.

The results from Foley et al. 35 are useful for modelling purposes as they establish a difference in time between three diverging clinical pathways, up to the point of confirmatory testing by CT scan. This may support evidence linkage to outcomes further in the lung cancer management pathway.

• Time to diagnosis

Four studies35–38 from the cost-effectiveness review reported time to diagnosis. In three,35,36,38 this was calculated as the date of the initial CXR to the date the diagnosis was confirmed (either the date of the diagnostic test or the date on which a clinical diagnosis was confirmed by the lung cancer MDT if no pathological sample was taken). In the smallest of the studies, Woznitza et al. 37 used date of radiological diagnosis confirmed at MDT. The results of histological diagnosis were reported separately, but no data on timing for this were provided.

Foley et al. 35 found a significant difference in time to diagnosis between CX codes, with a mean of 89.7 days for those with CX1, normal but a CT scan is recommended to exclude malignancy; 65.3 days for those with CX2, alternative diagnosis; and 30.2 days for those with CX3, suggestive of cancer.

Bradley et al. 36 also reported time to diagnosis by initial CX codes but used median number of days: code 1, suspicion of lung cancer identified/urgent investigation needed, 36 (IQR 23–63) days; code 2, abnormality identified/non-urgent investigation indicated including diagnoses of pneumonia or consolidation even if repeat imaging was not explicitly suggested, 93 (IQR 55–154) days; code 3, abnormality identified but no further investigation/assessment indicated, 211 (IQR 181–296) days; and code 4, normal CXR, no abnormalities identified, 193 (IQR 87–279) days. When calculated by author-defined ‘positive’ (codes 1 and 2) and ‘negative’ (codes 3 and 4), time to diagnosis was 43 (IQR 27–78) days and 204 (IQR 105–287) days, respectively. 36

Woznitza et al. 38 presented both mean and median days to diagnosis for those who had IR and those who had SR of their CXR image. The mean number of days was 47.2 (SD 35.8) for IR and 81.6 (78.5) for SR. When median days to diagnosis of 32 (IQR 19–70) for IR and 63 (29–78) for SR were analysed, statistical significance was found (p = 0.03). 38

Woznitza et al. 37 also looked at mean time to diagnosis for IR and SR, with study findings of 4.1 and 10.6 days, respectively. However, this was for a small sample of 11 patients, and as discussed this was for radiological diagnosis at MDT only and so did not account for additional waiting time due to biopsy. 37

All four studies35–38 reported substantial variation in time to diagnosis, demonstrating that this outcome measure can be affected by multiple factors, including CXR result and the subsequent diagnostic pathway followed35,36 and different reporting practices in a radiology department. 37,38 Establishing that AI software has an impact on time to diagnosis beyond fluctuating departmental factors, examining the mechanism by which that impact is produced (by increasing test accuracy, reducing report TAT or other means) and quantifying the impact would require a prospective study, in real-world settings, ideally across multiple sites in the UK.

Once established, change in time to diagnosis may support evidence linkage to outcomes further in the lung cancer management pathway.

• Ease of use/acceptability of the software by clinicians

In the UK study,22 10 of the 11 clinicians responded to questions about acceptability of the AI. Eighty per cent stated that reporting was not slower when using AI, and 90% stated that the AI ‘heatmaps’ produced were ‘helpful to understand the algorithm’s attention points’.

Clinical outcomes for consideration may include:

• morbidity

• mortality.

Costs will be considered from an NHS and PSS perspective.

Costs for consideration may include:

• cost of each AI software available for this indication

• costs of training staff to use software

• costs associated with healthcare professional time to read and report CXR

• costs of diagnostic testing and treatment.

Sources of cost and resource use inputs are discussed in question 5, What are the cost and resource use considerations relating to the use of adjunct AI to detect lung cancer.

Summary

Importantly, the EAG did not identify any studies in the clinical effectiveness review that met the inclusion criteria and addressed the outcomes for discussion, highlighting the gap in evidence for all measures of AI software to inform cost-effectiveness analysis at this time.

Four studies35–38 from the cost-effectiveness review looking at CXR referral from primary care referrals without AI software informing model parameters, but all had limitations on their applicability due to study type and reported outcomes.

The use of hospital-reported data to conduct retrospective studies shows promise to provide good-quality information on outcomes under current CXR review practices without AI software. The reporting of consistently defined, key clinical pathway outcomes by standardised CXR report codes would allow a comparison between studies and provide more straightforward translation for use in cost-effectiveness modelling.

Coordinated research efforts are required to generate research on all outcome measures identified in for inclusion in the conceptual model. Evidence needs to demonstrate impact on intermediate outcomes over a sustained period of time in the NHS environment to account for differences in outcomes due to the widespread variation in current practices and pathways between individual hospitals sites and trusts. This can be achieved through well-designed studies, with large sample sizes, conducted over a sufficient period to capture the main outcomes of interest. This would reduce the reliance on evidence linkage, which remains particularly weak with regard to impact on stage at diagnosis.

Question 4

What would a health economic model to estimate the cost-effectiveness of adjunct AI to detect lung cancer look like?

This section describes a conceptual model developed by the EAG to identify the structure and components required in any future health economic models estimating the cost-effectiveness of adjunct AI compared with radiologist or reporting radiographer review alone of CXR images to detect lung cancer.

The proposed structure is suitable for both symptomatic and incidental primary care populations referred by their GP for CXR, with certain model parameters varying where appropriate for the specific population. For use in decision-making in the UK setting, an NHS and PSS perspective is adopted.

The conceptual model follows the illustrative pathways shown in Figure 7.

FIGURE 7.

Illustrative model structure for the detection of lung cancer. The illustrative pathway for CXR review by radiology specialist with adjunct AI software is identical to the structure presented here for CXR review by radiology specialist alone. If AI was used for triage, an additional step prior to the CXR could be included.

Strategies

For people undergoing a CXR, the CXR image is read by either a radiology specialist alone (current usual practice) or a radiology specialist with adjunct AI software.

Proposed model structure

A decision tree structure is used to depict the pathway from CXR imaging and review to point of diagnosis. We considered a decision tree structure appropriate to capture the short-term costs and benefits associated with the strategies used to identify people with lung cancer.

A positive CXR result (findings suspicious of lung cancer) follows pathway A, where a CT scan confirms the positive result and provides provisional staging. A utility decrement is applied to a positive result lasting until treatment. Treatment according to stage at diagnosis then commences when utility values for that stage are attributed for true-positive cases. False-positive cases revert to general population utility values.

People with false-negative results follow pathway B, C, D or E depending on whether findings are reported as ‘normal’ or ‘abnormal (alternative diagnosis)’. Either these people eventually undergo a CT scan as part of further clinical investigations along their respective pathways or they are assumed to present at an emergency department later. Any false negatives not detected at first CT scan along any pathway are also assumed to present later as an emergency. These pathways are longer than the most direct route to diagnosis (pathway A), and it is assumed that the delay in time to diagnosis confers a stage shift for a proportion of these people. Treatment then commences by stage at diagnosis.

People who receive a false-positive result at CXR imaging also follow pathway A and go on to receive a CT scan as a minimum further investigation, with a proportion undergoing further testing (e.g. PET scan, biopsy, bronchoscopy) until a true-negative lung cancer diagnosis is confirmed. A temporary utility decrement is applied for a false-positive test result for the duration until a confirmatory test is received showing no lung cancer present. A utility decrement associated with further invasive diagnostic procedures (biopsy and bronchoscopy) is applied to people with true-positive results and a proportion of those with false-positive results.

As for those people with false-negative results, people with true-negative results follow pathway B, C, D or E depending on whether findings are reported as ‘normal’ or ‘abnormal (alternative diagnosis)’. Additional testing is specific to each pathway.

Pathways A, B, C, D and E are described in detail in Clinical pathway for representation in model. Within the model, separate costs and health-related quality-of-life (HRQoL) outcomes are assigned to each pathway. All pathways that lead to a diagnosis of lung cancer complete the decision tree at a fast-track lung cancer clinic. Total costs and HRQoL outcomes (expressed as QALYs) to point of diagnosis are accrued according to the proportion of people assigned to each pathway as a result of CXR review by the two strategies under comparison.

At the end of the decision tree branches, long-term treatment costs and utility values over a 5-year time horizon are assigned based on stage of lung cancer at diagnosis. These are added to those accumulated during the diagnostic component of the model to provide overall outcomes for each strategy.

The results of a subsequent analysis (in a fully executable model) would be presented in terms of an ICER, where the difference between total costs of CXR review by radiology specialists with and without adjunct AI is divided by the difference between total QALYs for each, to give a cost-per-QALY figure. Prices would be based on the current cost year, with discounting of cost and outcomes applied at 3.5% over the total model time horizon in line with the NICE reference case. 18

For the conceptual model, information is required about the prevalence of lung cancer and the performance of radiology specialists to detect findings indicative of lung cancer on review of CXRs both with and without AI used as an adjunct. These inputs are specific to the population of interest, so figures are required for prevalence and diagnostic accuracy in both symptomatic and incidental primary care populations.

Prevalence figures used in the literature are sourced by Bajre et al. 32 for use in their economic evaluation from Field et al. 44 and by Geppert et al. 45 for modelling in the DAP060 AI for chest CT diagnostic assessment review from Horeweg et al. 46 Both sources44,46 contain estimates of lung cancer prevalence in the screening population. For modelling purposes in Bajre et al. 32 and Geppert et al.,45 these prevalence estimates are assumed to be the same for their population of interest. The EAG did not find any more relevant sources, but searches were not exhaustive and more recent estimates of prevalence in the UK population would be advisable for use in future modelling.

For the specific clinical pathways (A, B, C, D and E), people may follow through the decision tree, information on costs and resource use of diagnostic tests and clinical management input is required. The proportion of people taking each pathway and the mean time from initial CXR to diagnosis is also required for each of these pathways, under each strategy.

An example using clinical pathway A

Chest X-ray findings suggestive of malignancy are flagged to secondary care lung cancer teams who request the CT scan and await the formal referral from the patient’s GP to the suspected lung cancer clinic. Once reported, CT scans are triaged by lung cancer team consultants. If the scan suggests probable lung cancer, an urgent lung cancer team appointment is arranged with appropriate tests, for example lung function tests and planned biopsy (endobronchial ultrasound) (Alberto Alonso, personal communication). Diagnosis, histopathological staging and treatment options are then discussed at the fast-track lung cancer MDT clinic. 16

This process incurs the cost per person of a CT scan (£153), lung function tests (£285) and biopsy (£1670). 47

Input to direct these further tests is required by the secondary care lung cancer team on two occasions: (1) to review CXR results, refer for CT scan and notify the GP to make a suspected lung cancer pathways referral; and (2) to review CT scan results and refer for lung function tests and biopsy prior to fast-track lung cancer MDT clinic review. The unit cost of a lung cancer MDT meeting (£146),47 or part thereof, would be assigned for encounters with the lung cancer team and the fast-track clinic team. Average times to discuss a case during these meetings are necessary for more accurate costing.

Similarly, utility decrements are also assigned to pathway A. Suspicious lung cancer findings on CXR attract a disutility of −0.06348 applied over the length of time until confirmatory diagnosis. A disutility of −0.2 is applied for biopsy investigation for a period of 3 months. 49,50

The total costs and QALYs accrued are then attributed to the proportion of people in the model who take pathway A as a true-positive or false-positive case (Table 8).

The conceptual model presented captures the following important outcomes in the diagnostic process.

Clinical outputs from the model:

• number of false positives

• number of additional CT scans

• number of people referred for follow-up CXR

• number of people identified as ‘normal’ (no lung cancer present) and discharged

• number of cancers missed and detected

• proportion of cancers detected at each stage.

Long-term outcomes from the model:

• total costs per strategy

• total QALYs per strategy

• costs per QALY.

These outcomes would be based on a cohort of 1000 patients entering the model.

Question 5

What are the cost and resource use considerations relating to the use of adjunct artificial intelligence to detect lung cancer?

This section identifies the costs and resource use of adding AI software to CXR review taking an NHS and PSS perspective. Costs are required of each AI software, costs of training staff to use software, resource use and costs associated with healthcare professional time to read and report CXR, and costs of diagnostic testing and treatment. All costs are presented in 2021 prices. Costs obtained from the literature were uprated to current prices using the Hospital and Community Health Services index from Unit Costs of Health and Social Care 2021. 51 Cost categories are listed with resource use considerations discussed alongside and any potential sources of information identified.

Cost of software

AI software costs were obtained directly from the companies. Five of the 14 companies identified in the final scope9 were registered as stakeholders in this EVA and provided cost information to the EAG via NICE communications (Annalise AI, Behold AI, Infervision, Lunit Inc. and Siemens Healthineers).

Pricing structures were either fixed annual subscription fees (Annalise AI, Behold AI, Infervision and Siemens Healthineers) or volume-based annual pricing tiers (Infervision and Lunit Inc.). All companies charge a one-off implementation fee in the first year, which covers installation, integration with existing Picture Archive and Communication System/Radiology Information System and staff training. Ongoing subscription costs are renewable on an annual basis, with fees covering software licensing, annual maintenance, support and updates. Pricing is calculated per trust by Annalise AI, Infervision, Lunit Inc. and Siemens Healthineers. By contrast, Behold AI’s implementation and subscription fees are per hospital, with a 30,000 annual CXR volume allocation. (Confidential information has been removed.)

The annual subscription cost depends on the volume of CXRs to be processed in either each trust (Annalise AI, Infervision, Lunit Inc. and Siemens Healthineers) or each hospital (Behold AI) annually. The resource use would, therefore, be determined by the number of primary care referrals for CXR for the symptomatic and incidental populations per year.

This information is available through trust databases and has been reported in the literature through retrospective database studies. 35,36

Table 9 shows the disaggregated costs of AI software by company based on a number of 25,000 CXR images per NHS trust.

Cost of staff training

Staff training is provided by the AI software companies and the cost is included in the one-off implementation fee (see Cost of software). Companies reported that the training time for radiologists/reporting radiographers was 1 hour for Lunit and 30 minutes for Infervision. For Behold AI, no training time was given. Instead, the company advised that a training deck is customised for each trust and used to train designated trainers from each organisation, and then the deck is given to the trainers so that they can provide training to their radiologists.

Under the assumption that training is undertaken during protected staff-training time within radiology departments, no further costs would be attributed beyond the implementation fee.

Cost of staff time to read and report chest X-ray

The hourly cost of a radiologist or reporting radiographer was obtained from the literature. Two methods were identified that had been used in previous economic evaluations. 32,52 In the first of these, Bajre et al. 32 used the figure of £156 per hour for a radiologist and £53 for a band 7 reporting radiographer. This was originally calculated by Lockwood52 based on salary, on-costs and education for the 2015–6 cost year. In the second economic evaluation, the hourly cost of a band 9 radiographer (£147) from Unit Costs of Health and Social Care 202153 was used as a proxy for a radiologist. 45

The cost of staff time to read and report a single CXR can then be calculated using TAT. Published evidence has suggested no statistically significant difference in reading times of CXRs between readers with and readers without AI. 24,25 However, these data are from two studies24,25 that do not meet the inclusion criteria for the present EVA, and are of uncertain applicability to clinical practice (see What are the practical implications of adjunct artificial intelligence software to detect lung cancer on chest X-rays?).

Feedback from the SCM suggested timings without the use of AI from 1 minute on average, faster for normal and slower for very abnormal, up to 5 minutes. From the literature, Bajre et al. 32 assumed a 2-minute reporting time for both radiologists and reporting radiographers.

Cost of further diagnostic tests

Following the initial CXR, further testing may be required. This could include additional CXR, CT scan of chest, CT scan of abdomen (performed with or without contrast), PET scan, bronchoscopy and biopsy, with various combinations of each possible.

To direct these further tests, clinical input from the GP, respiratory specialists, radiologists and appropriate MDTs is required. The costs of these services can be obtained from National Schedule of NHS Costs 2020/2147 and Unit Costs of Health and Social Care 2021. 53

The costs of further tests depend on outcomes along the clinical pathway, including the:

• number of people referred for a CT scan

• number of people referred for follow-up CXR

• number of people identified as ‘normal’ (no lung cancer present) and discharged

• number of cancers missed/detected

• stage of cancer at detection.

No evidence was identified in the clinical search that addressed these outcomes as a result of AI software assistance in the reading of CXRs. We therefore have no evidence with which to determine whether the use of adjunct AI will increase, decrease, or not affect the number of people requiring additional testing.

Cost of treatment (including costs of any adverse events)

Total treatment costs are assigned according to stage of disease.

Several sources were identified in the literature. Bajre et al. 32 and Geppert et al. 45 used Cancer Research UK 201454 values originally reported in the 2014–5 price year and included cost of retreatment.

Snowsill et al. 33 used figures based on a 2-year costing approach, with index year costs from a UK teaching hospital55 and second year costs estimated from the index year using a subsequent year ratio from database analysis in England. 56 The same authors in an interim update to the UK National Screening Committee34 also used a 5-year microcosting approach with resource use based on the most recent National Lung Cancer Audit secondary care estimates for those in the 55 to 75 years age range to reflect more modern available treatment options, including immunotherapies.

Table 10 summarises the costs required for the proposed model.

For use in any future modelling, all sources where costs are obtained from the literature will need to be uprated to current prices at the time using the Hospital and Community Health Services index from the most recent version of the Unit Costs of Health and Social Care.

Summary

Potential sources to inform all unit costs for the cost parameters in the conceptual model proposed by the EAG have been identified. Primarily, these costs can be obtained from the literature and published national index costs and directly from AI software companies.

Evidence to support resource use relating to adjunct AI to detect lung cancer was not identified. Therefore, the total values of cost inputs for all cost parameters could not be calculated.

No evidence was found to determine what, if any, effect AI will have on resource use and in what direction this might take with respect to costs. At this stage, all we are able to determine is that AI represents a new cost, as AI software needs to be purchased and used in addition to the costs and resources consumed in the current clinical pathway.

Summary

Budget impact results vary greatly between companies, but the EAG cautions against direct comparison, as the AI software presented has varying capabilities, and some may be used in different positions early in the diagnostic pathway. For example, Siemens AI software points to a region of interest on the CXR, whereas Annalise AI software identifies a specific location, gives characteristics of the anomaly on CXR and provides a preliminary diagnosis and rating of confidence when used in a concurrent review of images. Similarly, Behold AI and Annalise AI software can provide a triage of CXR images prior to radiology specialist review in order to prioritise reporting, as well as assist with the detection of abnormalities and diagnosis. These differing capabilities may affect the way the AI software is used in practice, with a variety of practical, clinical and cost implications later in the diagnostic pathway. Therefore, without future modelling it is unclear how budget impact estimates for different AI software brands might be comparable.

Test accuracy, practical implications and clinical effectiveness

No studies met the review inclusion criteria. There is currently no evidence, applicable to this review, on the use of adjunct AI software for the detection of suspected lung cancer on CXR in either people referred from primary care with symptoms of lung cancer or people referred from primary care for other reasons. This finding, however, satisfies the secondary aim of this review, which was to identify evidence gaps in this field and inform future research. This is discussed in more detail below.

To provide context to the decision problem, summary results were presented from six studies that did not meet the review inclusion criteria because of unclear populations but were selected for discussion post hoc. The referral status and symptom status of the study participants are unknown, but the studies did provide comparisons of CXRs read by radiologists with and without the use of commercial AI software. Few outcomes were reported in these studies. They provide some insight into two of the key questions of this EVA:

1. What is the test accuracy and test failure rate of adjunct AI software to detect lung on CXR?

2. What are the practical implications of adjunct AI software to detect lung cancer on CXR?

None of the studies provided evidence for the clinical effectiveness of adjunct AI software applied to CXR (question 3).

For question 1, one study reported a higher sensitivity for lung cancer detection by readers with adjunct AI compared with readers alone, with no difference in specificity or cancer detection rate. 22 In the four studies for which data are available for publication, no significant between-group differences were found in test accuracy metrics in relation to lung nodules. 23–26

For question 2, no significant between-group differences were found for reading time24,25 or hypothetical referrals for CT scan. 22,24 Data from one study indicated that clinicians generally responded positively to the use of AI software. 22

This synopsis of study results is illustrative only of the type of evidence that is currently available on commercial AI to aid the interpretation of CXR. Caution is required in extrapolating from these studies as not only did they not meet the review inclusion criteria, but there were also differences between the studies, and limitations within them. For example, some studies included nodules with differing levels of detection difficulty from easy to challenging, while others excluded images on which nodules were below a certain size; studies used retrospective designs; data were reported from the mean performance across several readers with varying degrees of experience; readers had their findings from the first reading present at the time of the second reading and there was a lack of detailed reporting of key results. There were also differences in the reference standards, making comparisons between studies difficult. Furthermore, generalisability to the UK primary care referred population is unclear in all six summarised studies, and generalisability to the UK population overall is likely to be low in three studies that were conducted in the Republic of Korea.

Conceptual cost-effectiveness modelling

The conceptual modelling process aimed to explore both the structure and evidence requirements for parameter inputs for future model development. There was no evidence available on AI software impact on any of the intermediate outcomes identified to inform parameterisation. Results of EAG searches for evidence to inform these outcomes for the comparator alone (i.e. radiology specialist review of CXR in the detection of lung cancer in the primary care population) varied by study design and the way outcome measures were reported, limiting the way data could be used.

A simplistic model structure was outlined due to the paucity of evidence and tentative links to long-term outcomes.

Key points:

• Artificial intelligence needs to show changes to intermediate outcomes over a sustained period of time in the NHS environment as pathway variation and clinical practice/structure in radiology departments vary considerably between trusts and individual sites. Unless evidence is produced that is statistically powered to account for a difference in outcomes due to the current variation, evidence linkage to improved outcomes that may demonstrate cost-effectiveness cannot be made. Ideally this would be in the form of a large-scale, multisite, UK-based clinical trial with AI software as an adjunct to radiology specialist review compared directly with existing practice.

• It is not clear that evidence to suggest stage shift in detection of lung cancer can be achieved through CXR identification of suspected lung cancer in any event.

Strengths

• Extensive searches were undertaken, including electronic databases, existing reviews, company submissions and known studies, which reduced the risk of missing studies.

• Clinical experts were involved in the review and asked to provide details of any potentially eligible studies.

Limitations of the review/early value assessment process

• This review employed rapid evidence synthesis methods. 57 While this approach is used internationally by policy-makers to make expedited assessments of evidence,58,59 it is not without risks. In the present review, one reviewer conducted all elements of the review in full (i.e. title/abstracting sifting, full-text assessment), with a second reviewer assessing/checking 20% of each review task. Therefore, 80% of review tasks were only conducted by a single reviewer. Any errors made by the first reviewer relating to this 80% would not have been detected. As a result, there is the possibility that eligible studies were missed.

• We only searched for and included studies published in the English language. Therefore, we do not know if there are relevant papers in other languages.

• Targeted searches were used to retrieve a manageable number of records to screen. Therefore, it is possible that some studies (e.g. broad reviews) were not retrieved. To counter this, we used different combinations of concepts, sources and search methods and tested the overall search strategy’s ability to retrieve a set of known studies (found by a variety of methods during the scoping stage).

• Owing to the abridged timescale and limitations in resource for the evidence reviewing processes of this pilot EVA, there was no opportunity to follow up any uncertainties in studies with their authors or to seek further clarification of responses received from the few companies that provided submissions. Additional time was required to clarify the complex eligibility criteria in the scope before the protocol was signed off, and this also impacted on the reviewing timescale.

• As no studies met the eligibility criteria for the review, a pragmatic decision was taken following discussions with NICE to apply additional criteria to the excluded studies to select evidence closest to the review eligibility criteria. This selection process was iterative and involved discussion between two reviewers but was undertaken in the absence of a priori defined criteria. As already discussed, studies summarised were those where the population referral route and symptom status for the CXR were unknown (not reported). These populations are likely to be no different from those in other excluded studies with better descriptions of their populations. In addition, only summary results were extracted, and no formal risk-of-bias tool was applied to these studies. These results are illustrative only, and the results do not provide evidence on the use of adjunct AI software for the detection of suspected lung cancer on CXR in people referred from primary care.

• The selection of cost-effectiveness studies was undertaken by one reviewer, with wide inclusion/exclusion criteria aimed at the pragmatic identification of literature to support the development of a conceptual model and inform a rudimentary BIA of AI software in the NHS in the UK. Therefore, it is possible that without the rigorous methodology of systematic review processes (including quality appraisal), there may be biases from an individual reviewer, and studies identified in this report may not be fully representative of all those available. Through additional searches of references lists of identified studies, publication bibliographies of relevant authorship, several targeted searches and liaison with specialist committee members and clinical experts, the EAG endeavoured to mitigate the risk of missing pertinent evidence for this report.

Limitations of evidence base

This review found no applicable evidence on which to assess AI software for analysing CXR to identify suspected lung cancer among people referred from primary care.

Totals

Total from databases: 3879

Total after duplicates removed: 3049

MEDLINE (via Ovid)

Searched 25 November 2022

Ovid MEDLINE^® ALL 1946 to 23 November 2022

1. exp artificial intelligence/ or exp machine learning/ or exp deep learning/ or exp supervised machine learning/ or exp support vector machine/ or exp unsupervised machine learning/ 160,931

2. ai.kf,tw. 39,919

3. ((artificial or machine or deep) adj5 (intelligence or learning or reasoning)).kf,tw. 124,190

4. exp Neural Networks, Computer/ 53,917

5. (neural network* or convolutional or CNN or CNNs).kf,tw. 90,349

6. exp Diagnosis, Computer-Assisted/ 86,384

7. Pattern Recognition, Automated/ 26,362

8. ((automat* or autonomous or computer aided or computer assisted) adj3 (detect* or identif* or diagnos*)).kf,tw. 33,565

9. (support vector machine* or random forest* or black box learning).kf,tw. 37,636

10. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 [AI] 396,688

11. exp Radiography, Thoracic/ 40,528

12. X-Rays/ 31,129

13. (((chest or lung* or thora*) adj3 (radiograph* or radiogram* or radiology or roentgen* or x-ray* or xray* or film*)) or CXR*).kf,tw. 66,459

14. 11 or 12 or 13 [CXR] 121,772

15. 10 and 14 [AI and CXR] 3865

16. limit 15 to “reviews (best balance of sensitivity and specificity)” [AI and CXR and Reviews] 349

17. (metaanalys* or meta analys* or NMA* or MAIC* or indirect comparison* or mixed treatment comparison*).mp. 288,007

18. (systematic* adj3 (review* or overview* or search or literature)).mp. 328,557

19. 17 or 18 459,498

20. 15 and 19 [AI and CXR and SRs] 40

21. 16 or 20 [AI and CXR and Reviews/ SRs] 360

22. exp Lung Neoplasms/ or Solitary Pulmonary Nodule/ 268,336

23. ((lung or lungs or pulmon* or intrapulmon* or bronch*) adj3 (abnormal* or nodul* or lesion* or mass or masses or cancer* or neoplas* or tumor* or tumour* or carcino* or malignan* or adenocarcinom* or blastoma*)).kf,tw. 326,364

24. ((pancoast* or superior sulcus or pulmonary sulcus) adj4 (tumor* or tumour* or syndrome*)).kf,tw. 946

25. (sclc or nsclc).kf,tw. 64,440

26. 22 or 23 or 24 or 25 [Lung Cancer/ Nodule] 398,150

27. 10 and 26 [AI and Lung Cancer/ Nodule] 6749

28. (metaanalys* or meta analys* or NMA* or MAIC* or indirect comparison* or mixed treatment comparison*).mp. 288,007

29. (systematic* adj3 (review* or overview* or search or literature)).mp. 328,557

30. 28 or 29 [SRs] 459,498

31. 27 and 30 [AI and Lung Cancer/ Nodule and SRs] 100

32. 10 and 14 and 26 [AI and CXR and Lung Cancer/ Nodule] 707

33. AI-Rad Companion Chest X-ray*.kf,tw,in. 1

34. Annalise CXR*.kf,tw,in. 1

35. Auto Lung Nodule Detection*.kf,tw,in. 0

36. ChestView*.kf,tw,in. 0

37. (Chest X-Ray Classifier* or Quibim*).kf,tw,in. 46

38. CheXVision*.kf,tw,in. 0

39. (ClearRead Xray* adj2 Detect).kf,tw,in. 0

40. InferRead DR Chest*.kf,tw,in. 0

41. JLD-02K*.kf,tw,in. 0

42. Lunit INSIGHT CXR*.kf,tw,in. 4

43. Milvue Suite*.kf,tw,in. 0

44. ChestEye Quality*.kf,tw,in. 0

45. (qXR* or Qure*).kf,tw,in. 6815

46. (red dot* or behold*).kf,tw,in. 1090

47. SenseCare-Chest DR Pro*.kf,tw,in. 0

48. VUNO Med-Chest X-Ray*.kf,tw,in. 0

49. (X1* and Visionairy Health).kf,tw,in. 0

50. 33 or 34 or 35 or 36 or 37 or 38 or 39 or 40 or 41 or 42 or 43 or 44 or 45 or 46 or 47 or 48 or 49 [Technology Names/ Companies] 7956

51. 50 and 14 [Technology Names/ Companies and CXR] 61

52. 50 and 26 [Technology Names/ Companies and Lung Cancer/ Nodules] 90

53. 51 or 52 [Technology Names/ Companies and CXR/ Lung Cancer/ Nodules] 136

54. 21 or 31 or 32 or 53 1190

55. limit 54 to english language 1134

56. limit 54 to no language specified 0

57. 55 or 56 1134

58. exp animals/ not humans.sh. 5,066,999

59. 57 not 58 1128

60. limit 59 to (comment or editorial or letter) 9

61. 59 not 60 1119

EMBASE (via Ovid)

Searched 29 November 2022

EMBASE Classic+EMBASE 1947 to 2022 week 47

1. exp artificial intelligence/ or exp machine learning/ 373,033

2. ai.kf,tw. 55,274

3. ((artificial or machine or deep) adj5 (intelligence or learning or reasoning)).kf,tw. 146,615

4. (neural network* or convolutional or CNN or CNNs).kf,tw. 108,457

5. computer assisted diagnosis/ or computer assisted radiography/ 44,996

6. ((automat* or autonomous or computer aided or computer assisted) adj3 (detect* or identif* or diagnos*)).kf,tw. 44,987

7. (support vector machine* or random forest* or black box learning).kf,tw. 46,703

8. 1 or 2 or 3 or 4 or 5 or 6 or 7 [AI] 530,438

9. exp thorax radiography/ 230,425

10. X ray/ 119,143

11. (((chest or lung* or thora*) adj3 (radiograph* or radiogram* or radiology or roentgen* or x-ray* or xray* or film*)) or CXR*).kf,tw. 107,803

12. 9 or 10 or 11 [CXR] 379,945

13. 8 and 12 [AI and CXR] 5577

14. limit 13 to “reviews (best balance of sensitivity and specificity)” [AI and CXR and Reviews] 657

15. (metaanalys* or meta analys* or NMA* or MAIC* or indirect comparison* or mixed treatment comparison*).mp. 414,514

16. (systematic* adj3 (review* or overview* or search or literature)).mp. 520,359

17. 15 or 16 695,121

18. 13 and 17 [AI and CXR and SRs] 117

19. 14 or 18 [AI and CXR and Reviews/ SRs] 678

20. exp lung tumor/ or lung nodule/ 495,858

21. ((lung or lungs or pulmon* or intrapulmon* or bronch*) adj3 (abnormal* or nodul* or lesion* or mass or masses or cancer* or neoplas* or tumor* or tumour* or carcino* or malignan* or adenocarcinom* or blastoma*)).kf,tw. 493,166

22. ((pancoast* or superior sulcus or pulmonary sulcus) adj4 (tumor* or tumour* or syndrome*)).kf,tw. 1328

23. (sclc or nsclc).kf,tw. 116,762

24. 20 or 21 or 22 or 23 [Lung Cancer/ Nodule] 655,493

25. 8 and 24 [AI and Lung Cancer/ Nodule] 12,931

26. (metaanalys* or meta analys* or NMA* or MAIC* or indirect comparison* or mixed treatment comparison*).mp. 414,514

27. (systematic* adj3 (review* or overview* or search or literature)).mp. 520,359

28. 26 or 27 [SRs] 695,121

29. 25 and 28 [AI and Lung Cancer/ Nodule and SRs] 313

30. 8 and 12 and 24 [AI and CXR and Lung Cancer/ Nodule] 1114

31. AI-Rad Companion Chest X-ray*.kf,tw,in. 1

32. Annalise CXR*.kf,tw,in. 1

33. Auto Lung Nodule Detection*.kf,tw,in. 0

34. ChestView*.kf,tw,in. 0

35. (Chest X-Ray Classifier* or Quibim*).kf,tw,in.57

36. CheXVision*.kf,tw,in. 0

37. (ClearRead Xray* adj2 Detect).kf,tw,in. 0

38. InferRead DR Chest*.kf,tw,in. 0

39. JLD-02K*.kf,tw,in. 0

40. Lunit INSIGHT CXR*.kf,tw,in. 6

41. Milvue Suite*.kf,tw,in. 0

42. ChestEye Quality*.kf,tw,in. 0

43. (qXR* or Qure*).kf,tw,in. 14,268

44. (red dot* or behold*).kf,tw,in. 1520

45. SenseCare-Chest DR Pro*.kf,tw,in. 0

46. VUNO Med-Chest X-Ray*.kf,tw,in. 0

47. (X1* and Visionairy Health).kf,tw,in. 0

48. 31 or 32 or 33 or 34 or 35 or 36 or 37 or 38 or 39 or 40 or 41 or 42 or 43 or 44 or 45 or 46 or 47 [Technology Names/ Companies] 15,850

49. 48 and 12 [Technology Names/ Companies and CXR] 267

50. 48 and 24 [Technology Names/ Companies and Lung Cancer/ Nodules] 234

51. 49 or 50 [Technology Names/ Companies and CXR/ Lung Cancer/ Nodules] 466

52. 19 or 29 or 30 or 51 2362

53. limit 52 to english language 2271

54. limit 52 to no language specified 1

55. 53 or 54 2272

56. animal experiment/ not (human experiment/ or human/) 2,472,698

57. 55 not 56 2263

58. limit 57 to (editorial or letter) 65

59. 57 not 58 2198

Cochrane Database of Systematic Reviews (via Wiley)

Search name: qXR EVA Reviews

Date run: 30 November 2022 19:30:29

ID Search Hits

1. [mh “artificial intelligence”] OR [mh “machine learning”] OR [mh “deep learning”] OR [mh “supervised machine learning”] OR [mh “support vector machine”] OR [mh “unsupervised machine learning”] 1540

2. ai:ti,ab,kw 5002

3. ((artificial OR machine OR deep) NEAR/5 (intelligence OR learning OR reasoning)):ti,ab,kw 3847

4. [mh “Neural Networks, Computer”] 217

5. ((“neural” NEXT network*) OR convolutional OR CNN OR CNNs):ti,ab,kw 1738

6. [mh “Diagnosis, Computer-Assisted”] 1943

7. [mh ^“Pattern Recognition, Automated”] 193

8. ((automat* OR autonomous OR “computer aided” OR “computer assisted”) NEAR/3 (detect* OR identif* OR diagnos*)):ti,ab,kw 2092

9. ((“support vector” NEXT machine*) OR (“random” NEXT forest*) OR “black box learning”):ti,ab,kw 935

10. #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 13,357

11. [mh “Radiography, Thoracic”] 363

12. [mh ^X-Rays] 59

13. (((chest OR lung* OR thora*) NEAR/3 (radiograph* OR radiogram* OR radiology OR roentgen* OR x-ray* OR xray* OR film*)) OR CXR*):ti,ab,kw 5878

14. #11 OR #12 OR #13 5948

15. #10 AND #14 120

16. [mh “Lung Neoplasms”] OR [mh ^”Solitary Pulmonary Nodule”] 8755

17. ((lung OR lungs OR pulmon* OR intrapulmon* OR bronch*) NEAR/3 (abnormal* OR nodul* OR lesion* OR mass OR masses OR cancer* OR neoplas* OR tumor* OR tumour* OR carcino* OR malignan* OR adenocarcinom* OR blastoma*)):ti,ab,kw 28,597

18. ((pancoast* OR “superior sulcus” OR “pulmonary sulcus”) NEAR/4 (tumor* OR tumour* OR syndrome*)):ti,ab,kw 17

19. (sclc OR nsclc):ti,ab,kw 12,248

20. #16 OR #17 OR #18 OR #19 29,193

21. #10 AND #20 348

22. #15 OR #21 421

Cochrane Reviews: 0

CENTRAL (via Wiley)

Search Name: qXR EVA Trials

Date run: 30 November 2022 22:52:13

Comment: 30 November 2022

ID Search Hits

1. [mh “artificial intelligence”] OR [mh “machine learning”] OR [mh “deep learning”] OR [mh “supervised machine learning”] OR [mh “support vector machine”] OR [mh “unsupervised machine learning”] 1540

2. ai:ti,ab,kw 5002

3. ((artificial OR machine OR deep) NEAR/5 (intelligence OR learning OR reasoning)):ti,ab,kw 3847

4. [mh “Neural Networks, Computer”] 217

5. ((“neural” NEXT network*) OR convolutional OR CNN OR CNNs):ti,ab,kw 1738

6. [mh “Diagnosis, Computer-Assisted”] 1943

7. [mh ^“Pattern Recognition, Automated”] 193

8. ((automat* OR autonomous OR “computer aided” OR “computer assisted”) NEAR/3 (detect* OR identif*OR diagnos*)):ti,ab,kw 2092

9. ((“support vector” NEXT machine*) OR (“random” NEXT forest*) OR “black box learning”):ti,ab,kw 935

10. #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 13,357

11. [mh “Radiography, Thoracic”] 363

12. [mh ^X-Rays] 59

13. ((chest OR lung* OR thora*) NEAR/3 (radiograph* OR radiogram* OR radiology OR roentgen* OR x-ray* OR xray* OR film* OR CXR*)):ti,ab,kw 5878

14. #11 OR #12 OR #13 5948

15. [mh “Lung Neoplasms”] OR [mh ^”Solitary Pulmonary Nodule”] 8755

16. ((lung OR lungs OR pulmon* OR intrapulmon* OR bronch*) NEAR/3 (abnormal* OR nodul* OR lesion* OR mass OR masses OR cancer* OR neoplas* OR tumor* OR tumour* OR carcino* OR malignan* OR adenocarcinom* OR blastoma*)):ti,ab,kw 28,597

17. ((pancoast* OR “superior sulcus” OR “pulmonary sulcus”) NEAR/4 (tumor* OR tumour* OR syndrome*)):ti,ab,kw 17

18. (sclc OR nsclc):ti,ab,kw 12,248

19. #15 OR #16 OR #17 OR #18 29,193

20. #10 and #14 and #19 47

21. (“AI-Rad Companion Chest” NEXT X-ray*) 0

22. (“Annalise” NEXT CXR*) 0

23. (“Auto Lung Nodule” NEXT Detection*) 0

24. ChestView* 0

25. ((“Chest X-Ray” NEXT Classifier*) OR Quibim*) 0

26. CheXVision* 0

27. ((“ClearRead” NEXT Xray*) NEAR/2 Detect) 0

28. (“InferRead DR” NEXT Chest*) 0

29. JLD-02K* 0

30. (“Lunit INSIGHT” NEXT CXR*) 2

31. (“Milvue” NEXT Suite*) 0

32. (“ChestEye” NEXT Quality*) 0

33. (qXR* OR Qure*) 921

34. ((“red” NEXT dot*) OR behold*) 71

35. (“SenseCare-Chest DR” NEXT Pro*) 0

36. (“VUNO Med-Chest” NEXT X-Ray*) 1

37. (X1* AND “Visionairy Health”) 0

38. #21 OR #22 OR #23 OR #24 OR #25 OR #26 OR #27 OR #28 OR #29 OR #30 OR #31 OR #32 OR #33 OR #34 OR #35 OR #36 OR #37 995

39. #14 and #38 4

40. #19 and #38 7

41. #39 or #40 8

42. #20 or #41 53

Trials: 52

Epistemonikos

Searched 1 December 2022

(title:((“AI” OR “artificial intelligence” OR “artificial learning” OR “artificial reasoning” OR “machine intelligence” OR “machine learning” OR “machine reasoning” OR “deep intelligence” OR “deep learning” OR “deep reasoning” OR “neural network” OR “neural networks” OR “neural networking” OR convolutional OR “CNN” OR “CNNs” OR ((automat* OR autonomous OR “computer aided” OR “computer assisted”) AND (detect* OR identif* OR diagnos*)) OR “support vector machine” OR “support vector machines” OR “support vector network” OR “support vector networks” OR “random forest” OR “random forests” OR “black box learning”) AND ((((chest OR lung* OR thora*) AND (radiograph* OR radiogram* OR radiology OR roentgen* OR x-ray* OR xray* OR film*)) OR CXR*) OR ((lung OR lungs OR pulmon* OR intrapulmon* OR bronch*) AND (abnormal* OR nodul* OR lesion* OR mass OR masses OR cancer* OR neoplas* OR tumor* OR tumour* OR carcino* OR malignan* OR adenocarcinom* OR blastoma*)) OR ((pancoast* OR superior sulcus OR pulmonary sulcus) AND (tumor* OR tumour* OR syndrome*)))) OR abstract:((“AI” OR “artificial intelligence” OR “artificial learning” OR “artificial reasoning” OR “machine intelligence” OR “machine learning” OR “machine reasoning” OR “deep intelligence” OR “deep learning” OR “deep reasoning” OR “neural network” OR “neural networks” OR “neural networking” OR convolutional OR “CNN” OR “CNNs” OR ((automat* OR autonomous OR “computer aided” OR “computer assisted”) AND (detect* OR identif* OR diagnos*)) OR “support vector machine” OR “support vector machines” OR “support vector network” OR “support vector networks” OR “random forest” OR “random forests” OR “black box learning”) AND ((((chest OR lung* OR thora*) AND (radiograph* OR radiogram* OR radiology OR roentgen* OR x-ray* OR xray* OR film*)) OR CXR*) OR ((lung OR lungs OR pulmon* OR intrapulmon* OR bronch*) AND (abnormal* OR nodul* OR lesion* OR mass OR masses OR cancer* OR neoplas* OR tumor* OR tumour* OR carcino* OR malignan* OR adenocarcinom* OR blastoma*)) OR ((pancoast* OR superior sulcus OR pulmonary sulcus) AND (tumor* OR tumour* OR syndrome*)))))

Publication type:

Systematic review: 44

Broad synthesis: 1

ACM Digital Library

Searched 1 December 2022

Search for reviews

https://dl.acm.org/search/advanced

Selected ACM Guide to Computing Literature

Title:(((“AI” OR “artificial intelligence” OR “artificial learning” OR “artificial reasoning” OR “machine intelligence” OR “machine learning” OR “machine reasoning” OR “deep intelligence” OR “deep learning” OR “deep reasoning” OR “neural network” OR “neural networks” OR “neural networking” OR convolutional OR “CNN” OR “CNNs” OR (automat* OR autonomous OR “computer aided” OR “computer assisted”) AND (detect* OR identif* OR diagnos*) OR “support vector machine” OR “support vector machines” OR “support vector network” OR “support vector networks” OR “random forest” OR “random forests” OR “black box learning”) AND ((((chest OR lung* OR thora*) AND (radiograph* OR radiogram* OR radiology OR roentgen* OR x-ray* OR xray* OR film*)) OR CXR*) OR ((lung OR lungs OR pulmon* OR intrapulmon* OR bronch*) AND (abnormal* OR nodul* OR lesion* OR mass OR masses OR cancer* OR neoplas* OR tumor* OR tumour* OR carcino* OR malignan* OR adenocarcinom* OR blastoma*)) OR ((pancoast* OR “superior sulcus” OR “pulmonary sulcus”) AND (tumor* OR tumour* OR syndrome*))))) OR Abstract:(((“AI” OR “artificial intelligence” OR “artificial learning” OR “artificial reasoning” OR “machine intelligence” OR “machine learning” OR “machine reasoning” OR “deep intelligence” OR “deep learning” OR “deep reasoning” OR “neural network” OR “neural networks” OR “neural networking” OR convolutional OR “CNN” OR “CNNs” OR (automat* OR autonomous OR “computer aided” OR “computer assisted”) AND (detect* OR identif* OR diagnos*) OR “support vector machine” OR “support vector machines” OR “support vector network” OR “support vector networks” OR “random forest” OR “random forests” OR “black box learning”) AND ((((chest OR lung* OR thora*) AND (radiograph* OR radiogram* OR radiology OR roentgen* OR x-ray* OR xray* OR film*)) OR CXR*) OR ((lung OR lungs OR pulmon* OR intrapulmon* OR bronch*) AND (abnormal* OR nodul* OR lesion* OR mass OR masses OR cancer* OR neoplas* OR tumor* OR tumour* OR carcino* OR malignan* OR adenocarcinom* OR blastoma*)) OR ((pancoast* OR “superior sulcus” OR “pulmonary sulcus”) AND (tumor* OR tumour* OR syndrome*)))))

Filter by

Content type:

Review article: 12

Systematic review register: search summary

Trials registers: search summary

Cost-effectiveness searches