Software with artificial intelligence-derived algorithms for analysing CT brain scans in people with a suspected acute stroke: a systematic review and cost-effectiveness analysis

The condition

Stroke is a serious life-threatening medical condition defined by the World Health Organization (WHO) as a clinical syndrome consisting of ‘rapidly developing clinical signs of focal (at times global) disturbance of cerebral function, lasting more than 24 hours or leading to death with no apparent cause other than that of vascular origin’. 1 Stroke can occur without any warning and leads to interruption or restriction of the blood flow to the brain causing reduction of the flow of oxygen and nutrients to the brain and subsequently brain cell death. The effects of a stroke depend on which area of the brain is affected, the extent of damage and the time to treatment. 2

There are two main types of stroke:

• Ischaemic stroke – the most frequently occurring type of stroke resulting from reduced blood flow due to arterial occlusion. Approximately 87.1% of patients in the United Kingdom (UK) will suffer from this type of stroke. Arterial blockage can be caused by the formation of atherosclerotic plaques (fatty deposits building up in the walls of arteries). As well as narrowing the artery, making it harder for blood to pass through it, the fatty deposits can break down or become inflamed. When this happens, a blood clot forms, which can block the artery, or a clot can travel from a distant location, such as from the heart or blood vessels in the neck and block the blood vessel in the brain (embolisation); the majority of the ischaemic strokes are caused by this mechanism rather than in situ thrombosis. Other causes of ischaemic stroke are small-vessel disease leading to vessel damage, heart conditions such as atrial fibrillation (AF), patent foramen ovale, endocarditis or arterial dissection. 2,3

• Haemorrhagic stroke, also referred to as intracranial haemorrhage (ICH) or cerebral haemorrhage, accounts for approximately 12.5% of all strokes in the UK and is caused by bleeding from blood vessels in or around the brain. This type of stroke can be intracerebral (bleed within the brain) or subarachnoid (bleed on the surface of the brain in the subarachnoid space). Intracerebral haemorrhagic stroke is most associated with high blood pressure, resulting in the bursting of an artery, whereas subarachnoid haemorrhagic stroke is most frequently caused by a burst aneurysm. 2,3

A transient ischaemic attack (TIA), sometimes known as a mini stroke, is differentiated from ischaemic stroke in that symptoms are time limited/self-resolving. Patients who have experienced one or more TIAs are at increased risk for ischaemic stroke. 2 The diagnosis of TIA is not considered in this assessment.

In 2018–19, there were 224,172 hospital admissions for stroke (including stroke mimics) in the UK and the in-hospital crude mortality rate for 2017–19 was reported to be 13.4%. 4 In the same year, there were over 1.2 million stroke survivors in the UK with stroke prevalence (defined as patients who have had a stroke or TIA on a general practice register) ranging from 1.77% in England to 2.28% in Scotland. 5

Symptoms and risk factors

Common symptoms include drooping of one side of the face, problems with speaking and vision, loss of sensation in an arm or leg and slurred or garbled speech. Other symptoms can include nausea, vomiting, vertigo and decreased level of consciousness. 2

The Sentinel Stroke National Audit Programme (SSNAP), the UK national healthcare quality improvement programme, collects patient data from England, Wales and Northern Ireland and provides information on patient characteristics and outcomes, and the infrastructure of stroke services. Among 89,280 stroke patients for whom data were collected between April 2019 and March 2020, the median age of patients with acute stroke in the UK was 77 years. 3 The risk of stroke increases with age due to continuous changes in brain arteries. 2 Females accounted for 48% of all acute stroke patients in the UK. 3

It is estimated that approximately 90% of strokes are attributable to risk factors that can be modified during a patient’s lifetime (e.g. management of high blood pressure, diabetes, changes in smoking habits and addressing physical inactivity). 2 According to SSNAP, 55.1% and 22.5% of acute stroke patients in the UK, respectively, suffered from hypertension and diabetes before their stroke. 3

Diagnosis and treatment

Timely and effective management of patients with suspected stroke substantially impacts their outcomes. As stroke mimics account for approximately 20–25% of all acute presentations, the patient’s history is crucial to establish the potential cause of their symptoms and to avoid misdiagnosis. 6

Outside the hospital setting, patients with suspected stroke should be assessed using the ‘face, arm, speech’ test (FAST) and they must be transported to the hospital as quickly as possible, preferably to a stroke unit. 7 Specialised stroke units are trained in the management of stroke patients and have access to specialist medical staff, diagnostic imaging equipment, time-sensitive procedures such as thrombectomy and thrombolysis and other services. In the UK, these units are known as comprehensive stroke centres (CSCs), defined as centres providing hyperacute, acute and inpatient rehabilitation including thrombectomy and neurosurgery services. Non-specialist units may be unable to provide access to specialist medical staff or some crucial medical procedures, which can affect the timely and effective selection and treatment of patients suffering from a stroke. In the UK, these units are known as acute stroke centres, defined as centres which provide hyperacute, acute and inpatient rehabilitation, but excluding thrombectomy and neurosurgery; all acute stroke centres are expected to have an intrahospital thrombectomy transfer pathway to transfer patients from acute stroke centres to CSCs.

In the emergency department, patients should be assessed with the Recognition of Stroke in the Emergency Room (ROSIER) scale. 7,8 After admission, a CT or a magnetic resonance imaging (MRI) brain scan should be performed at the next available imaging slot, within an hour from arrival, to rule out other causes of symptoms, to provide information on the potential cause and to show the extent of damage and decide on the best treatment option. 2 A CT scan is quick and effective method ruling out ICH, which is often sufficient to make thrombolysis decisions for patients with ischaemic stroke. However, the specificity of a CT scan might be compromised in patients with acute ischaemia because of continuing changes in the brain since the onset of symptoms. 6 Other tests may be needed, especially for patients with haemorrhagic stroke, to provide more information on the cause of stroke. In the UK, only 55.2% of patients with acute stroke are scanned within 1 hour from admission, with the numbers rising to 95.5% for a scan within 12 hours from patient admission. 3 Admission directly to a stroke unit and assessment by a stroke specialist can lead to improved patient outcomes and reduction in complications. Patients who are seen in a specialist stroke unit are also more likely to receive more targeted secondary care. 2 Based on the SSNAP, between April 2019 and March 2020, the stroke unit was the first ward of admission for 79.9% of acute stroke patients in the UK. 3 Some patients, however, may be initially transported to other units where direct specialist care is not available.

Patients with an ischaemic stroke can be treated with thrombolysis, which uses alteplase to dissolve the clot blocking the artery in the brain. 2 The shorter the time between symptom onset and thrombolysis, the higher a patient’s chance of better recovery; however, only a limited number of patients can benefit from this treatment due to the number of contraindications and potential complications that need to be considered. For stroke patients with unknown time of symptom onset, a 2021 systematic review showed that patients treated with alteplase thrombolysis had over three-times greater risk of symptomatic intracranial haemorrhage (sICH; an adverse effect of thrombolysis) when compared with patients receiving conservative medical treatment. There was no increase in the risk of death at 3 months, and patients had a similar likelihood of functional independence. 9 Treatment with alteplase is also associated with an increased risk of ICH, compared with conservative treatment, in patients with a clearly defined time of stroke onset. 10

Some patients with ischaemic stroke may benefit from thrombectomy (i.e. extraction of arterial obstruction with a device). Thrombectomy is considered if the obstruction is present in a large artery11 and has been shown to be superior to best medical therapy alone (e.g. thrombolysis alone) for patients with anterior circulation large artery occlusion. 6,12 In patients with an ischaemic stroke, thrombolysis can be administered before mechanical thrombectomy without an increase in the incidence of sICH or mortality at 90 days when compared with thrombectomy alone. Similarly, there is no difference between treatments (thrombolysis plus thrombectomy vs. thrombectomy alone) in the rates of successful recanalisation or the level of patients’ functional independence at 90 days. 13

Patients with haemorrhagic stroke require intensive blood pressure-lowering medications or reversal of antithrombotic medications at the early stages of their treatment. Patients may undergo surgery to seal a burst aneurysm or relieve the pressure on the brain. Severe headaches can be addressed with pain relief medication. 2

More information regarding the patient pathway, available treatments and patient eligibility for treatment in the NHS setting is provided in the Care pathway section.

icobrain CT

The neuroimaging platform icobrain® CT (icometrix, Leuven, Belgium) is a CE marked class 1 medical device that uses AI-derived algorithms to detect abnormalities in brain CT scans; icobrain CT can generate two output reports related to stroke diagnosis:

• Report 1, from icobrain CTP, details a quantitative assessment of perfusion in the brain based on a CT scan with contrast. It analyses the flow of blood in areas of the brain to determine the presence of potentially salvageable tissues in ischaemic stroke. The analysis includes a calculation of abnormality in parameters such as mean transit time, cerebral blood flow, cerebral blood volume and time to maximum of residue function.

• Report 2, from icobrain TBI, can give a quantitative assessment of ICH based on a non-enhanced CT scan. This report also has application in traumatic brain injury. Some of the NCCT parameters measured include midline shift and asymmetry index between the left and right lateral ventricle.

The company notes that its AI-derived neuroimaging platform integrates with existing RIS and PACS. The software is intended for automatic labelling, visualisation and volumetric quantification of segmentable brain structures from a set of CT images. It receives digital images as input and generates an electronic report on quantitative parameters and annotated images. Results can be viewed as visual reports through digital imaging and communication in medicine (DICOM) output images, e-mail notifications and on a web browser. The report highlights stroke-related changes that guide clinician diagnosis. Data from and into the PACS are transferred securely over a software icobridge, installed on site. icobrain CT has had two major releases, versions 4.0 and 5.0. The company notes that performance of icobrain in detecting ICH and for CTP analysis has been tested on a series of scenarios that cover specific aspects of the software performance. icobrain CT algorithms send and receive information over a secure cloud ‘icometrix’. Icometrix is ISO13485 and ISO27001 certified and UK General Data Protection Regulation (GDPR) and United States Health Insurance Portability and Accountability Act (HIPAA) compliant for privacy and security.

The company provides a training manual for health professionals, which gives guidance on how to use the software and interpret reports. Customer support is also available from the company. Prior to deployment in clinical practice the company carries out a clinical and technical test phase. icobrain CT is currently a self-certified class 1 medical device under the Medical Device Directive. The company notes that it will be up-classified to a class 2a medical device under the Medical Device Regulation, in line with the transition from the Medical Device Directive to the Medical Device Regulation.

Aidoc intracranial haemorrhage, Aidoc large-vessel occlusion, Aidoc mobile

The Aidoc® software, also called ‘BriefCase’ (Aidoc, Tel Aviv, Israel), is a CE marked class 1 medical device AI triage and notification platform. This neuroimaging platform uses AI-derived algorithms to detect abnormalities in brain CT scans. Algorithms related to stroke diagnosis include:

• Aidoc ICH for detecting suspected ICH on non-contrast head CT

• Aidoc LVO for detecting suspected LVOs on CTA.

The third component of the platform relevant to stroke diagnosis is Aidoc mobile, which is for communication between clinical stakeholders in the stroke pathway to facilitate peer review.

The company notes that its software can integrate with existing radiology workstations, including PACS, reporting system and radiology workflow solutions. The platform can prioritise worklist, triage and generate notification on suspected stroke cases. Analysis done by the AI-derived software is intended to supplement CT scan review by a neuroradiologist or stroke specialist.

The company provides an initial product training, which lasts around 30 minutes, and where necessary, additional training on specific workflows can be provided. Recurring annual training is also available to review new features, enhancements and algorithms. Prior to deployment of the software on a site, the company carries out an automated performance assessment through its AI operations centre. Aidoc is ISO13485 and ISO27001 certified. The Aidoc software is currently a self-certified class 1 medical device under the Medical Device Directive; the company notes that it will be up-classified to a class 2a medical device under the Medical Device Regulation, in line with the transition from the Medical Device Directive to the Medical Device Regulation.

icometrix and Aidoc ‘comprehensive stroke solution’

Aidoc and icometrix have partnered to provide a stroke solution in which the Aidoc software detects ICH and large-vessel occlusion (LVO) and the icobrain software is used for CTP analysis to detect ischaemic stroke. Figure 1 shows how the technologies are intended to be implemented in clinical practice.

FIGURE 1.

The icometrix and Aidoc ‘comprehensive stroke solution’ pathways.

Rapid Alberta Stroke Program Early Computed Tomography Score, Rapid ICH, Rapid CTA, Rapid LVO, Rapid CTP algorithms

RapidAI® (iSchemaView, Menlo Park CA, USA) is a CE marked class 2a medical device neuroimaging platform that uses AI-derived software for detecting abnormalities in brain CT scans. The CT algorithms relevant to stroke diagnosis are:

• Rapid ICH is an image processing software that analyses non-enhanced CT head scans to detect, and flag suspected ICH. Cases with suspected findings can be notified through e-mail and the mobile application. The notification includes compressed images that are for informational purposes only and not intended to be diagnostic. The notified clinician is responsible for viewing non-compressed images on a diagnostic viewer and carrying out necessary patient evaluation.

• Rapid CTA is an image processing software that analyses head CT angiograms scans to provide neurological vasculature maps with indications of hemispheric differences in the intracranial internal carotid artery (ICA)/middle cerebral artery (MCA) region, which may indicate a LVO.

• Rapid LVO is an image processing software that analyses head CT angiogram scans to highlight and notify cases with suspected LVO.

• Rapid CTP enables the assessment of salvageable brain tissue through the delivery of quantified and colour-coded CTP maps that identify brain regions with reduced cerebral blood flow, volume and transit time that exceed prespecified thresholds. Imaging data sets acquired from CT or cone beam CT or magnetic resonance (MR) perfusion and mismatch, MR diffusion, and CT/MR angiography are analysed to measure parameters that determine suitability for thrombectomy.

• RAPID Alberta Stroke Program Early CT Score (ASPECTS) is not intended for the primary interpretation of CT images. It assists the clinician in evaluating patients presenting for diagnostic imaging with known MCA or ICA occlusion, to assess the extent of disease on NCCT scans. Extent of disease refers to the number of ASPECTS regions affected. Image data and AI analysis of morphological features is used to generate a single ASPECTS. This score is useful in characterising early signs of brain ischaemia, areas of irreversible tissue injury and to help the clinician assess patient eligibility for thrombectomy or thrombolysis.

The RapidAI platform runs on a standard computer or a virtual platform, such as VMware® (VMware Inc., Palo Alto, CA, USA), and can be used to perform image viewing, processing and analysis. The software receives DICOM compliant images as input primarily CT, CTA, cone beam CT and MR. Results from on the Rapid platform can be viewed as visual reports through PACS, e-mail notifications and the Rapid mobile application. Notifications have a sound option for positive cases and can be set to user defined thresholds to enable prioritisation. Results from multiple sites can be viewed and organised in one location. RapidAI is ISO certified and complies with GDPR and data security requirements.

The company provides training, which includes online role-based product training, virtual instructor-led sessions led by clinical experts and performance support content.

e-Alberta Stroke Program Early Computed Tomography Score, e-computed tomography perfusion, e-computed tomography angiography

The e-Stroke platform (Brainomix Ltd, Oxford, UK) is a CE marked class 2a medical device neuroimaging platform that uses AI-derived software for detecting anomalies in brain CT scans. The platform includes the following algorithms relevant to stroke diagnosis:

• e-ASPECTS analyses NCCT scans for clot detection, signs of hypodensity and generates a heat map of regional ischaemic change, volume of the change and an automatic ASPECTS score.

• e-CTP analyses CTP scans to generate perfusion summary maps, report parameters, such as mismatch volume and ratio, hypoperfusion intensity ratio and assesses eligibility for mechanical thrombectomy.

• e-CTA analyses CTA scans to detect the location of LVOs and to generate a CT collateral score, which is used to assesses eligibility for mechanical thrombectomy.

The software integrates with current imaging systems and results can be viewed as visual reports through DICOM output images, e-mail notifications and a web browser.

Viz

The Viz platform (Viz.ai, Viz.ai Inc., San Francisco, CA, USA) is a CE marked class 1 medical device software that uses static AI-derived algorithms to detect abnormalities in brain scans in clinical practice. The algorithms relevant to stroke detection include:

• Viz LVO analyses CTA images of the brain and sends notification to the clinician if a suspected LVO has been detected. Notifications include compressed images that can be previewed for information purposes only. They are not intended to be diagnostic. The notified clinician is responsible for viewing non-compressed images on a diagnostic viewer and carrying out necessary patient evaluation.

• Viz ICH analyses NCCT images of the brain and sends notification to the clinician if a suspected ICH has been detected.

• Viz CTP has communication and analysis capabilities for CTP scans. The analysis includes the calculation of parameters related to tissue perfusion and tissue blood volume.

The company notes that the Viz platform integrates with currently available CT scanners and is designed to receive DICOM images, which can be transferred securely to Viz.ai’s GDPR-compliant Amazon Web Services cloud. Within the cloud, Viz.ai will analyse the imaging data for specific neurovascular disease. The platform can be used by hospital networks and trained clinicians.

The Viz platform is GDPR/HIPAA compliant and has ISO and SOC-2 certifications. Viz is currently a self-certified class 1 medical device under the Medical Device Directive, the company notes that it will be up-classified to a class 2a medical device under the Medical Device Regulation, in line with the transition from the Medical Device Directive to the Medical Device Regulation.

qER

qER (Qure.ai Technologies, Mumbai, India) is a CE marked triage and notification tool that detects and quantifies a range of brain abnormalities intracerebral bleeds and their subtypes, infarcts, mass effect, midline shift and cranial fractures following NCCT imaging. Based on information from AI for Radiology (https://grand-challenge.org/aiforradiology), qER currently has class 2a CE mark. The software populates a radiology reporting template with preliminary findings, patient prioritisation and alert systems including mobile notifications. It integrates with current imaging systems.

Zebra triage

Zebra-Med (Zebra Medical Vision, Shefayim, Israel) is a CE marked software that detects and annotates ICH after NCCT imaging and automates patient prioritisation and a real-time alert system. Based on information from AI for Radiology (https://grand-challenge.org/aiforradiology), Zebra-Med currently has class 2a CE mark. It integrates with the current imaging worklist and viewer with an accompanying alert widget.

Zebra Medical Vision was acquired by Nano-X Imaging in November 2021 and now operates as Nanox. The product is now called Neuro Solution.

CT Perfusion 4D neuro

CT Perfusion 4D Neuro® (GE HealthCare, Chalfont St Giles, UK) is a CE marked medical device for CTP image analysis of images obtained by cine imaging (in the head and body) after the intravenous (IV) injection of contrast. It produces image data and generates information regarding changes in image intensity over time and in calculation of the various perfusion-related parameters (including regional blood flow, regional blood volume, mean transit time and capillary permeability).

BrainScan

BrainScan® CT (BrainScan SA, Gdansk, Poland) is a CE marked AI-derived platform that enables automatic detection and classification of pathological changes occurring in CT examinations of the brain. Based on information from AI for Radiology (https://grand-challenge.org/aiforradiology), BrainScan CT currently has class 2a CE mark.

Cercare Stroke

Cercare® Stroke (Cercare Medical, Aarhus, Denmark) is a CE marked AI-enabled stroke CT and MRI software. The technology uses inputs from perfusion maps and additional maps of oxygen extraction and metabolism to provide an overview of brain tissues status in stroke. Based on information from AI for Radiology (https://grand-challenge.org/aiforradiology), Ceracare Stroke currently has class 2a CE mark.

CINA Head

CINA Head (Avicenna.ai, La Ciotat, France) uses CE marked class 1 medical device AI software for detecting abnormalities in brain CT scans. The algorithms in CINA head include:

• CINA ICH identifies suspected ICH on NCCT scans and prioritises them on the radiologist’s worklist.

• CINA LVO detects and prioritises the review of suspected LVOs on CTA.

• CINA ASPECTS analyses NCCT and creates heat maps that indicate signs of hypodensity which help characterise early ischaemic brain tissue injury.

Accipio

Accipio® (MaxQ AI, Tel Aviv, Israel) is a CE marked AI-derived software that analyses NCCT scan to identify and prioritise suspected ICH. Based on information from AI for Radiology (https://grand-challenge.org/aiforradiology), Accipio has class 2b CE mark. It was discontinued in 2022.

BioMind

BioMind® [Hanalytics (BioMind), Singapore, Singapore] is a CE marked (class not available publicly) AI-derived software used for detecting the location of intracerebral haemorrhage on CT scans and assessing its severity.

Stroke care service provision

The NHS Long Term Plan17 identifies stroke as a clinical priority and sets out in section 3.78 of the Plan the NHS’s ambition to support the national scaling of technology that will assist the expansion of life-changing treatments to more patients, which includes CTP scans to assess the reversibility of brain damage, improved access to MRI scanning and the potential use of AI in the interpretation of CT and MRI scans to support clinical decisions regarding suitability for thrombolysis and thrombectomy.

The National Stroke Service Model: Integrated Stroke Delivery Networks18 outlines best practices for stroke care, people with a suspected stroke should typically receive care within 4 hours in:

• a hospital with a comprehensive stroke centre that provides hyperacute, acute and inpatient rehabilitation including thrombectomy and neurosurgery services, or

• an acute stroke centre that provides hyperacute, acute and inpatient rehabilitation, but excluding thrombectomy and neurosurgery. All acute stroke centres are expected to have an intrahospital thrombectomy transfer pathway to transfer patients from acute stroke centres to CSCs.

Hyperacute stroke care usually covers the first 72 hours after a person is admitted. Services provided in the hyperacute phase include specialist clinical assessment, urgent imaging and skilled clinical interpretation of images, delivery of IV thrombolysis 24 hours a day, 7 days a week and transfer or treatment for thrombectomy. Imaging ensures that appropriate diagnosis is made, and time-dependent interventions are delivered. The guidance describes an optimal stroke imaging pathway.

Initial assessment

The diagnosis and initial management of suspected stroke are discussed in NICE guideline NG128. 7 For a diagnosis of stroke or TIA, patients with sudden onset of neurological symptoms outside hospital should be assessed using, for example, FAST and checked for a potential episode of hypoglycaemia. For patients admitted to the emergency department, the early diagnosis should be established using, for example, a ROSIER tool. 7

The guideline NG128 recommends: ‘Admit everyone with suspected stroke directly to a specialist acute stroke unit after initial assessment, from either the community, the emergency department, or outpatient clinics. (An acute stroke unit is a discrete area in the hospital that is staffed by a specialist stroke multidisciplinary team. It has access to equipment for monitoring and rehabilitating patients. Regular multidisciplinary team meetings occur for goal setting.). 7 Similarly, NICE Quality Standard QS21 states ‘Adults presenting at an accident and emergency (A&E) department with suspected stroke are admitted to a specialist acute stroke unit within 4 hours of arrival’.

For patients with an initial diagnosis of acute stroke and an indication of prompt brain imaging, NG1287 recommends immediate (i.e. ‘ideally the next slot and definitely within 1 hour, whichever is sooner’) brain imaging with a non-enhanced CT to rule out or confirm ICH, if any of the following apply:

• indications for thrombolysis or thrombectomy

• on anticoagulant treatment

• a known bleeding tendency

• a depressed level of consciousness [Glasgow Coma Scale (GCS) score below 13]

• unexplained progressive or fluctuating symptoms

• papilloedema, neck stiffness or fever

• severe headache at onset of stroke symptoms.

For patients with ischaemic stroke, CT with contrast angiography should be performed following an initial non-enhanced CT scan to confirm the presence of occlusion and/or clot. Addition of CTP imaging, or MR equivalent, is recommended if thrombectomy is indicated beyond 6 hours of symptom onset to assess potential salvage of brain tissue. 7

Patients with suspected acute stroke without indication for immediate brain imaging should be scanned as soon as possible and within 24 hours of symptom onset. 7

The National Stroke Service Model guidance18 describes an optimal stroke imaging pathway and recommends that stroke imaging, interpretation and transfer decisions are made within 20 minutes of patient’s arrival.

Treatment

Initially, patients with acute stroke must have their blood glucose concentration maintained and can be offered supplemental oxygen therapy if oxygen saturation drops below 95%. 7 The treatment options for patients with suspected or confirmed ischaemic or haemorrhagic stroke are summarised below.

Search strategy

Searches were undertaken to identify interventions using AI to diagnose acute stroke, as recommended in the CRD guidance for undertaking reviews in health care and the Cochrane Handbook for Diagnostic Test Accuracy Reviews. 22,24

Candidate search terms were identified from target references, browsing database thesauri (e.g. MEDLINE MeSH and Embase). Strategy development involved an iterative approach testing candidate text and indexing terms across a sample of bibliographic databases, so as to reach a satisfactory balance of sensitivity and specificity. Search strategies were developed specifically for each database and the keywords and thesaurus terms were adapted according to the configuration of each database. No restrictions on language, publication status or date were applied.

• MEDLINE (Ovid) 1946 to 7 July 2021

• MEDLINE In-Process Citations (Ovid) to 7 July 2021

• MEDLINE Daily Update (Ovid) to 7 July 2021

• MEDLINE Epub Ahead of Print (Ovid) to 7 July 2021

• Embase (Ovid) 1974 to 7 July 2021

• Cochrane Database of Systematic Reviews (CDSR; Wiley) to July 2021/Iss7

• Cochrane Central Register of Controlled Trials (CENTRAL; Wiley) to July 2021/Iss7

• Science Citation Index (SCI; Web of Science) 1988 to 6 July 2021

• Database of Abstracts of Reviews of Effects (DARE; www.crd.york.ac.uk/CRDWeb) to 31 March 2015

• Health Technology Assessment Database (HTA; www.crd.york.ac.uk/CRDWeb) 31 March 2018

• KSR Evidence (KSR Ltd) to 7 July 2021

• Epistemonikos (www.epistemonikos.org) to 7 July 2021

• International Network of Agencies for Health Technology Assessment (INAHTA) Publication (www.inahta.org) to 6 July 2021

• National Institute for Health and Care Research (NIHR) HTA programme (www.nihr.ac.uk) to 2 July 2021

• Aggressive Research Intelligence Facility (ARIF) database (www.birmingham.ac.uk/research/activity/mds/projects/HaPS/PHEB/ARIF/index.aspx) searched 2 July 2021

• PROSPERO (CRD; www.crd.york.ac.uk/prospero) to 7 July 2021

• International Platform of Registered Systematic Review and Meta-analysis Protocols (INPLASY, https://inplasy.com) to 2 July 2021

• Latin American and Caribbean Health Sciences Literature (LILACS; http://regional.bvsalud.org/php/index.php?lang=en) to 2 July 2021.

The main Embase search strategy was independently peer reviewed by a second information specialist, using the Peer Review of Electronic Search Strategies checklist. 25

Completed and continuing trials were identified by searches of the following resources:

• ClinicalTrials.gov (US National Institutes of Health; www.clinicaltrials.gov) to 2 July 2021

• European Union Clinical Trials Register (www.clinicaltrialsregister.eu/ctr-search/search) to 28 July 2021

• WHO International Clinical Trials Registry Platform (ICTRP; www.who.int/ictrp/en) to 2 July 2021

• ScanMedicine (https://scanmedicine.com) to 2 July 2021.

Inclusion and exclusion criteria

Separate inclusion criteria were developed for each of the three research questions, and these are summarised in Table 2.

Comparative studies, which reported secondary outcomes only (time to intervention and acceptability to clinicians), were included to maximise the available information for these outcomes. However, it should be noted that these outcomes alone are not sufficient to inform meaningful estimates of the clinical and cost-effectiveness of software using AI-derived algorithms for analysing CT brain scans in people with a suspected acute stroke. Because it is possible, for example, for the use of such software to reduce time to intervention while also being associated with poorer clinical outcomes, secondary outcome data are only useful for decision-making when combined with data on higher-level outcomes (clinical outcomes or measures of diagnostic performance).

Inclusion screening and data extraction

Two reviewers independently screened the titles and abstracts of all reports identified by searches and any discrepancies were discussed and resolved by consensus. Full copies of all studies deemed potentially relevant were obtained and the same two reviewers independently assessed these for inclusion; any disagreements were resolved by consensus. Details of studies excluded at the full paper screening stage are presented in Appendix 4, together with reasons for exclusion.

Studies cited in materials provided by the manufacturers of software with AI-derived algorithms for analysing CT brain scans in people with suspected stroke were first checked against the project reference database, in EndNote X20; any studies not already identified by our searches were screened for inclusion following the process described above.

Where available, data were extracted on the following: study design/details, participant characteristics, details of the AI-derived software (e.g. manufacturer, version used, mode of implementation), details of the CT scanner and imaging protocol(s), details of comparator (i.e. who reviewed the scans), clinical outcomes such as the Modified Rankin Scale (mRS), 2 × 2 data to calculate test performance outcome measures [sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV)], technical failure rates and time to intervention (time from imaging to IV thrombolysis or to groin puncture for mechanical thrombectomy). Data were extracted by one reviewer using standard data extraction forms. A second reviewer checked data extraction and any disagreements were resolved by consensus or discussion with a third reviewer.

Quality assessment

The methodological quality of studies reporting diagnostic accuracy data was assessed using QUADAS-2. 27 To provide optimal relevance to the current topic, the methodological quality of observational ‘before and after’ studies was assessed using a checklist devised by the authors for this review; this checklist included items relating to both risk of bias and reporting quality that were considered important for the interpretation of studies assessing the implementation of AI-derived software-assisted review of CT images in stroke patients. Quality assessment was undertaken by one reviewer and checked by a second reviewer, and any disagreements were resolved by consensus or discussion with a third reviewer.

The results of the quality assessments are summarised and presented in Tables 5 and 6 (see Study quality) and are provided in full, by study, in Appendix 3.

Methods of analysis/synthesis

Where multiple studies evaluated the accuracy of the same AI-derived software for the same target condition, the hierarchical summary receiver operating characteristic (HSROC) model was used to estimate summary sensitivity and specificity with 95% confidence intervals (CIs) and prediction regions around the summary points, and to plot HSROC curves. 28–30 This approach allows for between-study heterogeneity in sensitivity and specificity, and for the trade-off (negative correlation) between sensitivity and specificity commonly seen in diagnostic meta-analyses. Analyses were performed in Stata® 13 (StataCorp LP, College Station, TX, USA), mainly using the metandi command.

All other results, including those from ‘before and after’ studies of the implementation of AI-derived software technologies, were summarised in a narrative synthesis.

The results of included studies are grouped by research question addressed, AI-derived software evaluated and study type.

Overview of included studies

Based on the searches and inclusion criteria described above, a total of 30 publications33–61 relating to 22 studies33–36,39–41,43–46,48–52,55,56,59–62 were included in the review; the results section of this report cites studies using the primary publication and, where this is different, the publication in which the referenced data were reported.

The studies included in this review evaluated AI-derived software technologies produced by iSchemaView, Viz, Brainomix and Avicenna. For iSchemaView, three studies evaluated Rapid CTA,33,35,36 two studies evaluated Rapid LVO,41,55 one study evaluated Rapid CTP50 and two studies assessed the effects of implementing RapidAI (comprising Rapid CTA and Rapid CTP). 34,49 Eight studies evaluated Viz LVO40,43,45,46,52,59–61 and one study evaluated Viz ICH. 39 For Brainomix, one study evaluated e-CTA,56 one study evaluated e-ASPECTS,62 one study assessed the effects of implementing the e-ASPECTS and e-CTA components of the e-Stroke Suite44 and one study evaluated an un specified ‘AI-based algorithm developed by Brainomix’. 48 The remaining study evaluated CINA LVO, produced by Avicenna. 51 We did not identify any studies that evaluated the remaining AI-derived software technologies described in the Intervention technologies section of this report.

We did not identify any studies conducted in the UK that met the inclusion criteria for this assessment. However, one study reported that, to assess whether the sample was clinically representative of patients admitted to hospital with stroke, it was prespecified that age, sex, stroke severity, time since symptom onset and final diagnosis of included participants would be similar to data from the UK SSNAP (April 2018 to March 2019; www.strokeaudit.org), pooled randomised controlled trials (RCTs) and registries. 62 A total of 12 of the 22 included studies were conducted in USA,33,34,39,40,43,45,46,51,52,59–61 1 study each was conducted in Australia,36 Canada,55 Germany56 and Hungary,44 3 studies were multicentre studies conducted in USA, Brazil and Switzerland,41 in USA and the Netherlands,50 and in the UK and Germany (population validated for applicability to the UK setting using UK SSNAP data);62 the remaining three studies did not report information on geographical location. 35,48,49

Of the 22 included studies, 8 reported receiving some support from the manufacturers of AI-derived software technologies (including shareholdings, consulting fees and employment in relation to individual study authors),35,36,41,46,51,56,59,61 3 studies reported receiving no funding,33,34,52 2 studies were publicly funded50,62 and 9 studies reported no information about funding. 39,40,43–45,48,49,55,60

Full details of the characteristics of study participants, study inclusion and exclusion criteria, AI-derived software technologies evaluated and reference standard (for diagnostic test accuracy studies) or comparator (for before and after studies) are reported in the data extraction tables presented in Appendix 2, Tables 32 and 33.

Study quality

The methodological quality of the 15 studies36,39–41,43,48,50,51,55,56,59–62 that reported diagnostic test accuracy data was assessed using QUADAS-2. 27 No study reported accuracy data for more than one AI-derived software technology. Studies were generally poorly reported and information about how the AI-derived software technology (index text) was implemented; for example threshold or criteria used to determine the presence or absence of the target condition, was lacking. Five studies were published as conference abstracts only,39,40,43,48,60 and two studies were prepublication (not yet peer reviewed) texts. 55,62 All but one61 of the included studies were retrospective analyses and the remaining study61 did not report sufficient information to determine whether participants were recruited prospectively or retrospectively. The main potential sources of bias in the included diagnostic test accuracy studies relate to patient spectrum. There were also concerns regarding the applicability of the patient population and the index test to the research questions specified for this assessment (see Objective and Inclusion and exclusion criteria, Table 2). The results of QUADAS-2 assessments are summarised in Table 5; full QUADAS-2 assessments for each study are provided in Appendix 3. A summary of the risks of bias and applicability concerns within each QUADAS-2 domain is provided below.

Research question 1

1. Is the use of AI-derived software to assist review of non-enhanced CT brain scans to guide thrombolysis treatment decisions for people with suspected acute stroke a clinically effective intervention?

Four studies39,44,48,62 reported some limited information relevant to research question 1 and three of these four studies were reported as conference abstracts only. 39,44,48 The results of these studies are summarised below and detailed study characteristics are provided in Appendix 2. Three studies provided data on the diagnostic performance of AI-derived software technologies for the detection of ICH in patients with suspected AIS (Table 7); one study evaluated Viz ICH in a random sample taken from a cohort of stroke patients with and without ICH,39 the second study evaluated an un-specified Brainomix AI-derived software technology in patients with suspected AIS48 and the final study evaluated Brainomix e-ASPECTS in a clinically representative of patients admitted to hospital with stroke. 62 The sensitivity and specificity estimates were 90.2% (95% CI 83.9% to 94.2%) and 100% (95% CI 97.5% to 100%) for Viz ICH,39 91.1% (95% CI 82.8% to 95.6%) and 88.9 (95% CI 80.2% to 94.0%) for the unspecified Brainomix AI-derived software technology48 and 93.8% (95% CI 91.6% to 95.4%) and 82.8% (95% CI 81.4% to 84.1%) for Brainomix e-ASPECTS. 62 The study that evaluated Brainomix e-ASPECTS also provided sensitivity and specificity estimates for the detection of AIS; these were 68.5% (95% CI 66.4% to 70.5%) and 74.1% (95% CI 71.95% to 76.1%), respectively. 62 It should be noted that all these studies were retrospective analyses of previously acquired images that assessed the performance of the AI-derived software technology alone; no study provided information about the performance of an AI-derived software technology as an adjunct or aid to human interpretation (as it would be used in clinical practice, as its use is recommended by the manufacturer and as specified in the inclusion criteria for this assessment).

The remaining publication44 reported an observational ‘before and after’ study, evaluating the effects on time to treatment of implementing the e-ASPECTS and e-CTA modules of Brainomix e-Stroke in a centre which did not offer thrombectomy (patients requiring thrombectomy were transferred to another unit); the results of this study are summarised in Table 8. The publication stated that ‘delivery of stroke care was otherwise unchanged’. e-ASPECTS analyses NCCT scans for clot detection, signs of hypodensity and generates a heat map of regional ischaemic change, volume of the change and an automatic ASPECTS score, and e-CTA analyses CTP scans to generate perfusion summary maps, report parameters such as mismatch volume and ratio, hypoperfusion intensity ratio, and assesses eligibility for mechanical thrombectomy, hence, only the implementation of e-ASPECTS is relevant to research question 1. However, the effects of implementation were not reported separately for e-ASPECTS and e-CTA. 44 The proportion of patients receiving thrombolysis was 11.5% before implementation and 18.1% after implementation (absolute numbers not reported), and the proportion of patients transferred for thrombectomy was 2.8% before implementation and 4.8% after implementation (absolute numbers not reported). 44 For patients receiving thrombolysis, the mean time from door to treatment was 44 minutes before implementation and 41 minutes after implementation (no estimates of variance reported). 44 For patients transferred for thrombectomy, the mean time from first CT to groin puncture was 174 minutes before implementation and 145 minutes after implementation (no estimates of variance reported). 44 It should also be noted that this study did not report any information comparing clinical outcomes before and after implementation, such as would be needed to inform decision-making.

We did not identify any studies, conducted in patients with suspected AIS, evaluating Aidoc ICH, Rapid ICH, Rapid ASPECTS, qER, Zebra-Med, Brainscan, Avicenna CINA ICH, Avicenna CINA ASPECTS, MaxQ AI Accipio, or Biomind, the remaining AI-derived software technologies used in the analysis of NCCT images, as indicated in Table 1 and described in the Intervention technologies section of this report.

Research question 2a

1. (2a) Is the use of AI-derived software to assist review of CTA brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke a clinically effective intervention?

Eighteen studies reported information relevant to research question 2a. 33–36,40,41,43–46,49,51,52,55,56,59–61 Eleven studies reported sufficient information to allow calculation of measures of the diagnostic performance of AI-derived software technologies for the detection of LVO;35,36,40,41,43,51,55,56,59–61 2 studies evaluated Rapid CTA35,36 and 2 studies evaluated Rapid LVO,41,55 5 studies evaluated Viz LVO,40,43,59–61 1 study evaluated Brainomix e-CTA56 and 1 study evaluated Avicenna CINA LVO. 51 The remaining seven studies in this section were observational ‘before and after’ studies, which evaluated the effects of implementing AI-derived software technologies in clinical practice. 33,34,44–46,49,52 Four studies reported, specifically, on the implementation of AI-derived software technologies for the analysis of CTA images, one on the implementation of Rapid CTA33 and three on the implementation of Viz LVO. 45,46,52 The remaining three studies assessed the effects of implementation of AI-derived software technologies which were unclearly reported or included multiple components;34,44,49 one study44 reported on the implementation of e-ASPECTS and e-CTA and is described in Research question 1 and Table 8, and two studies34,49 reported on the implementation of Rapid technologies and are described in Research question 2b and Table 17. One study, which provided diagnostic performance data for Viz LVO, also reported the effect of implementing Viz LVO on time from door to groin puncture, in patients who were transferred for thrombectomy. 43

The results of studies in this section are grouped by AI-derived software technology. Detailed study characteristics are provided in Appendix 2.

We did not identify any studies conducted in patients with AIS that evaluated Aidoc LVO, the remaining AI-derived software technology used in the analysis of CTA images, as indicated in Table 1 and described in the Intervention technologies section of this report.

Viz LVO

Five studies reported sufficient information to calculate measures of the diagnostic performance of Viz LVO. 40,43,59–61 The target condition varied across studies, with respect to the anatomical location of occlusions,40,43,60 and two studies did not provide any definition of LVO. 59,61 The summary estimates of sensitivity and specificity, derived from all five studies, were 88.0% (95% CI 76.9% to 94.2%) and 89.9% (95% CI 85.5% to 93.0%), respectively (Figure 3). A sensitivity analysis, excluding one study where the reported target condition included posterior circulation occlusions,43 resulted in a higher summary estimate of sensitivity [91.3% (95% CI 84.9% to 95.1%)] and a similar summary estimate of specificity (89.3, 95% CI 83.5% to 93.2%; Figure 4). One study also reported that, for those patients who were transferred between centres for thrombectomy (number not reported), the median time from door to groin puncture was significantly shorter after implementation of Viz LVO, 141 (95% CI 128.5 to 168) minutes, compared with before implementation, 185 minutes (95% CI 151 to 241 minutes, p = 0.027). 43 This study did not report any comparison of clinical outcomes for the periods before and after implementation of Viz LVO. 43

FIGURE 3.

Summary receiver operating characteristic – all studies Viz LVO.The summary point, illustrated, should be interpreted with caution, given the absence of a clear definition of threshold in any of the included studies and the potential for variation in the anatomical area included in the target condition

FIGURE 4.

Summary receiver operating characteristic – sensitivity analysis Viz LVO.The summary point, illustrated, should be interpreted with caution, given the absence of a clear definition of threshold in any of the included studies and the potential for variation in the anatomical area included in the target condition

It should be noted that all five studies that provided data on the diagnostic performance of Viz LVO were retrospective analyses, of previously acquired images, which assessed the performance of the AI-derived software technology alone; no study provided information about the performance of Viz LVO as an adjunct or aid to human interpretation (as it would be used in clinical practice, as its use is recommended by the manufacturer and as specified in the inclusion criteria for this assessment). Full diagnostic performance data for Viz LVO are provided in Table 11.

TABLE 11 - Accuracy of Viz LVO for the identification of LVO

These summary estimates should be interpreted with caution, given the absence of a clear definition of threshold in any of the included studies and the potential for variation in the anatomical area included in the target condition.

Study details	Population	Target condition	Threshold	TP	FP	FN	TN	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Barreira 2018a60	All	Intracranial anterior circulation LVO (ICA, carotid terminus or M1 segment of the MCA)	NR	362	83	40	390	90.0 (86.7 to 92.6)	82.5 (78.8 to 85.6)	81.3 (77.5 to 84.7)	90.7 (87.6 to 93.1)
Chatterjee 201840		Intracranial anterior LVO (ICA, carotid terminus or M1 segment of the MCA) or M2 segment of the MCA occlusion		31	3	3	17	91.2 (77.0 to 97.0)	85.0 (64.0 to 94.8)	91.2 (77.0 to 97.0)	85.0 (64.0 to 94.8)
Dornbos 202043		Intracranial anterior LVO (ICA, carotid terminus or M1 segment of the MCA), distal M2 segment of the MCA or posterior circulation occlusion		45	55	23	557	66.2 (54.3 to 76.3)	91.0 (88.5 to 93.0)	45.0 (35.6 to 54.8)	96.0 (94.1 to 97.3)
Shalitin 202061		LVO (not defined)		157	147	6	2234	96.3 (92.2 to 98.3)	93.8 (92.8 to 94.7)	51.6 (46.0 to 57.2)	99.7 (99.4 to 99.9)
Yahav-Dovrat 202159	All ‘stroke protocol’	LVO (not defined)		59	33	13	299	81.9 (71.5 to 89.1)	90.1(86.4 to 92.8)	64.1 (53.9 to 73.2)	95.8 (93.0 to 97.5)
Summary estimatea (five studies)40,43,59–61								88.0 (76.9 to 94.2)	89.9 (85.5 to 93.0)
Sensitivity analysis,a excluding Dornbos 202043								91.3 (84.9 to 95.1)	89.3 (83.5 to 93.2)

Three further observational ‘before and after’ studies, reported information about the effects of implementing Viz LVO in clinical settings (Table 12). 45,46,52 One study reported that, for patients transferred between centres for thrombectomy, the median time from CTA to groin puncture was significantly shorter after implementation of Viz LVO, 127 minutes (range 39–622 minutes), compared with before implementation, 216 minutes (range 109–608 minutes, p = 0.026). 46 This study also reported a small reduction in the length of hospital stay after implementation of Viz LVO [mean difference (MD) −2.5 days, 95% CI −4.7 to −0.3 days)] and no significant change in the proportion of patients who were functionally independent at 90 days post-procedure (mRS ≤ 2, OR 1.67, 95% CI 0.45 to 6.23) or rates of in-hospital complications (OR 0.60, 95% CI 0.06 to 6.28) or in-hospital mortality (OR 1.33, 95% CI 0.31 to 5.73). 46 A second study from the same research group reported that the mean time from door to groin puncture was also reduced following implementation of Viz LVO for patients who were treated with thrombectomy within centre (MD −86.7 minutes, 95% CI −125.9 to −47.5 minutes). 45 Again, this study found no significant change in the proportion of patients who were functionally independent at 90 days post procedure (mRS ≤ 2, OR 0.88, 95% CI 0.46 to 1.69) or rates of in-hospital complications (OR 0.87, 95% CI 0.46 to 1.62) or in-hospital mortality (OR 1.10, 95% CI 0.55 to 2.21) and, additionally, reported no significant change (p = 0.103) in the median length of hospital stay. 45 The final study reported a significant reduction in the mean time from CTA to groin puncture (MD −44.6 minutes, 95% CI −68.6 to −20.6 minutes) after implementation of Viz LVO for patients transferred between centres for thrombectomy. 52 This study also reported no significant change in the mean 90-day mRS after implementation of Viz LVO (MD −1.0, 95% CI −2.1 to 0.1). 52

TABLE 12 - Effects of implementing Viz LVO for the analysis of CTA in patients with AIS, who are potential candidates for thrombectomy

IQR, interquartile range; NC, not calculable; NR, not reported; SD, standard deviation.

Calculated value.

Study details	Time to treatment outcome	Pre implementation	Post implementation	Mean difference (95% CI)	Clinical outcome	Pre implementation	Post implementation	Mean difference (95% CI) or OR (95% CI)
Dornbos 202043	Median (IQR) minutes from door to groin puncture, for transferred patients	185 (151–241), (n = NR)	141 (128.5–168), (n = NR)	NC	None reported	NA	NA	NA
Hassan 202046	Median (min, max) minutes from CTA to groin puncture, for transferred patients	216 (109–608), (n = 28)	127 (39–622), (n = 11)	NC	90-day mRS ≤ 2	8/28	6/15	1.67 (0.45 to 6.23)a
					In-hospital complications	3/28	1/15	0.60 (0.06 to 6.28)a
					In-hospital mortality	6/28	4/15	1.33 (0.31 to 5.73)a
					Mean (SD) hospital stay (days)	9.7 (4.9), (n = 28)	7.2 (2.5), (n = 15)	−2.5 (−4.7 to −0.3)a
Hassan 202145	Mean (SD) minutes from door to groin puncture, within centre	206.6 (169.1), (n = 86)	119.9 (83.0), (n = 102)	−86.7 (−125.9 to −47.5)a	90-day mRS ≤ 2	24/86	26/102	0.88 (0.46 to 1.69)a
					In-hospital complications	27/86	29/102	0.87 (0.46 to 1.62)a
					In-hospital mortality	18/86	23/102	1.10 (0.55 to 2.21)a
					Median (IQR) hospital stay (days)	7.0 (4.0–11.0)	7.5 (4.0–12.0)	NC
Morey 202052	Mean (SD) minutes from CTA to groin puncture, for transferred patients	161.3 (51.1), (n = 29)	146.7 (39.4), (n = 26)	−44.6 (−68.6 to −20.6)a	Mean (SD) 90-day mRS	4.3 (2.1), (n = 29)	3.3 (1.9), (n = 26)	−1.0 (−2.1 to 0.1)a

All four studies43,45,46,52 that provided information about the effects of implementing Viz LVO in clinical settings reported that implementation was associated with reductions in time to treatment for thrombectomy patients and, where reported, with no significant change in clinical outcomes. 45,46,52 However, it should be noted that two of these studies46,52 evaluated the implementation of Viz LVO in the context of providing an automated alert system (i.e. not as specified in the scope for this assessment) and the remaining two studies43,45 were reported as conference abstracts, which did not provide sufficient information to determine how Viz LVO had been implemented. It should also be noted that, although these studies provide some information about the effects of implementing Viz LVO in ‘real-world’ clinical settings, the information provided is limited to those patients who underwent thrombectomy; that is, there is no information about the performance of Viz LVO on the identification of patients who are candidates for thrombectomy.

Research question 2b

1. (2b) Is the use of AI-derived software-assisted review of CT perfusion brain scans to guide mechanical thrombectomy treatment decisions for people with an ischaemic stroke, after a CTA brain scan, a clinically effective intervention?

Three studies, two reported as journal articles34,50 and one as a conference abstract,49 provided some limited information relevant to research question 2b. All three studies evaluated iSchemaView Rapid products. The results of these studies are summarised below; detailed study characteristics are provided in Appendix 2, Tables 32 and 33.

One article reported sufficient information to allow the calculation of measures of the diagnostic performance of Rapid CTP for identifying patients who are suitable candidates for thrombectomy (Table 16). 50 The objectives of the study concerned the quantification and characterisation of failures occurring during the automated post-processing of imaging data with Rapid CTP. The study was a retrospective analysis of AIS patients, from a database, who had undergone CTP for thrombectomy; potential causes of Rapid CTP post-processing failures were evaluated by two clinicians (experience not specified) in consensus, who had access to all imaging data available at the time of patient evaluation and failures were reprocessed manually using IntelliSpace software (Philips, Best, Netherlands). A total of 176 AIS patients were included in the analysis and Rapid CTP post-processing failures accrued in 20 (11%) patients. Causes for failures were severe motion (n = 14, 70%), streak artefact (n = 3, 15%) and poor arrival of contrast (n = 3, 15%). Of the 176 patients, 126 (72%) received thrombectomy, based on clinical information and interpretation of CTP imaging, which included correction for failures. Based on information about the results of Rapid CTP image analysis provided in the paper and using treatment received as the reference standard, it was possible to calculate measures of the diagnostic performance of Rapid CTP alone (without correction) in identifying patients who are suitable candidates for thrombectomy; the estimated sensitivity was 95.2% (95% CI 90.0% to 97.8%) and the estimated specificity was 80.0% (95% CI 67.0% to 88.8%), and the estimates of PPV and NPV were 92.3% (95% CI 86.4% to 95.8%) and 87.0% (95% CI 74.3% to 93.9%), respectively.

The remaining two publications reported the results of observational ‘before and after’ studies, evaluating the effects on time to treatment and clinical outcome of implementing Rapid (details not specified) in the context of providing an automated e-mail alert system49 (i.e. not as specified in the scope for this assessment) and the RapidAI Mobile Application. 34 Neither study provided separate results for the effects of the CTA and CTP analysis algorithms in Rapid. The results of these studies are summarised in Table 17. One study reported no significant change in the mean time from door to groin puncture, in thrombectomy patients, following the implementation of RapidAI (MD 2.0 minutes, 95% CI −12.9 to 16.9 minutes). 49 Clinical outcome, as indicated by the proportion of patients (for whom data were available) with a mRS ≤ 3 (time point not specified), was also similar before (58/119, 48.7%) and after (23/41, 56.1%), implementation (calculated OR 1.34, 95% CI 0.66 to 2.74). 49 By contrast, the study that assessed the effects of implementing the RapidAI Mobile Application reported a reduction in the mean time from door to groin puncture after implementation (MD −33.2 minutes, 95% CI −60.2 to −6.2 minutes); this study also reported that implementation of the RapidAI Mobile Application had no effect on mean 90-day mRS (2.9, no estimate of variance reported) both before (n = 29) and after (n = 26) implementation. 34

We did not identify any studies conducted in patients with LVO that evaluated icobrain CT, Brainomix e-CTP, Viz CTP, CTP 4D (CEC Healthcare) or Ceracare Stroke, the remaining AI-derived software technologies used in the analysis of CTP images, as indicated in Table 1 and described in the Intervention technologies section of this report.

Selection of diagnostic accuracy estimates for inclusion in cost-effectiveness modelling

There is no evidence, in any population, about the accuracy of AI-derived software technologies in combination with clinicians. The available diagnostic accuracy studies were retrospective analyses of previously acquired images, which assessed the performance of the AI-derived software technology alone; no study provided information about the performance of an AI-derived software technology as an adjunct or aid to clinician interpretation (as it would be used in clinical practice and as specified in the decision problem for this assessment). This might imply that a cost-effectiveness analysis (CEA) is not feasible for any of the three research questions (1, 2a or 2b). However, we have chosen to conduct a CEA in relation to the research question (2a), where there is most evidence about the performance of AI-derived software technologies alone and one study comparing an AI-derived software technology alone with clinicians alone. 56 These studies were not considered appropriate to inform cost-effectiveness modelling, but formed the basis by which the accuracy of AI plus human reader could be elicited by expert opinion. The expert elicitation process, undertaken to inform cost-effectiveness modelling, is described in detail in the Model parameters section.

Diagnostic accuracy data sets were selected for use in the background information provided with the expert elicitation tool, based on comparability of the target condition across the different AI-derived software technologies assessed by included studies, comparability with the target condition in the study used to inform estimates of the effectiveness of thrombectomy in cost-effectiveness modelling,65 availability of comparator data56 and match to the target condition specified during the scoping phase of this assessment (see Table 2). The common target condition was intracranial anterior circulation LVO (ICA, carotid terminus or M1 segment of the MCA) or M2 segment of the MCA occlusion and the corresponding diagnostic performance estimates, for AI-derived software technologies and the comparator (human readers alone), provided with the expert elicitation tool are given in Table 18. These estimates were presented to the clinical experts for elicitation of sensitivity and specificity of the intervention (clinician plus AI) and the comparator (clinician only). It should be noted that the estimates for clinician alone could have been used directly in the model to inform the effectiveness of the comparator. However, given that so few data were available, from only one study,56 it was considered more appropriated to use these estimates to inform expert elicitation.

The decision to undertake an expert elicitation process was made, given the complete absence of applicable evidence in the literature, with a view to providing the diagnostic appraisal committee with a framework to consider the potential cost-effectiveness of AI as it would be used in practice and in order to facilitate the development of research recommendations. Nevertheless, no comparison of different AI-derived software technologies was feasible, and the results of this CEA (reported in the Assessment of cost-effectiveness section) need to be regarded with caution.

Search strategy

A series of literature searches were performed to identify published economic evaluations and cost-effectiveness data and utility studies for diagnostic techniques and procedures used in the investigation of patients with stroke that were not included within the scope of the clinical effectiveness searches. The searches aimed to identify studies that could be used to support the development of a health economic model, to estimate the model input parameters and to answer the research questions of the assessment but not to perform a systematic review. Searches were therefore pragmatic in design and date limits applied where appropriate.

Methodological study design filters were included in the search strategies where relevant. No restrictions on language or publication status were applied. Limits were applied to remove animal studies. The main Embase strategy for each search was independently peer reviewed by a second information specialist, using the Canadian Agency for Drugs and Technologies in Health peer review checklist. 25 Identified references were downloaded in EndNote software for further assessment and handling. References in retrieved articles were checked for additional studies. In addition, the EndNote library created for the clinical effectiveness section (Search strategy) was also screened to identify potentially relevant economic studies.

The following databases were searched for relevant studies with from 2005 to September 2021:

• NHS Economic Evaluation Database (NHS EED; www.crd.york.ac.uk/CRDWeb) to March 2015

• MEDLINE 1946 to 15 September 2021

• MEDLINE In-Process Citations to 15 September 2021

• MEDLINE Daily Update to 15 September 2021

• MEDLINE Epub Ahead of Print to 15 September 2021

• Embase 1974 to 15 September 2021

• EconLit (EBSCO) to 21 September 2021

• SCI 1988 to 21 September 2021

• Research Papers in Economics (http://repec.org) to 21 September 2021.

Inclusion criteria

Studies reporting outcomes of a full CEA, examining (quality-adjusted) life-years, with (at least) one AI-derived software-assisted review strategy, were eligible for inclusion.

Results

The literature search identified 2990 records from bibliographic database searches and supplementary searching (e.g. reference/citation checking, additional database searches including the database search for the assessment of clinical effectiveness). After title and abstract screening, 28 records are considered to be potentially relevant; after full-text screening one cost-effectiveness study (identified by hand-searching as it was published after conducting the literature search), was considered eligible for inclusion. This study is described in more detail below. Figure 5 shows the flow of studies through the review process.

FIGURE 5.

Flow of studies through the review process (review of economic analyses).

An additional economic model was submitted by Brainomix to NICE. This submission was not considered in this review as it was not specifically focused on one of the research questions nor does it adopt an approach (e.g. decision tree combined with a state transition model) typically adopted for diagnostic assessments.

Intervention and comparators

The health economic analysis focused on research question 2a:

1. (2a) Does AI-derived software-assisted review of CT angiography brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke represent a clinically and cost-effective use of NHS resources?

All diagnostic accuracy studies, identified by the systematic review conducted for this assessment (Assessment of clinical effectiveness section) assessed the accuracy of AI-derived software technologies as stand-alone interventions. As a result, information about how AI-derived software technologies would perform when used as an adjunct/aid to human readers (i.e. as recommended by the manufacturers, as specified for this assessment and as they would be used in clinical practice) is lacking. This is because the accuracy of the device by itself tells us nothing about how, or indeed if, it might improve the accuracy of a human reader. It would not make sense to infer that any of the variation in sensitivity observed between stand-alone AIs can tell us something about precisely the variation in a hypothetical, small improvement in sensitivity of the human reader. To still be able to perform a CEA, we elicited expert opinion to estimate the diagnostic accuracy of AI as adjunct to human reader. Experts were provided with the evidence on AI alone and human reader alone. Because it was considered too difficult for experts to differentiate between different AI-derived software-assisted review technologies, AI-derived software-assisted review in general (not specified by manufacturer or specific technology) is considered.

Model structure

This assessment uses the CEA by van Leeuwen et al. 67 (identified in the literature review as the only assessment focusing on a similar decision problem) as a starting point. In addition, recent cost-effectiveness assessments (mainly on the cost-effectiveness of thrombectomy) that have been identified informally (through the cost-effectiveness review, checking references) have also been used to support the development of the model. Consistent with the focus of AI-derived software-assisted review on triage and supporting the thrombectomy decision, the current assessment primarily considers the question of whether AI-derived software-assisted review could provide a more accurate diagnosis of LVO than usual care.

The de novo developed model consisted of a decision tree (short-term) and a state transition model (long-term) to calculate the mean expected costs and QALYs for people with ischaemic stroke and suspected LVO.

The decision tree was used to estimate short-term costs and consequences (first 90 days). For this purpose, a distinction is made between patients who have a LVO and those who do not. The definition of LVO was LVOs in the proximal anterior circulation (ICA, A1, M1, M2). This definition was chosen for two main reasons: consistency with the recommendations of NICE guidelines and with the meta-analysis by Román et al.,65 the source of the effectiveness of thrombectomy used in the model (Model parameters section). Subsequently, patients with LVO are classified as either eligible for thrombectomy or not eligible. Eligibility for thrombectomy is determined by a number of factors beyond the location of the occlusion, including timing and salvageability of brain tissue as determined by CTP scanning (Initial assessment section). Those with both LVO and eligibility for thrombectomy are further classified, based on the sensitivity of the diagnostic strategy, into whether a LVO was detected (and thus thrombectomy received) or not. Based on the classification in the decision tree, patients were subdivided into the health states according to the mRS. mRS is a commonly used scale for measuring the degree of disability or dependence in daily activities of people who have suffered a stroke and was the predominant outcome to define health states in published cost-effectiveness models in this disease area. Notably, patients without LVO were subdivided, based on the specificity of the diagnostic strategy, into whether a LVO was incorrectly detected or not. If a LVO was incorrectly detected (i.e. a false positive), this had cost consequences only (e.g. due to potential unnecessary transfer to experienced stroke centre qualified to perform thrombectomy) as, based on clinical opinion and consistent with the assessment by van Leeuwen et al. ,67 it was assumed that the LVO would be classified as a false positive (i.e. in fact no LVO) before proceeding to thrombectomy. The rationale for this was that specialists (e.g. neuroradiologists or neurointerventionalists) would review the imaging before agreeing to take patients for thrombectomy and then detect the false positive. The decision tree is shown in Figure 6.

FIGURE 6.

Decision tree structure (90 days).

The long-term consequences in terms of costs and QALYs were estimated using a state transition cohort model (Figure 7) with a lifetime horizon. The cycle time was 1 year. The following health states were included:

• 1) mRS 0

• 2) mRS 1

• 3) mRS 2

• 4) mRS 3

• 5) mRS 4

• 6) mRS 5

• 7) mRS 6 (death).

FIGURE 7.

State transition model structure.

The de novo model was developed in R Shiny (R Foundation for Statistical Computing, Vienna, Austria)69 to leverage the benefits of using modern programming languages such as R (R Foundation for Statistical Computing, Vienna, Austria)70 while providing an accessible interface through the Shiny package. To improve model transparency as well as model credibility and for consistency with suggested good practices and conventions, the technical implementation of the computational model was inspired by recent work of the Data Analytics Research and Technology in Healthcare Group71,72 and others. 73

Model parameters

Decision tree probabilities

Proportion of ischaemic strokes that are large-vessel occlusions

The proportion of ischaemic strokes correctly suspected to be caused by LVOs was estimated by pooling the prevalence of LVOs in the diagnostic accuracy studies35,40,51 (random-effects model using logit transformation), resulting in an estimated prevalence of 46.1% (95% CI 43.0% to 49.1%).

Eligibility for medical thrombectomy

Not all patients with LVO are eligible for thrombectomy. Based on the UK study by McMeekin et al. in 2017,74 including early presenters (within 4 hours of onset) as well as late presenters and those for which the timing was unknown, the proportion of patients with LVO eligible for thrombectomy was 41.2% (95% CI 40.6% to 41.8%).

Accuracy of clinician and artificial intelligence-derived software-assisted review of computed tomography angiography brain scans

Expert elicitation methods

As was outlined in Selection of diagnostic accuracy estimates for inclusion in cost-effectiveness modelling, the available accuracy estimates were not appropriate for the decision problem. We therefore elicited expert opinion to inform sensitivity and specificity of clinician review of CTA brain scans and of AI-assisted review of CTA brain scans. In addition, we also elicited the throughput of patients with ischaemic stroke and suspected LVO per an average centre (this was not used in the end). This translated into five elicitation questions. The sensitivity question was phrased in terms of proportion of LVOs missed (= 1 – sensitivity). The specificity question was phrased in terms of proportion of non-LVOs falsely classed as LVOs (= 1 – specificity; screenshots of the questions and background information provided to clinical experts are presented in Appendix 6, Figures 22, 23 and 24).

We used the EXPLICIT tool (EXPert eLICItation Tool) developed by Grigore et al. 75 to facilitate remote expert elicitation. This tool has been validated, follows established methodological guidance for expert elicitation,76–78 and has the advantage that it is relatively easy to use. The tool includes an informed consent form, training exercises and explanations of some important heuristics. We also included background information on the evidence on accuracy of AI standalone and human reader alone, identified in Selection of diagnostic accuracy estimates for inclusion in cost-effectiveness modelling. Experts were asked for the mode and the upper and lower bounds to each estimate. A beta-PERT (program evaluation and review technique) distribution was then fitted. Mathematical aggregation of elicited expert estimates was performed using linear pooling, that is by taking the arithmetic average over all experts for each elicited quantity.

Expert elicitation results

Five UK clinical experts sent complete responses (a consultant in emergency medicine, a clinical associate professor and honorary consultant stroke physician, a senior lecturer and honorary consultant neurosurgeon, a senior clinical lecturer and honorary consultant neuroradiologist and an honorary consultant neuroradiologist). The elicited mean sensitivity and specificity, as well as parameters for the beta-PERT distribution are presented in Table 19. Probability distributions are shown for the pooled experts’ estimates of sensitivity (Figure 8) and specificity (Figure 9) as well as for individual experts (Figure 10).

TABLE 19 - Results of expert elicitation

Assuming current care mix of expertise and circumstance.

	Mean	Lower bound	Mode	Upper bound
Cliniciana only sensitivity	93.00	83.60	94.20	97.60
Cliniciana only specificity	94.09	88.00	94.58	98.20
AI + cliniciana sensitivity	94.13	87.80	94.80	97.80
AI + cliniciana specificity	93.77	84.80	94.80	98.60

FIGURE 8.

Elicited sensitivity estimates (pooled).

FIGURE 9.

Elicited specificity estimates (pooled).

FIGURE 10.

Probability distributions of individual experts.

Initial distribution over modified Rankin Score states for patients with large-vessel occlusion

We performed a pragmatic review to inform the distribution over the disability post stroke health states at 90 days after thrombectomy or standard medical therapy (i.e. for those ineligible for thrombectomy) at the end of the decision tree. A study by Román et al. 65 of the effectiveness of thrombectomy was identified in which mRS outcomes at 90 days were estimated based on an individual patient-level data meta-analysis, combining from seven randomised trials: Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischaemic Stroke in the Netherlands (MR CLEAN),79 Endovascular Treatment for Small Core and Anterior Circulation Proximal Occlusion with Emphasis on Minimizing CT to Recanalization Times (ESCAPE),80 Extending the Time for Thrombolysis in Emergency Neurological Deficits – Intra-Arterial (EXTEND-IA),81 Solitaire with the Intention for Thrombectomy as Primary Endovascular Treatment (SWIFT PRIME),82 Randomized Trial of Revascularization with Solitaire FR Device versus Best Medical Therapy in the Treatment of Acute Stroke Due to Anterior Circulation Large-vessel Occlusion Presenting within Eight Hours of Symptom Onset (REVASCAT),83 Mechanical Thrombectomy After Intravenous Alteplase Versus Alteplase Alone After Stroke (THRACE)84 and Pragmatic Ischaemic Stroke Thrombectomy Evaluation (PISTE). 85 This study was deemed to be the most recent meta-analysis on this topic and included all relevant, high-quality, randomised trials. Eligibility for thrombectomy in those trials was also consistent with the vessel occlusions in the proximal anterior circulation (ICA, A1, M1, M2). 65 Given that Román et al. 65 presented only stratified estimates of the distribution of mRS outcomes (i.e. stratified for ASPECTS categories), results were pooled to obtain an estimate for the full population (Table 20).

TABLE 20 - Pooled estimates of mRS state distribution at day 90

IAT, intra-arterial thrombectomy.

Note: pooled estimates based on Román et al. 65

Treatment	mRS0	mRS1	mRS2	mRS3	mRS4	mRS5	mRS6
mRS after LVO treated with IAT (n = 856)	96 (11.1%)	154 (18.1%)	159 (18.6%)	137 (16.1%)	136 (15.9%)	47 (5.5%)	125 (14.7%)
mRS after LVO treated without IAT (n = 862)	54 (6.3%)	84 (9.8%)	122 (14.2%)	139 (16.2%)	216 (25.1%)	94 (10.9%)	150 (17.5%)

It is unclear to what extent these distributions are generalisable to the current UK NHS setting, given that there is no information on the proportion of early compared with late presenters or proportion of patients receiving alteplase. The impact of potential problems with generalisability here is considered to be small as all patients will receive standard medical therapy, regardless of their true-negative or false-positive status. Likewise, the proportion of patients who are early presenters is the same irrespective of test outcome, which would mean that the distribution over mRS states would also be the same irrespective of test outcome.

Initial distribution over modified Rankin Score states for patients without large-vessel occlusion

To inform the distribution of patients with small-vessel occlusion (i.e. the true negatives and false positives) over mRS states, we performed a pragmatic review of five systematic reviews and meta-analyses9,86–89 of studies assessing the effectiveness of thrombolysis. None of these studies reported the distribution over mRS states, but individual studies included did; hence, we reviewed all studies informing these meta-analyses and ruled out those that did not report the distribution over mRS states at least 3 months after stroke that did not focus on small-vessel occlusion, and were based on small sample sizes in the thrombolysis group (n < 150). Only two studies were included: Choi et al. 90 and Paek et al. 91 Both were based on South Korean registries and had similar sample sizes. Choi et al. 90 used a retrospective analysis of the Clinical Research Center for Stroke – Fifth Division Registry database with n = 194 in the unmatched sample; Paek et al. 91 used a prospective registry of 15 South Korean stroke centres with n = 193 in the thrombolysis group (Table 21). Owing to limitations to data availability with Choi et al. (see NA in Table 21),90 we used the distribution reported by Paek et al. 91 in the evidence review group base case. While it is unclear whether this is representative of UK patients, the proportion of small-vessel occlusion and the accompanying mRS distribution is the same regardless of test outcome and will therefore not be influential in terms of incremental results.

TABLE 21 - Modified Rankin Score state distribution for small-vessel occlusion at 90 days based on two studies (implemented in the model using a Dirichlet distribution)

Study	mRS0 (%)	mRS1 (%)	mRS2 (%)	mRS3 (%)	mRS4 (%)	mRS5 (%)	mRS6 (%)
Choi et al. 201590 (n = 194)	NA (39.2)	NA (31.4)	NA (11.3)	NA (10.3)	NA (NA)	NA (NA)	NA (NA)
Paek et al. 201991 (n = 192)	42 (21.8)	68 (35.2)	46 (23.8)	24 (12.4)	9 (4.7)	4 (2.1)	0 (0.0)

Transition probabilities for the state transition model

We performed a pragmatic review to inform the transition probabilities for the state transition model using the identified CEA for thrombectomy studies as a starting point. Consistent with most CEA studies, we assumed that no transitions were possible between mRS states unless a recurrent stroke occurred (only in Lobotesis et al. 201692 patients could improve or deteriorate by one mRS state at the end of year 1). Other relevant transition probabilities included the probability of having a recurrent stroke and the mRS distribution after a recurrent stroke. After a recurrent stroke, we assumed that patients could either stay in their mRS state or move to a more severe mRS state. The distribution over the mRS states, after recurrent stroke, was based on that for patient’s ineligible for mechanical thrombectomy (see Table 20) to reflect a worse outlook after recurrent stroke compared with first stroke.

Recurrent stroke with transitions to the same or worse modified Rankin Score states

In the CEA model by van Leeuwen et al. ,67 the annual rate for recurrent stroke was 2.8%, based on a study by Pennlert et al. 93 In this study sex- and age-adjusted annual risk of stroke recurrence was estimated for patients at 28 days after an ischaemic stroke in the Swedish population-based Monitoring Trends and Determinants of Cardiovascular Disease stroke incidence registry (n = 5885 with ischaemic stroke, mean age = 64.2 years, range 24–74 years and proportion male = 60.6%). The index stroke occurred between 1995 and 1998. The average rate for recurrent stroke over the use calculated based on data provided in online supplement table 1 of Pennlert et al. 93 was 2.8%. Pennlert et al. 93 also observed that there was a decline in recurrent stroke rates over time, but we did not include this in the model.

An alternative source is the study by Mohan et al. ,94 who performed a systematic review and meta-analysis of 13 included studies reporting cumulative risk of recurrence after first-ever stroke (both ischaemic and haemorrhagic stroke). Only 3 of the 13 studies were from the UK. Some of these studies dated back a long time; for example, the oldest started data collection in 1961, whereas the newest started in 2003. The pooled cumulative risk of stroke recurrence was 3.1% (95% CI 1.7% to 4.4%) at 30 days; 11.1% (95% CI 9.0% to 13.3%) at 1 year; 26.4% (95% CI 20.1% to 32.8%) at 5 years; and 39.2% (95% CI 27.2% to 51.2%) at 10 years after initial stroke. 94 A Weibull model was fitted but model parameters were not provided. Lobotesis et al. 92 used the cumulative risk at 5 years reported by Mohan et al. 94 and estimated the annual risk of recurrence for the rest of patients’ lives at 2.0%.

We considered that Pennlert et al. 93 provided estimates specifically for ischaemic stroke while these were not available from Mohan et al. 94 We therefore used estimates by Pennlert et al. 93 in the base case and explored the use of Mohan et al. 94 in a scenario.

In the model, risk of recurrent stroke was the same across all mRS health states (consistent with assumptions in other CEA studies). 67,92

Death conditional on functional status after stroke

The study by Slot et al. 95 estimated the risk of stroke-related mortality conditional on functional dependency using two cohorts: the Oxfordshire Community Stroke Project (n = 320), the Lothian Stroke Register (n = 448) and the First International Stroke Trial (n = 1563), all UK studies. The authors found a significant impact of functional status on the cause of death. In particular, functionally dependent patients (i.e. those with mRS scores of 3–5) were more likely to die of recurrent stroke [relative risk (RR) 1.68 (95% CI 1.49 to 1.91)] than functionally independent patients. Stroke-related causes of death were present in 794 (49%) of the functionally dependent patients compared with 207 (29%) of the independent patients in all three cohorts combined and included International Classification of Diseases (ICD) codes for cerebrovascular diseases (ICD-9 430–438; ICD-10 I60–I69), either mentioned in the death certificate as a primary cause of death or a contributing factor (i.e. secondary, tertiary or quaternary cause of death). The risk of stroke-related cause of death increased by mRS score.

We estimated (recurrent stroke-related) mortality by: (1) multiplying recurrent stroke probability with the mRS 6 probability (see Table 20), and (2) applying the RR of dying per mRS state reported by Slot et al. 95 (Table 22) to the general population mortality. 96 The maximum probability of these two approaches was used in the economic model. This prevented underestimating mortality in the more severe mRS health states; ensured that mortality was consistent with the age-adjusted general population mortality while including mRS health state dependent mortality; and prevented double-counting.

TABLE 22 - Risk of stroke-related death, by mRS, at 6 months post stroke

Note: Oxfordshire Community Stroke Project and Lothian Stroke Register cohorts combined.

Source: Slot et al. 2009,95 table 4.

mRS	RR (95% CI)
0–1	1.00 (baseline)
2	1.12 (0.82 to 1.56)
3	1.66 (1.24 to 2.23)
4	1.92 (1.41 to 2.61)
5	2.57 (1.92 to 3.43)

Health-related quality of life

To identify studies reporting utility values associated with the model health states (i.e. the mRS states), we performed a pragmatic search and review (see Search strategy). In addition, we also reviewed the identified CEA studies and searched their references. If necessary, references of articles identified in that way were also searched. This pragmatic review resulted in seven studies reporting utility values for the mRS states. 97–104 The studies reporting EuroQol Five Dimensions (EQ-5D) time-trade-off values using the UK value set were: Ali et al. ,101 Rivero-Arias et al. ,98 Rebchuk et al. 103 and Wang et al. 104 Ali et al. 101 was a multicountry study and the sample size of UK patients and utilities valued with the UK value set was small (n = 70). Rebchuk et al. 103 presented utility values averaged over nine studies that collected EQ-5D data, but most of these studies did not use the UK value set. Similarly, Wang et al. 104 presented utility values averaged over six studies, but not all of them using the UK value set. Therefore, we used total utility values from the study by Rivero-Arias et al. 98 in the base case (sample size of at least n = 365 at 1 month and more in subsequent months). Utilities by Rebchuk et al. 103 and Wang et al. 104 were applied in scenarios. Utility values are summarised in Table 23.

TABLE 23 - Utility values for mRS states

mRS	Rivero-Arias et al.98 utility values (SD)	Rebchuk et al.103 utility values (CI) (n)	Wang et al.104 utility values (SD) (n)
0	0.936 (0.127)	0.93 (0.96 to 0.9) (3624)	0.97 (0.1) (3148)
1	0.817 (0.183)	0.86 (0.89 to 0.83) (2376)	0.88 (0.16) (4968)
2	0.681 (0.211)	0.68 (0.72 to 0.64) (1149)	0.72 (0.21) (1950)
3	0.558 (0.284)	0.57 (0.61 to 0.53) (957)	0.54 (0.25) (2327)
4	0.265 (0.294)	0.31 (0.35 to 0.26) (1101)	0.23 (0.33) (1618)
5	−0.054 (0.264)	0.06 (0.12 to 0.00) (400)	−0.17 (0.21) (858)

Rivero-Arias et al. 98 derived mRS and EQ-5D three-level version (EQ-5D-3L) information from stroke or TIA patients identified as part of the Oxford Vascular (OXVASC) study. Ordinary least squares regression was used to predict UK EQ-5D-3L tariffs from mRS scores. Data were available at months 1, 6, 12 and 24 with sample sizes for the EQ-5D-3L varying by measurement point (n = 365, n = 478, n = 346 and n = 539, respectively).

Utility values used in the model were age-adjusted using the UK population norms for the EQ-5D-3L reported by Janssen et al. 105

Resource use and costs

Costs of artificial intelligence-derived software technologies

Based on information provided by each company, mean costs per patient were calculated for using AI-derived software technologies. In the base-case analysis, a mean estimate was used based on all four technologies.

To calculate an overall mean price of the AI-derived software technologies, annual license fees for each device were applied to the UK situation in terms of number of CSCs, primary stroke centres (PSCs) and total number of stroke patients in the UK based on the SSNAP. 106 The mean cost price of AI-derived software technologies in the base-case analysis was assumed to be £49.24 (£12.31).

Table 24 presents all relevant inputs as well as intervention-specific cost estimates.

TABLE 24 - Costs of AI-derived software technologies

Se, standard error.

Se was assumed to be 25% of the mean.

Fixed estimates for each AI technology			Source
Number of CSCs	25		Sentinel Stroke National Audit Programme
Number of PSCs	177		Sentinel Stroke National Audit Programme
Number of stroke patients in UK	87,635		Sentinel Stroke National Audit Programme
Intervention-specific inputs	Lowest price	Highest price	Source
Rapid CTA
AI licence annual fee for CSC:	£20,000	N/A	Provided by company
AI licence annual fee for PSC:	£20,000	N/A	Provided by company
Training costs:	£5000	N/A	Assumption
Total costs:	£5,050,000	N/A
Cost per patient (Se*)	£57.63 (£14.41)
Viz.ai
AI licence annual fee for CSC:	£40,000	£55,000	Provided by company
AI licence annual fee for PSC:	£20,000	£30,000	Provided by company
Training costs:	£7241	£7241	Provided by company
Total costs:	£6,002,682	£8,147,682
Cost per patient	£68.50	£92.97
Mean cost per patient (SE@)a	£80.73 (£20.18)
Avicenna
AI license Annual fee for CSC:	N/A	N/A	Avicenna only works with price per patient
AI license Annual fee for PSC:	N/A	N/A	Avicenna only works with price per patient
Training costs:	N/A	N/A	The company stated that no training was required to work with the software
Mean cost per patient (SE@a)	£7.08 (£1.77)		Avicenna only works with price per patient (this price is for centres up to 5000 scans per year)
Brainomix
AI license Annual fee for CSC:	£30,000	£30,000	Provided by company
AI license Annual fee for PSC:	£15,000	£15,000	Provided by company
Training costs:	£3000	£8000	Provided by company
Total costs:	£4,011,000	£5,021,000	Provided by company
Cost per patient	£45.77	£57.29
Mean cost per patient (Se*)	£51.53 (£12.88)
Mean cost price of AI-derived software technologies in base-case analysis
Mean cost per patient (Se*)	£49.24 (£12.31)

We performed a pragmatic review to inform resource use and costs parameters in the model. A study by Lobotesis et al. 92 was identified, which served as the main source of input parameters related to resource use and costs. In that study, a UK healthcare provider perspective was assumed, and all (treatment) cost estimates were broken down into units and unit prices, enabling us to calculate treatment costs using a bottom-up approach. In line with that study, short-term costs (< 90 days) consisted of costs for treatment, hospitalisation and management of adverse events. In Lobotesis et al. ,92 to estimate treatment costs, unit costs for each cost item were presented in combination with the corresponding number of units that each cost item was used in which unit costs were sourced from Personal Social Services Research Unit (PSSRU), Unit Costs of Health and Social Care,107 and treatment and device costs for the stent retriever were provided by Medtronic. Costs and resource use associated with IV tissue plasminogen activator were derived from the NICE Technology Appraisal for alteplase. 19 Using these numbers, treatment costs were calculated using a bottom-up approach (Table 25).

TABLE 25 - Short-term costs (< 90 days): costs for treatment, hospitalisation and management of adverse events

BNF, British National Formulary; t-PA, tissue plasminogen activator.

Note: Based on information from Lobotesis et al. 92

Cost items	Unit price (£)	Source	Units	Total price (indexed to 2020)
Mechanical thrombectomy
Stent retriever	3190	Covidien internal pricing	1.2	4161
Catheter/support kit	920	Covidien internal pricing	1	1.000
Procedure pack	35	Covidien internal pricing	1	38
Drapes/gowns/gloves	80	Covidien internal pricing	1	87
Sheath	15	Expert clinical opinion	1	16
Interventional suite	150	Expert clinical opinion	3	489
Anaesthetist	157	Expert clinical opinion (cost not available in PSSRU)	4	683
Anaesthetist assistant	58	PSSRU (nurse team manager)	4	252
Radiographer	58	PSSRU (nurse team manager)	3	189
Consultant interventional neuroradiologist	140	PSSRU (medical consultant)	3	457
Registrar	60	PSSRU (registrar)	3	196
Nurse (band 7)	58	PSSRU (nurse team manager)	3	189
Scrub nurse (band 5)	49	PSSRU (nurse team leader)	3	160
Subtotal				7916
IV thrombolysis
Nurse activates stroke team	49	PSSRU (nurse team leader)	0.08	4
Stroke team assessment (registrar grade)	60	PSSRU (registrar)	0.5	33
Blood test	5	ISD Scotland24	1	6
Registrar accompanies patient to CT scan	60	PSSRU (registrar)	1	65
Consultant reviews CT results and discusses with relatives	140	PSSRU (medical consultant)	0.5	76
Nurse assessment	58	PSSRU (nurse team manager)	0.08	5
IV t-PA infusion (registrar time)	60	PSSRU (registrar)	1.25	82
Additional 12 routine observations	49	PSSRU (nurse team leader)	1	53
1 : 1 care for 5 hours with senior nurse	58	PSSRU (nurse team manager)	5	315
Junior staff review	60	PSSRU (registrar)	0.42	27
Overnight junior staff review	60	PSSRU (registrar)	0.17	11
Consultant review after infusion	140	PSSRU (medical consultant)	0.33	50
Alteplase drug costs	576	BNF 25	1	626
Subtotal				1354
Non-thrombolytic treatment
Emergency department doctor assessment	140	PSSRU (medical consultant)	0.25	38
Blood test	5	ISD Scotland 24	1	6
CT scan (brain imaging)	91	NHS Reference Costs 26	1	98
Nurse to accompany to CT scan	49	PSSRU (nurse team leader)	1	53
Nurse assessment	49	PSSRU (nurse team leader)	0.08	4
Routine nurse observation (4 in 24 hours)	49	PSSRU (nurse team leader)	0.33	18
Junior staff review	60	PSSRU (registrar)	0.21	14
Consultant review at 24 hours	140	PSSRU (medical consultant)	0.25	38
Subtotal				269

In line with van Leeuwen et al. ,67 for patients with LVO receiving mechanical thrombectomy, it was assumed that 85% would receive both mechanical thrombectomy and IV thrombolysis, 10% to receive IV thrombolysis only and 5% to receive IV thrombolysis and going for mechanical thrombectomy but who appeared revascularised during angiography. Moreover, for patients with LVO not receiving mechanical thrombectomy, it was assumed that 40% would receive IV thrombolysis and 60% would receive non-thrombolytic treatment. 67 Treatment costs for non-LVO patients were assumed to be equal to the costs of one day in the acute stroke unit based on Patel et al. 108 Lastly, the additional costs of patients without LVO incorrectly classified as having LVO were assumed to be equal to the costs of an ambulance ride and a stroke unit day, using cost estimates from Patel et al. 108 An overview of the resulting short-term costs (< 90 days) for each branch of the decision tree is presented in Table 26.

TABLE 26 - Short-term costs (< 90 days) for each branch of the decision tree (2020 prices)

SE, standard error.

SE was assumed to be equal to 25% of the mean estimates.

Branch in decision tree	Costs (SE)a	Source
LVO receiving mechanical thrombectomy	8794 (2198)	Lobotesis et al.,92 van Leeuwen et al.67
LVO not receiving mechanical thrombectomy	702 (176)	Lobotesis et al.,92 van Leeuwen et al.67
LVO	745 (186)	Patel et al.108
Non-LVO incorrectly classified as LVO	559 (140)	Patel et al.108

Acute stroke costs (< 90 days) and long-term costs (annually) were attributed to the different mRS states and included costs of personal social services, such as nursing and residential care costs (i.e. for long-term costs). To this extent, Lobotesis et al. 92 used data from the OXVASC study. 109 As data were only available for three levels of post-stroke disability (i.e. mRS 0–2, mRS 3–4 and mRS 5), the authors employed a consensus-based approach by using three clinical experts from whom weights were elicited. By applying a weighting on the three levels of post-stroke disability, individual costs by mRS were calculated for mRS levels/states. Acute and long-term costs of acute ischaemic stroke by mRS are presented in Table 27.

TABLE 27 - Acute and long-term costs of acute ischaemic stroke by mRS

Source: Lobotesis et al. 92

mRS state	Mean acute costs (£) (SD)	Mean annual long-term costs (£) (SD)
0	3145 (8333)	2846 (3998)
1	3700 (8333)	3348 (3998)
2	4255 (8333)	3850 (3998)
3	16,409 (20,657)	13,697 (8343)
4	22,200 (20,657)	18,532 (8343)
5	26,367 (17,704)	30,093 (16,209)
6 (cost of death)	3328 (3055)	–

Overview of main model assumptions and input parameters

The main assumptions in the health economic analyses were:

1. Consistent with the focus of the AI-derived software-assisted review on triage and supporting the thrombectomy decision, the current assessment primarily considers the claim that AI-derived software-assisted review could provide a more accurate diagnosis of LVO.

2. Thrombectomy eligibility is independent of the diagnostic strategy.

3. For recurrent strokes, the mRS distribution of patients without thrombectomy is used.

4. Consistent with most cost-effectiveness studies in this disease area, it was assumed that transitions between health states mRS 0–5 were only possible in case a (recurrent) stroke occurred. After a recurrent stroke, patients could either stay in their mRS health state or move to a more severe mRS health state.

5. The risk of recurrent stroke was assumed the same across all mRS health states.

6. False positives have cost consequences only.

A summary of model input parameters is provided in Table 28.

Sensitivity analysis

Deterministic one-way sensitivity analyses were performed, using all stochastic input parameters, to assess the impact of input parameters on the estimated outcomes. The results of these analyses are presented using optimal strategy plots and plotting the input parameters versus outcomes. Info-rank plots, based on the probabilistic sensitivity analyses, are presented to explore the relative ‘importance’ of each parameter in terms of the expected value of information. Finally, two-way sensitivity analyses were performed, including the most influential AI-specific parameters.

Scenario analyses

Various deterministic scenario analyses were performed to assess the impact of assumptions on the estimated outcomes:

1. assuming that the AI technology costs are increased to £100 per patient

2. assuming that the proportion of LVO patients eligible for mechanical thrombectomy with AI is increased to 50%

3. assuming that the AI technology plus clinician sensitivity is increased to 96%

4. assuming the AI technology plus clinician sensitivity is decreased to 90%

5. assuming that the LVO prevalence is increased to 50%

6. assuming that the LVO prevalence is decreased to 40%

7. assuming that recurrent strokes are LVOs eligible for thrombectomy with appropriate mRS distribution

8. assuming that recurrent strokes are non-LVOs

9. assuming that additional false positive costs are increased to £2000

10. assuming that the annual recurrent stroke probability is decreased to 2%

11. assuming that the annual recurrent stroke probability is increased to 4%

12. assuming that the proportion of patients eligible for thrombectomy is increased to 50% (both strategies)

13. assuming that the proportion of patients eligible for thrombectomy is decreased to 35% (both strategies)

14. utility values based on Rebchuk et al. (0.93, 0.86, 0.68, 0.57, 0.31 and 0.06 for mRS 0–5)

15. utility values based on Wang et al. (0.97, 0.88, 0.72, 0.54, 0.23 and –0.17 for mRS 0–5)

16. assuming no mortality cap (allowing mortality to be potentially lower than general population mortality)

17. assuming no utility cap (allowing utility values to be potentially higher than general population utility values)

18. assuming neither mortality cap nor utility cap

19. assuming accuracy for current practice without AI is based on Seker et al. 56 (neuroradiologist grader)

20. assuming that accuracy for current practice without AI is based on Seker et al. 56 (resident graders).

Base-case analysis

The probabilistic results indicated that the addition of AI to detect LVO is potentially more effective, (QALY gain of 0.003), more costly (increased costs of £8.61) and cost-effective for willingness-to-pay thresholds of £3380 per QALY and higher (Table 29). The cost-effectiveness plane (see Figure 11) illustrates the negative correlation between incremental costs and incremental QALYs (i.e. if a technology is more effective it also tends to be less costly). The cost-effectiveness acceptability curves (Figure 12) indicates that at willingness-to-pay thresholds of £20,000 and £30,000 per QALY gained the probabilities of current practice with AI being cost-effective are 53.6% and 56.2%, respectively. The expected risks per patient associated with adding AI, at willingness-to-pay thresholds of £20,000 and £30,000 per QALY gained, were £80 and £95, respectively (these were £122 and £163, respectively, without adding AI; see expected loss curves; see Figure 12). On a population level (assuming 87,635 patients per annum in the UK) the estimated annual risks associated with adding AI were £7.0 million and £8.4 million, at willingness-to-pay thresholds of £20,000 and £30,000 per QALY gained, respectively. The deterministic results (Table 30) were similar to the probabilistic results.

FIGURE 11.

Convergence plot (incremental cost-effectiveness ratio), cost-effectiveness plane and expected incremental benefit.

FIGURE 12.

Cost-effectiveness acceptability and expected loss curves.

Intermediate results (probabilistic base case)

The diagnostic pathway (after the 90-day decision tree) results were similar for both strategies (Figure 13). The proportion of detected LVOs (and thus patients receiving thrombectomy) was increased from 17.6% to 17.9% when AI is added to current practice. As a result, the mRS 0–3 proportions were slightly higher while mRS 4–6 proportions were slightly lower when AI is added (differences < 0.1%). Moreover, the average traces (across all probabilistic sensitivity simulations) were very similar for both technologies (Figure 14). Similar to the 90-day decision tree results, the average trace differences per cycle were less than 0.1% with the addition of AI, resulting in slightly higher proportions in the lower mRS health states. When considering the cumulative costs and QALYs over time (Figure 15), the cost difference is largest in cycle 1 (the addition of AI resulting in a cost increase of £58) decreasing over time to £9 at the end of the time horizon. In contrast, the QALY difference (Figure 16) is smallest in cycle 1 (the addition of AI resulting in a QALY increase of 0.0002) increasing over time to 0.0025 QALY (at the end of the time horizon).

FIGURE 13.

Diagnostic pathway results for current practice with AI (t2) and without AI (t1).

FIGURE 14.

Average state transition trace for current practice with AI and without AI.

FIGURE 15.

Cumulative costs for current practice with AI and without AI.

FIGURE 16.

Cumulative QALYs for current practice with AI and without AI.

Considering the disaggregated costs, the cost increase for AI was mainly driven by the short-term costs (including the AI technology costs), while overall costs related to the mRS4 and mRS5 health states are lower (due to lower occupancy for these health states) when AI is added. Although incremental QALYs are very low and similar across health states, the increased QALYs for AI are driven by QALY differences in the mRS0 and mRS1 health states (due to higher occupancy for these health states). Finally, the estimated LYs were very similar for both strategies (10.847 vs. 10.848).

Sensitivity analyses

The info-rank plot indicated that the sensitivity of both technologies was the most important input parameter (Figure 17). In addition, the optimal strategy plots (Figure 18) indicated that the proportion of patients with LVO who are eligible for mechanical thrombectomy is important to determine the most optimal strategy in terms of costs and QALYs. For the estimated costs, specificity, the additional costs of the AI technology, costs related to mRS4 and mRS5 were input parameters (in addition to those mentioned above) that can change the strategy that is most optimal. Deterministic one-way sensitivity analyses for all stochastic parameters are presented in Figure 19 (costs) and Figure 20 (QALYs).

FIGURE 17.

Info-rank plots.

FIGURE 18.

Optimal strategy plots.

FIGURE 19.

One-way sensitivity analyses (costs).

FIGURE 20.

One-way sensitivity analyses (QALYs).

Two-way sensitivity analyses were performed (Figure 21) between: (1) AI technology sensitivity; (2) AI technology costs; and (3) the proportion of LVO patients eligible for mechanical thrombectomy with AI. These analyses indicated that (given the 95% CI of these inputs), although the AI technology sensitivity is a main driver of the results, the AI technology costs and the proportion of LVO patients eligible for mechanical thrombectomy with AI can have an impact on the minimal AI technology sensitivity required for the AI technology to be cost-effective.

FIGURE 21.

Two-way sensitivity analyses (net monetary benefit with willingness to pay of £30,000 per QALY).

Scenario analyses

The results of the deterministic scenario analyses are provided in Table 31. The most influential scenario analyses improving the cost-effectiveness of the addition of AI were increasing the AI technology sensitivity to 96%, increasing the proportion of LVO patients eligible for mechanical thrombectomy with AI to 50%, removing the mortality cap and using Seker et al. 56 to inform accuracy for current practice without AI (resident graders); in these scenarios, the addition of AI was dominant. Decreasing the AI technology sensitivity to 90% and using Seker et al. 56 to inform accuracy for current practice without AI (neuroradiologist grader) resulted in current practice without AI being dominant while increasing the AI technology costs to £100 per patient would increase the incremental cost-effectiveness ratio to £22,072 per QALY gained.

Clinical effectiveness

The evidence base, to inform assessment of the clinical effectiveness of AI-derived software technologies for analysing CT brain scans in people with suspected stroke, was limited. This assessment focused on evaluating the effectiveness of AI-derived software technologies as adjuncts or aid to human interpretation (i.e. as they would be used in clinical practice and as recommended by the manufacturers). Our assessment included a systematic review to identify evidence to address three specific research questions:

1. (1) Does AI-derived software-assisted review of non-enhanced CT brain scans for guiding thrombolysis treatment decisions for people with suspected acute stroke represent a clinically and cost-effective use of NHS resources?

2. (2a) Does AI-derived software-assisted review of CTA brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke represent a clinically and cost-effective use of NHS resources?

3. (2b) Does AI-derived software-assisted review of CTP brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke after a CTA brain scan represent a clinically and cost-effective use of NHS resources?

The scope included multiple software products, from 13 manufacturers (described in the Intervention technologies section). For 9 of the 13 manufacturers, no studies were identified that met the inclusion criteria (see Table 2) for our systematic review. All studies identified concerned AI-derived software technologies from four manufacturers, Avicenna, Brainomix, iSchemaView and Viz, and the majority 1833–36,40,41,43–46,49,51,52,55,56,59–61 of 22 studies33–36,39–41,43–46,48–52,55,56,59–62 reported data to inform research question 2a (i.e. evaluated AI-derived software for the interpretation of CTA). All the studies identified by our systematic review were either studies assessing the diagnostic accuracy of AI-derived software alone (i.e. not as it would be used in clinical practice, as recommended by the manufacturers and as specified in the inclusion criteria for this assessment),35,36,39–41,43,48,50,51,55,56,59–62 or ‘before and after’ observational studies reporting information about the effects of implementing AI-derived software technologies for treated patients only. 33,34,44–46,49,52

Cost-effectiveness

Clinical effectiveness

Extensive literature searches were conducted to maximise retrieval of relevant studies. These included electronic searches of a variety of bibliographic databases, as well as screening of clinical trials registers and conference abstracts to identify unpublished studies. Because of the known difficulties in identifying test accuracy studies using study design-related search terms,110 search strategies were developed to maximise sensitivity at the expense of reduced specificity. Thus, large numbers of citations were identified and screened, relatively few of which met the inclusion criteria of the review.

The possibility of publication bias remains a potential problem for all systematic reviews. Considerations may differ for systematic reviews of test accuracy studies. It is relatively simple to define a positive result for studies of treatment; for example, a significant difference between the treatment and control groups, which favours treatment. This is not the case for test accuracy studies, which measure agreement between index test and reference standard. It would seem likely that studies finding greater agreement (high estimates of sensitivity and specificity) will be published more often. In addition, test accuracy data are often collected as part of routine clinical practice or by retrospective review of records; test accuracy studies are not subject to the formal registration procedures applied to RCTs and are therefore more easily discarded when results appear unfavourable. The extent to which publication bias occurs in studies of test accuracy remains unclear; however, simulation studies have indicated that the effect of publication bias on meta-analytic estimates of test accuracy is minimal. 111 Formal assessment of publication bias in systematic reviews of test accuracy studies remains problematic and reliability is limited. 24 We did not undertake a statistical assessment of publication bias in this review. However, our search strategy included a variety of routes to identify unpublished studies and resulted in the inclusion of a number of conference abstracts.

The rapidly evolving nature of research in this topic area presented a particular challenge. To be as inclusive as possible, we conducted a search of the medRvix the preprint server and asked clinical experts (specialist committee members for this topic) to provide details of any potentially relevant ongoing or unpublished studies, of which they were aware. One included study was identified from the medRvix search55 and a further study was provided, pre-publication, by a specialist committee member. 62 Results from these studies should be treated with appropriate caution, as they have not yet undergone peer review. To minimise the chances of omitting relevant new articles published since the original core strategies were run in July 2021, the main Embase and MEDLINE searches and the medRvix search were rerun in their entirety in October 2021 before submission of our draft report.

Clear inclusion criteria were specified in the protocol for this review, the review has been registered on PROSPERO (CRD42021269609) and the protocol is available from www.nice.org.uk/guidance/gid-dg10044/documents/final-protocol. The eligibility of studies for inclusion is therefore transparent. In addition, we have provided specific reasons for exclusion for all of the studies that were considered potentially relevant at initial citation screening and were subsequently excluded on assessment of the full publication (see Appendix 4). The review process followed recommended methods to minimise the potential for error and/or bias;22 studies were independently screened for inclusion by two reviewers and data extraction and quality assessment were done by one reviewer and checked by a second. Any disagreements were resolved by consensus.

The main limitations for this assessment were the paucity of evidence, particularly in relation to research questions 1 and 2b and, where evidence was identified, the applicability of that evidence to the specified questions.

The concerns regarding the applicability of the included studies were common across all three research questions.

The primary applicability concern, for studies that provided test accuracy data, was in relation to the implementation of the index test (AI-derived software technology). In all of these studies,35,36,39–41,43,48,50,51,55,56,59–62 the AI-derived software technology was evaluated as a stand-alone intervention, rather than as an adjunct or aid to human interpretation [as it would be used in clinical practice, as recommended by the manufacturers and as specified in the inclusion criteria for this assessment (see Table 2)].

In addition to diagnostic test accuracy studies, this assessment included some observational ‘before and after’ studies33,34,44–46,49,52 that assessed the effects of implementing AI-derived software technologies in ‘real-world’ clinical settings on time to intervention and in some cases,33,45,46,52 on clinical outcome. The information provided by studies of this type is limited in that it concerns only treated (i.e. test positive) patients; no information is provided about test-negative patients, hence there is no information about the extent to which AI-derived software technologies, as implemented, may miss patients with the target condition(s). In addition, no ‘real-world’ implementation study, included in this assessment, compared clinical outcomes with time to intervention in populations that were comparable (with respect to key baseline characteristics) before and after the implementation of the AI-derived software technology and where the AI-derived software technology was the only change to the care pathway. Differences in the study population (before and after implementation) and/or additional changes in the care pathway (other than implementation of the AI-derived software technology) mean that the extent to which any observed changes in time to intervention or clinical outcome are attributable to the implementation of the AI-derived software technology is highly uncertain. Studies that report only the effects of implementation of AI-derived software technologies on time to intervention are deficient in that they do not provide the information about clinical outcomes needed to inform decision-making. A reduction in time to intervention may not always be advantageous; for example, if the time saving is associated with a detrimental effect on clinical outcomes.

With respect to research question 1, ‘Is the use of AI-derived software to assist review of non-enhanced CT brain scans to guide thrombolysis treatment decisions for people with suspected acute stroke a clinically effective intervention?’ we were only able to identify three studies that reported information about the accuracy of AI-derived software technologies for the interpretation of NCCT in people with suspected acute stroke39,48,62 and one further observational ‘before and after’ study44 that assessed the combined effects of implementing Brainomix e-ASPECTS and e-CTA. Studies that evaluated included AI-derived software technologies frequently did not meet the population inclusion criteria for this assessment; for example, studies that evaluated the accuracy of AI-derived software technologies for the detection of ICH in all head CTs (i.e. including trauma patients and other suspected pathologies), with no separate data for patients with suspected stroke.

During the inclusion screening phase of our systematic review, we noted a number of articles reporting multivariable regression analyses, where good clinical outcome/functional independence (90-day mRS 0–2) was the dependent variable and baseline Brainomix e-ASPECTS score or e-Stroke-derived ischaemic core volume, on NCCT, was evaluated as a potential predictor of clinical outcome following thrombectomy. 112–115 These studies do not meet the inclusion criteria for our assessment because they do not provide a comparison of image interpretation with versus without the assistance of AI-derived software technologies to guide treatment decisions (e.g. whether or not to perform mechanical thrombectomy). In the examples cited, all participants had anterior circulation LVO strokes and underwent mechanical thrombectomy. 112–115 One study reported that in a multivariable regression analysis, adjusting for potential confounders including age, sex, hypertension, diabetes mellitus, AF, smoking status, baseline blood glucose, baseline National Institute of Health Stroke Scale (NIHSS) score, receipt of IV thrombolysis (tissue-type plasminogen activator) and time from last-known-well to imaging low ischaemic core volume, based on Brainomix e-Stroke software interpretation of baseline NCCT, was independently predictive of good outcome (adjusted OR 0.98, 95% CI 0.97 to 0.99). 112 Two further studies113,115 reported that the results of multivariable regression analyses indicated that e-ASPECTS score, on baseline NCCT, was an independent predictive of good outcome; adjusted OR 1.30 (95% CI 1.06 to 1.60; adjusted for age, premorbid mRS, baseline NIHSS, hypertension, hypercholesterolaemia, diabetes mellitus and prior stroke)113 and OR 1.37 (95% CI 1.01 to 1.84), co-variables not reported. 115 The final study of this type included variables for age, premorbid mRS, AF, previous stroke, baseline blood glucose, and haemoglobin A1c, baseline NIHSS, hyperdense vessel sign, e-ASPECTS, general anaesthesia, recanalisation and secondary ICH following IV thrombolysis and reported that e-ASPECTS was not independently predictive of good outcome. 114 Although they do not directly inform the research questions specified for this assessment, studies of this type may be of clinical interest in that they describe the potential of AI-derived parameters, taken from initial NCCT imaging, to predict clinical outcome following thrombectomy.

With respect to research question 2b, ‘Is the use of AI-derived software-assisted review of CT perfusion brain scans to guide mechanical thrombectomy treatment decisions for people with an ischaemic stroke, after a CTA brain scan, a clinically effective intervention?’ we were only able to identify one study that reported sufficient information to allow the calculation of measures of the diagnostic accuracy of Rapid CTP for identifying patients who are suitable candidates for thrombectomy, using treatment received as the reference standard50 and two further observational ‘before and after’ studies34,49 that assessed the effects of implementing RapidAI.

Of further note, two of the multivariable regression analyses described above also reported that low ischaemic core volume, assessed using iSchemaview Rapid CTP, was independently predictive of good clinical outcome (90-day mRS 0–2), adjusted OR 0.98 (95% CI 0.96 to 1.00)113 and adjusted OR 0.98 (95% CI 0.97 to 0.99). 112

This assessment did not identify sufficient evidence to support modelling of the cost-effectiveness of AI-derived software-assisted review of CTP brain scans to guide mechanical thrombectomy treatment decisions for people with an ischaemic stroke, after a CTA brain scan (research question 2b). However, although a systematic review of the effectiveness of treatments outside the scope of this assessment, it is notable that a number of key RCTs conducted in USA82,116,117 and Australia,81 supporting the effectiveness of thrombectomy in addition to IV thrombolysis, for patients with anterior circulation intracranial LVOs, used ischaemic core volume as a component of the participant selection criteria; in all instances, ischaemic core volume on CT was assessed using iSchemaview Rapid CTP, which may perhaps indicate that iSchemaView Rapid software is already widely used for the interpretation of CTP images.

Cost-effectiveness

Our CEA is the most comprehensive analysis to date focusing on AI-derived software-assisted review of CTA brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke. The de novo probabilistic model was based on a previously developed CEA by van Leeuwen et al. 67 For the present analysis, a number of adjustments were made to the model, but most of the assumptions were maintained. The adjustments included adding probabilistic analyses, discount rates in line with NICE reference and the choice of alternative input parameters where this was considered appropriate (e.g. for implementing mortality and health state utility values).

Our initial intention was to inform accuracy estimates in the economic model through a comprehensive, high-quality systematic review of diagnostic accuracy studies. However, the available evidence was not appropriate to inform of CEA, as the accuracy estimates available were for AI-derived software technologies as stand-alone interventions rather than as an adjunct or aid to human interpretation (as defined in the scope for this assessment). To be able to perform a CEA, we obtained accuracy estimates by means of elicitation of expert beliefs. For this, we used an established tool75 that has been validated and that follows established methodological guidance for expert elicitation. 76–78 We obtained responses from five clinical experts representing a range of relevant specialties. Additional parameters were, where necessary, based on a pragmatic literature review. Such a review is standard practice in economic modelling given the large number of parameters required.

As in any economic model, a number of major and minor assumptions had to be made. It is important to understand the impact of these assumptions, to correctly interpret the results of the model. The impact of most assumptions has been explored in sensitivity and scenario analyses. One assumption that might be a matter for discussion is the focus of the AI-derived software-assisted review on triage and supporting the thrombectomy decision, considering the claim that AI-derived software-assisted review could provide a more accurate diagnosis of LVO. Other potential benefits, such as potential reduced time to treatment through the addition of AI-derived software-assisted review were not considered in the base-case analyses. However, in scenario and sensitivity analyses the impact of additional benefits of AI-derived software-assisted review were considered, indicating that this might potentially be an influential assumption warranting further studies (see also discussion of uncertainties in the cost-effectiveness; see Cost-effectiveness).

Finally, another strength of this assessment was the use of R (instead of the commonly used Microsoft Excel®, Microsoft Corporation, Redmond, WA, USA) to estimate the cost-effectiveness. The use of R allows leveraging the benefits of using modern programming languages,70 including improved transparency, reproducibility, modifiability and computational efficiency. The accessibility of R models might be perceived as a barrier for users unfamiliar with programming languages. Therefore, an accessible user interface was provided to the R model through the R Shiny package. Through the R Shiny user interface, users can specify different assumptions, change input parameters values, run underlying R code and visualise results. With the simple instructions provided in the readme file (included with the submission), this model is accessible to those with no programming knowledge allowing critical inquiry from decision-makers and other stakeholders. Moreover, to aid model transparency, as well as model credibility and for consistency with suggested good practices and conventions, the technical implementation of the computational model was inspired by recent work by the Data Analytics Research and Technology in Healthcare group71,72 and others. 73

Clinical effectiveness

The key question, that is whether or not the addition of AI-derived software technologies can improve the performance of human readers at the decision points specified in the three research questions, and hence improve clinical outcomes for stroke patients, is not adequately addressed by either the diagnostic test accuracy studies or the observational ‘before and after’ studies included in this assessment.

We did not identify any studies that evaluated the diagnostic accuracy of an AI-derived software technology when used as an adjunct to human readers. One study included in this assessment, provided a direct comparison of the accuracy of an AI-derived software technology (Brainomix e-CTA) alone compared with individual human readers with different training and experience, for the detection of LVO (see Table 14). 56 This study found that the sensitivity of e-CTA alone (84.3%, 95% CI 74.0% to 91.0%) was similar to neurology resident 1 (85.7%, 95% CI 75.7% to 92.1%) or lower than neurology resident 2 (91.4%, 95% CI 82.5% to 96.0%), radiology resident (95.7%, 95% CI 88.1% to 98.5%), neuroradiologist (97.1%, 95% CI 90.2% to 99.2%) and that of unassisted human readers. 56 Based on these results, for it to be possible for the AI-derived software technology to improve the performance of human readers there would need to be a systematic difference in the reasons for a false negative (missed LVO) between the AI-derived software technology and human readers such that some or all of the small proportion of LVOs missed by human readers would be detected by the AI-derived software technology. However, it should be noted that it is unclear whether these unfavourable comparative accuracy results are reproducible or generalisable across different AI-derived software technologies and human readers in UK clinical settings; higher sensitivity estimates have been reported, using other AI-derived software technologies alone (Rapid CTA, Viz LVO and Avicennna CINNA LVO), for the detection of LVO (see Research question 2a) and we did not identify any UK studies comparing the accuracy of AI-derived software technologies alone to that of human readers.

The 2018 position statement on AI from the Royal College of Radiologists includes the following text, under the heading of Regulation: ‘A robust regulatory framework for the integration of AI into medical practice needs to be drawn up. Many companies of varying sizes are developing AI tools for use in radiology and clinical oncology. These companies are making claims about the power of these tools - some of which are unsubstantiated’. 14 The position statement goes on to specify, under the heading of Quality Assurance/Governance/Veracity, that: ‘Published results for sensitivity and specificity of AI tools will be necessary prior to the introduction of any technology in the radiology/clinical workflow’. 14 None of the AI-derived software technologies included in this assessment meet this requirement, in that we have not identified any estimates of the sensitivity and specificity (published or unpublished) of these interventions as they would be used in clinical practice (as an adjunct/aid to human interpretation of CT images). Some sensitivity and specificity estimates have been reported for the following AI-derived software technologies, evaluated as stand-alone interventions: Viz ICH and Viz LVO; iSchemaview Rapid CTA, Rapid LVO and Rapid CTP; Brainomix e-ASPECTS and e-CTA; Avicenna CINA LVO. These data are provided in the following sections: Research question 1, Research question 2a and Research question 2b of this report. For the remaining AI-derived software technologies included in the scope for this assessment and described in the section on Intervention technologies in this report, we did not identify any studies that met the inclusion criteria for this assessment.

Seven33,34,43–46,52 of the eight33,34,43–46,49,52 observational ‘before and after’ studies that assessed the effects of implementing AI-derived software technologies, in patients undergoing thrombectomy, reported results indicating that implementation was associated with a reduction in time to intervention. However, no study reported information to suggest that these reductions in time to intervention were associated with improvements in clinical outcome; all six studies that assessed clinical outcome reported results suggesting that the implementation of an AI-derived software technology had no effect on functional outcome, as indicated by mRS. 33,34,45,46,49,52 There is evidence from a meta-analysis of individual patient data118 and a multicentre RCT (the MR CLEAN study)119 to indicate a negative correlation between time to intervention and functional outcome in patients with LVO who undergo thrombectomy. The results of the meta-analysis of individual patient data indicated that earlier treatment with thrombectomy in addition to pharmacological thrombolysis was associated with lower degrees of disability, as indicated by 90-day mRS, than pharmacological thrombolysis alone and that this benefit remained statistically significant up to 7 hours and 18 minutes from onset of symptoms to arterial puncture; each hour of reperfusion delay was associated with a reduction in the proportion of patients achieving function independence (mRS 0–2), absolute risk difference −5.2% (95% CI −8.3% to −2.1%). 118 Similarly, the MR CLEAN study reported that thrombectomy remained an effective intervention, with respect to the proportion of patients achieving functional independence, up to 6 hours and 18 minutes from onset of symptoms to arterial puncture and that the absolute risk difference for achieving a good functional outcome was reduced by 6% for every hour of delay to reperfusion. 119 However, it remains unclear whether the potential reductions in time to intervention that might be achieved as a result of implementing of AI-derived software technologies would translate into improved clinical outcomes in ‘real-world’ settings. In addition, it should be remembered that the implementation of an AI-derived software technology has the potential to change not only the outcomes of patients who undergo thrombectomy but also which patients are selected for thrombectomy. Hence, evidence of a beneficial effect of implementation for patients undergoing thrombectomy is insufficient to show clinical effectiveness. This is because it would remain possible for there to be no effect or a detrimental effect on overall clinical outcomes in the scenario where implementation resulted in more patients who were suitable candidates for thrombectomy being missed (e.g. where an AI-derived software technology misses LVO in the same types of patients as a less experienced human reader and hence provides false reassurance).

The scope for this assessment specified one clinically relevant subgroup: ‘People over the age of 80 years with small-vessel disease and calcification of the cerebrovasculature’. 120 We did not identify any evidence to inform an assessment of the clinical effectiveness of any of the specified AI-derived software technologies in this population.

The inclusion criteria for this assessment (see Table 2) specified an early (last known well within 6 hours) window for research question 2a, on the clinical and cost-effectiveness of AI-derived software technologies for the interpretation of CTA, and a later (last known well more than 6 hours previously, but within 24 hours) window for research question 2b, on the clinical and cost-effectiveness of AI-derived software technologies for the interpretation of CTP following CTA. However, it remains unclear to what extent patients in the early window may benefit from additional imaging (CTP). Randomised controlled trials conducted in the UK85 and in the Netherlands79 in patients with LVO (detected on CTA, MRA or DSA) who were treated within 6 hours of symptom onset (i.e. the population specified for research question 2a; see Table 2), reported absolute differences the proportion of patients who were functionally independent (mRS 0–2) at 90 days of 11%85 and 13.5%79 in favour of thrombectomy. Of note, trials that additionally used utilised ischaemic core volume, assessed using Rapid CTP, as an imaging criterion to select patients for inclusion, within the 6-hour time window specified for research question 2a reported larger absolute differences the proportion of patients who were functionally independent (mRS 0–2) at 90 days of 31%81 and 25%82 in favour of thrombectomy.

It is unclear to what extent the diagnostic accuracy of AI-derived software technologies may vary according to the precise way in which the target condition is defined (e.g. the extent of the arterial anatomy included in the definition of a LVO). In addition, what constitutes a clinically appropriate definition of the target condition LVO may change over time as thrombectomy techniques improve and the evidence base on the efficacy of thrombectomy evolves.

Cost-effectiveness

The cost-effectiveness acceptability curves indicated that, at willingness-to-pay thresholds of £20,000 and £30,000 per QALY gained, the probabilities of current practice with AI being cost-effective were 54% and 56%, respectively. Moreover, the estimated annual risks associated with the addition of AI were estimated to be £7.0 million and £8.4 million, at willingness-to-pay thresholds of £20,000 and £30,000 per QALY gained, respectively. To reduce these risks, further evidence on the sensitivity of both technologies was considered as most important. This is particularly relevant given that the current accuracy estimates were based on expert elicitation (since empirical evidence was lacking for AI-derived software technologies as an adjunct/aid to human readers) that would require confirmation. In addition, sensitivity analyses indicated that in case the addition of AI resulted in a reduced time to treatment thereby increasing the proportion of patients with LVO who are eligible for mechanical thrombectomy, this would be an important outcome to consider in future studies. In that case, the clinical consequences (e.g. in terms of distribution over mRS states) of the reduced time to treatment through the addition of AI are an important consideration. The current base-case assessment did not consider any consequences of potentially reduced time to treatment through AI as this claim was not supported by available evidence. First, it is unclear whether the addition of AI would indeed reduce time to treatment: in the only studies where a reduction in time to treatment was observed, it was unclear whether this was potentially caused by redesign/optimisation of the logistic process. Caution is needed in interpreting these studies, as such findings are likely heavily context dependent and rely on the exact implementation of the addition of AI (e.g. implementation with automated alert system). Hence, the optimal implementation and place of AI is a potentially relevant topic for research. Notably, scenario analyses using alternative accuracy estimates for care as usual without AI, indicated that AI might be especially useful for non-expert graders, but this requires confirmation in future studies. Second, it is unclear whether the addition of AI would reduce the time to treatment and what the consequences would be in terms of impact on clinical outcomes such as distribution between mRS states. Moreover, from a cost perspective more evidence regarding the additional costs of the AI technology, and costs related to mRS4 and mRS5 would be informative.