You are here

Article Text

Article menu
Original research
Real-world evaluation of Brainomix e-Stroke software
  1. Dermot Mallon1,2,
  2. Matthew Fallon1,
  3. Eirini Blana1,
  4. Cillian McNamara1,
  5. Arathi Menon1,
  6. Chak Lam Ip3,
  7. Jack Garnham1,
  8. Tarek Yousry1,2,
  9. Peter Cowley1,
  10. Robert Simister2,3,
  11. David Doig1,2
  1. 1Lysholm Department of Neuroradiology, National Hospital for Neurology and Neurosurgery, University College London Hospitals NHS Foundation Trust, London, UK
  2. 2UCL Queen Square Institute of Neurology, London, UK
  3. 3Comprehensive Stroke Centre, University College London Hospitals NHS Foundation Trust, London, UK
  1. Correspondence to Dr Dermot Mallon; dermot.mallon{at}nhs.net

Abstract

Background and purpose Brainomix e-Stroke is an artificial intelligence-based decision support tool that aids the interpretation of CT imaging in the context of acute stroke. While e-Stroke has the potential to improve the speed and accuracy of diagnosis, real-world validation is essential. The aim of this study was to prospectively evaluate the performance of Brainomix e-Stroke in an unselected cohort of patients with suspected acute ischaemic stroke.

Methods The study cohort included all patients admitted to the University College London Hospital Hyperacute Stroke Unit between October 2021 and April 2022. For e-ASPECTS and e-CTA, the ground truth was determined by a neuroradiologist with access to all clinical and imaging data. For e-CTP, the values of the core infarct and ischaemic penumbra were compared with those derived from syngo.via, an alternate software used at our institution.

Results 1163 studies were performed in 551 patients admitted during the study period. Of these, 1130 (97.2%) were successfully processed by e-Stroke in an average of 4 min. For identifying acute middle cerebral artery territory ischaemia, e-ASPECTS had an accuracy of 77.0% and was more specific (83.5%) than sensitive (58.6%). The accuracy for identifying hyperdense thrombus was lower (69.1%), which was mainly due to many false positives (positive predictive value of 22.9%). Identification of acute haemorrhage was highly accurate (97.8%) with a sensitivity of 100% and a specificity of 97.6%; false positives were typically caused by areas of calcification. The accuracy of e-CTA for large vessel occlusions was 91.5%. The core infarct and ischaemic penumbra volumes provided by e-CTP strongly correlated with those provided by syngo.via (ρ=0.804—0.979).

Conclusion Brainomix e-Stroke software provides rapid and reliable analysis of CT imaging in the acute stroke setting although, in line with the manufacturer’s guidance, it should be used as an adjunct to expert interpretation rather than a standalone decision-making tool.

  • Stroke
  • Cerebral Infarction
  • CT perfusion
  • Thrombectomy

Data availability statement

No data are available. Not applicable.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/svn-2023-002859

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    WHAT IS ALREADY KNOWN ON THIS TOPIC

    • Automated interpretation of CT imaging in patients with suspected acute ischaemic stroke using artificial intelligence-based decision support tools, such as Brainomix e-Stroke, is becoming more common. Prior studies reporting the performance of Brainomix have been confined to patients with confirmed acute ischaemic stroke.

    WHAT THIS STUDY ADDS

    • As the first prospective evaluation of Brainomix e-Stroke in an unselected patient cohort, this study provides an accurate ‘real-world’ representation of the performance that can be expected in clinical practice.

    HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

    • This study will assist other centres in their decision to implement Brainomix e-Stroke or similar software. With false negatives in identifying acute ischaemia and large vessel occlusions, this study underlines the on-going need for the review of CT imaging by a suitably trained clinician before treatment decisions are made.

    Introduction

    Mechanical thrombectomy has become the standard of care for patients with acute ischaemic stroke due to a proximal anterior circulation large vessel occlusion.1 2 A major determinant of the clinical outcome following mechanical thrombectomy is the time to recanalisation.3 4 Analysis of imaging data by artificial intelligence (AI)-based decision support tools has the potential to improve the speed and accuracy of the interpretation of CT imaging in the context of acute stroke.5 6

    Brainomix (Oxford, UK) is one of the several commercial enterprises providing a cloud-based software suite, e-Stroke, for automated interpretation of non-contrast CT (NCCT) of the head (e-ASPECTS), intracranial CT angiography (e-CTA) and CT perfusion imaging (e-CTP). e-ASPECTS identifies acute ischaemia in the Middle Cerebral Artery (MCA) territory, anterior circulation vascular hyperdensity caused by acute thrombus, and acute haemorrhage. e-CTA identifies proximal anterior circulation large vessel occlusions. e-CTP automatically post-processes CTP data to provide estimates of the volume of core ischaemia (irreversibly injured brain) and penumbra (hypoperfused tissue at risk that could be salvaged by recanalisation therapy).

    Rigorous independent external validation of decision support tools is crucial.7 8 Prior studies assessing the performance of Brainomix e-Stroke, as well as similar software from other vendors, have relied on highly selected cohorts. For example, cohorts often only include patients with confirmed acute ischaemic stroke6 9 and have excluded patients with prior stroke10 or imaging degraded by artefact.11 12 Such study designs increase the risk of bias and have the potential to skew diagnostic statistics.

    While Brainomix e-Stroke is designed for use in the setting of acute ischaemic stroke, at our centre and in others, the software automatically analyses imaging from a wider range of patients—patients who will be diagnosed with acute stroke and those who will be diagnosed with a stroke mimic. Therefore, validation in a ‘real-world’ consecutive and unselected cohort is important.

    The aim of this study was to provide a ‘real-world’ prospective evaluation of Brainomix e-Stroke software in consecutive, unselected, patients with suspected acute ischaemic stroke admitted to a tertiary centre acute stroke unit.

    Methods

    Cohort

    The prospective evaluation of Brainomix e-Stroke was based on a consecutive cohort of patients admitted to the University College London Hospital Hyperacute Stroke Unit (HASU) for suspected acute stroke between 1 October 2021 and 5 April 2022. No patients were excluded based on clinical factors. All patients admitted with suspected acute ischaemic stroke undergo an NCCT of the head and CT angiography (CTA) of the carotid and intracranial arteries on arrival unless contraindicated. CT perfusion (CTP) is used for patients who are eligible for extended time window stroke reversal treatments and with a delayed presentation (>6 hours and <24 hours) or an unknown time of symptom onset.

    Imaging data was collected from Carestream PACS (V.12, Philips Medical Systems, Netherlands). Brainomix e-Stroke results were collected from the DICOM-based output available on PACS. Clinical data was collected from a prospectively maintained database of patients admitted to the University College London Hospital HASU.

    Patients were excluded if an NCCT was not available or if there were mass lesions diagnosed on the initial CT (eg, metastasis). Patients with a mass lesion that was not identified on the admission CT (but instead was only identified on follow-up imaging) were not excluded. Patients diagnosed with other conditions that did not affect the brain parenchyma (eg, pachymeningeal disease) were not excluded.

    CT scanning parameters

    Imaging was performed on Siemens X.cite CT scanners. NCCT images were acquired from the skull base to the vertex with the following parameters: slice thickness of 1 mm, 250 mAs, 120 kV and the Hr40 reconstruction kernel. CTA images were acquired from the arch of the aorta to the vertex with following parameters: slice thickness of 0.8 mm, 105 mAs, 90 kVp and the Bv44 reconstruction kernel. CTP images were acquired for a volume extending 9.5 cm from the skull base with the following parameters: slice thickness 5 mm, 170 mAs, 70 kV and the Hr36 reconstruction kernel. 50 mL of Omnipaque 350 iodinated contrast agent was injected at 4 mL/s for CTA and 5 mL/s for CTP.

    Brainomix automated analysis

    All studies labelled as ‘CT Acute Stroke’ are automatically sent to the Brainomix cloud-based servers for analysis (V.11.1). The output of e-ASPECTS, e-CTA and e-CTP is automatically sent back to PACS in DICOM format. The software was not provided information on the laterality of the symptoms or any other clinical information.

    Imaging evaluation

    The goal of the study was to determine the true performance of Brainomix e-Stroke (rather than a comparison with a neuroradiologist’s assessment). Therefore, the ground truth for the location of MCA territory acute infarction, hyperdense vessels, acute haemorrhage and large (Internal Carotid Artery [ICA] and/or M1 MCA) and medium (M2 MCA) vessel occlusions was determined using all available imaging (both prior and subsequent) and clinical information by a neuroradiologist. Any uncertainty in the assessment of the images was resolved by consensus with at least two neuroradiologists.

    The Alberta Stroke Program Early CT Score (ASPECTS) is a tool for quantifying the size of an acute MCA territory infarct on CT. From a maximum of 10 (ie, normal), the score decreases by 1 point for each affected region where there is parenchymal hypoattenuation and/or swelling.13 For identification of an acute infarct, a true positive was defined as an e-ASPECTS <10 involving at least one correct region on the correct side. In cases where there was bilateral pathology, classification was performed as described previously.14 If e-Stroke identified an acute infarction, vessel occlusion or hyperdense vessel on both sides where there was only a unilateral lesion, this was recorded as a true positive (ignoring the contralateral false positive). Lesions identified by e-Stroke on the wrong side in cases of unilateral disease were recorded as a false negative (ignoring the contralateral false positive).

    Acute infarcts were not considered if there was an acute haemorrhage because, by design, information on possible ischaemia is suppressed by e-ASPECTS if a hyperdensity volume (ie, acute haemorrhage) of more than 4 mL is detected.

    If there was no follow-up imaging, an ASPECTS of 10 was recorded if an acute infarct was not identified on the initial CT study and if a clinical diagnosis of stroke was not made. Acute infarcts outside the MCA territory were recorded as an ASPECTS of 10. A hyperdense vessel identified by e-ASPECTS was considered a false positive if there was no acute occlusion on the contemporaneous CTA.

    Diagnostic statistics for e-ASPECTS were calculated for subgroups based on time (time from symptoms onset/last known well to time of scan of <4.5 hours or ≥4.5 hours) and stroke severity (National Institute of Health Stroke Score (NIHSS) <6 or ≥6).

    Analysis of CTP

    e-CTP was compared with the output of the perfusion module in syngo.via (V.8.6, Siemens Healthcare GmbH, Germany). The core infarct for e-CTP was defined as the volume of tissue with a Relative Cerebral Blood Volume (rCBV) <30% of the contralateral side. As described previously, a threshold of <20% was used to define core ischaemia in syngo.via.15 Ischaemic penumbra was defined as the volume of tissue with Tmax of greater than 6 s and not already included in the core infarct. The mismatch ratio was defined as the area with a Tmax of greater than 6 s divided by the area of the core infarct. CTP metrics defining eligibility for mechanical thrombectomy were a core infarct volume of less than 70 mL, a mismatch volume of greater than 15 mL and a mismatch ratio of greater than 1.8.16

    Statistical analysis

    Data is presented as median (IQR) unless otherwise stated. Linear correlation between variables was assessed using Pearson’s correlation coefficient. The Dixon Q-test was used to test for outliers in the accuracy of individual regions for e-ASPECTS. Statistics were performed using statsmodel library as implemented in Python V.3.11.

    Brainomix e-Stroke software was made available to our institution through an AI in Health and Care Award, organised by the Accelerate Access Collaborative (AAC) in partnership with NHSx and the National Institute for Health and Care Research (NIHR) in the UK. This service evaluation of the performance of the deployed software was approved as part of the hospital’s clinical governance programme (approval no. 052 02 223-SE).

    Results

    A flow chart for the study cohort is shown in figure 1. Of the 582 patients who underwent imaging for suspected acute stroke, 11 were excluded because an NCCT was not available (n=6) or because of a diagnosis other than stroke (metastasis in four patients and high-grade glioma in one case) was diagnosed on the original CT. Demographics and clinical information of the cohort are shown in table 1.

    Figure 1
    Figure 1

    Flow chart of the study cohort. †No e-CTA due to no CTA in 13 cases. CTA, CT angiography; CTP, CT perfusion; NCCT, non-contrast CT.

    View this table:
    Table 1

    Demographics and clinical data of patient cohort

    Post-processing success rate and time

    In 551 patients admitted during the study period, 1163 studies were performed. Of these, 1130 (97.2%) were successfully processed by e-Stroke. e-ASPECTS produced an output for 551 out of 571 patients (96.5%). While exact reasons for post-processing failure were not provided, there was significant movement artefact in 14 patients. e-CTA successfully provided an output for 545 out of 558 patients (97.7%); in 7 patients there was significant movement artefact and in 3 patients there was poor arterial contrast opacification. e-CTP was successfully post-processed in all patients.

    The time from image acquisition to output appearing on PACS was 3 (2–3) min for e-ASPECTS, 7 (5–8) min for e-CTA and 8 (7–10) min for e-CTP. For all studies, the post-processing time was 4 (3–6) min.

    e-ASPECTS

    Of the 492 patients who did not have a haemorrhage, 128 (26.0%) patients had an acute infarct in the MCA territory.

    Figure 2A shows a heatmap of e-ASPECTS and ASPECTS. The median ASPECTS and e-ASPECTS for patients with an acute infarct were 8 (6–9) and 9 (7–10), respectively. e-ASPECTS and ASPECTS were strongly positively correlated within one another (ρ=0.802, p<0.001).

    Figure 2
    Figure 2

    Performance of e-ASPECTS. A heatmap shows strong positive correlation between the ASPECTS and e-ASPECTS (A). Diagnostic statistics are shown for the detection of acute infarct (ie, ASPECTS<10) (B), hyperdense vessels (HD) (C) and acute haemorrhage (D). NPV, negative predictive value; PPV, positive predictive value.

    The accuracy for e-ASPECTS in detecting acute ischaemia was 77.0% with a higher specificity (83.5%) than sensitivity (58.6%) and a higher negative predictive value (NPV, 85.2%) than positive predictive value (PPV, 55.6%) (figure 2B). e-ASPECTS and ASPECTS matched in 332 (67.5%) patients and were within 1 point of each other in 429 (87.2%) patients. In patients with an acute infarct, ASPECTS and e-ASPECTS were identical in 28 (21.9%).

    Taking an ASPECTS of <6 as a cut-off for mechanical thrombectomy eligibility, there were three patients (0.6% of all patients with ASPECTS ≥6) where a reliance on e-ASPECTS alone would have resulted in an inappropriate exclusion for mechanical thrombectomy.

    Figure 3 shows the accuracy by ASPECTS region. Accuracy was lowest in the lentiform nucleus and highest in the M4 region. There were no statistically significant outliers.

    Figure 3
    Figure 3

    e-ASPECTS accuracy by region at the level of the basal ganglia (A) and the supraganglionic level (B).

    Of 67 hyperdense vessels in the ICA or M1 MCA, 36 were correctly identified by e-ASPECTS (figure 2C). An accuracy of 69.1% was mainly a consequence of many false positives (n=121).

    In some instances, e-ASPECTS (and e-CTA) identified true areas of acute ischaemia (or vessel occlusion) but with insufficient confidence to register as an ASPECTS <10 or a vessel occlusion (online supplemental figure 1).

    Supplemental material

    The diagnostic performance was not significantly different for patients who were scanned before or after 4.5 hours from symptom onset (accuracy of 78% and 76%, respectively, online supplemental figure 3A–D). While there was no significant difference for patients with an NIHSS ≥6, patients with an NIHSS <6 had a significantly lower sensitivity and PPV (21% and 23%, respectively). All 59 patients with acute haemorrhage were identified by e-ASPECTS (figure 2D). Twelve false positives were caused by areas of calcification, typically within the choroid plexus (online supplemental figure 2A).

    e-CTA

    Of 545 patients with a CTA, there were 80 (14%) vessel occlusions, 62 of which (11% of whole cohort, 77.5% of all vessel occlusions) involved the ICA or M1 MCA. The diagnostic statistics for e-CTA are shown in figure 4. When considering M1 MCA and ICA occlusions, false positives were more common (PPV of 63.4%) than false negatives (NPV of 96.7%). False positives were caused by arterial stenoses and areas of old infarction (online supplemental figure 2B–D).

    Figure 4
    Figure 4

    Performance of e-CTA. Diagnostic statistics for all vessel occlusions (Internal Carotid Artery [ICA], M1 and M2 Middle Cerebral Artery [MCA]) (A) and for large vessel occlusions (LVOs) only (ICA and M1 MCA) (B). CTA, CT angiography; NPV, negative predictive value; PPV, positive predictive value.

    e-CT perfusion

    Core ischaemia volumes, penumbral tissue-at-risk volumes and the mismatch ratios derived from e-CTP and syngo.via were strongly positively correlated with one another (ρ=0.979, 0.775 and 0.811, respectively) (figure 5).

    Figure 5
    Figure 5

    Scatter plots of the core infarct volume (A), penumbra volume (B) and mismatch ratio (C) volumes derived from Brainomix e-CTP and Siemens syngo.via. Orange data points refer to values greater than 10. CTP, CT perfusion.

    In patients with a large vessel occlusion who underwent CTP, eligibility for thrombectomy based on perfusion metrics derived from each software package was concordant in 14 (87.5%) out of 16 patients. In the two discordant cases, the values straddled the eligibility cut-offs (mismatch volume of 14 mL vs 18 mL and core ischaemia 69 mL vs 71 mL).

    Discussion

    This is the first prospective evaluation of the full suite of Brainomix e-Stroke software in an unselected cohort of patients with suspected acute ischaemic stroke. This study design has the advantage of providing a reliable estimate of both the real-world diagnostic performance (including diagnostic statistics that are influenced by disease incidence) and the post-processing success rates (as all imaging was acquired in line with Brainomix recommendations).

    The performance of e-ASPECTS in this study (accuracy, sensitivity and specificity of 77%, 57% and 84%, respectively) falls within the range reported in the literature (67–87%, 14–83% and 57–99%, respectively).11 17 18 Direct comparison with these statistics is difficult because of the significant differences in the study cohorts, which exclusively included patients with a confirmed anterior circulation acute ischaemic stroke. A more direct comparison can be made with a study of e-ASPECTS by Mair et al that included an analysis of a representative cohort that was simulated by enriching CT scans from stroke trial data with scans that were normal or with different diseases. Further, as in this study, the ground truth was based on expert review with all imaging and clinical data. In that analysis, the accuracy, sensitivity and specificity were 71%, 68% and 74%, respectively. Differences in the diagnostic performance may be related to differences in the incidence of stroke in the cohorts (54.4% vs 26.0% in this study) and evaluation of different e-Stroke versions (9 or 10 vs 11 in this study).

    There was no statistically significant difference in the accuracy of e-ASPECTS by individual ASPECTS region although the lower accuracy in the lentiform and insular cortex has been reported previously.12

    We found e-ASPECTS to be a specific and reliable method of identifying patients for mechanical thrombectomy; only 3 (0.6%) patients were misclassified as ineligible for mechanical thrombectomy based on an ASPECTS of <6. This rate of misclassification is lower than what has been reported previously (3.4%19 and 4.4%14). This misclassification must also be considered in the context of the expanding eligibility criteria for mechanical thrombectomy;20 using an ASPECTS of 3 as a threshold for eligibility, a misclassification by e-ASPECTS would become even less likely. In contrast, algorithm performance was less reliable for patients with an NIHSS of <6, where the sensitivity and PPV were significantly lower than in the whole cohort. This is likely to partly reflect the smaller infarct volume expected in patients with a lower NIHSS.

    Identifying hyperdense vessels due to acute thrombus was the least accurate (69.1%) component of e-ASPECTS, which was hampered by a large number of false positives (121, 22.2%). Perhaps higher sensitivity (at the expense of lower sensitivity) is preferable in cases where a patient has presented early and parenchymal ischaemic change has not yet developed; after expert review, a CT angiogram could be performed to exclude acute thrombus. Nevertheless, one must consider the risk of a large number of false-positive hyperdense vessels prompting unnecessary CT angiograms.

    e-ASPECTS was highly accurate in detecting acute haemorrhage, which reflects prior studies.21 Accuracy was limited only by false positives mainly caused by choroid plexus calcification. While high sensitivity to acute haemorrhage is important prior to initiating time-sensitive thrombolysis, false positives carry the risk of causing an unnecessary delay while a specialist review is sought.

    e-CTA was more specific (93.6%) than it was sensitive (77.6%). Both of these values are higher than a prior study based on the retrospective analysis of 545 CTAs from trial data (sensitivity and specificity of 72%).21 Again, some of this difference may be related to the incidence of large vessel occlusions (53.5% vs 12.5% in this study) and different software versions. Medium vessel occlusions are outside the specifications of the software and therefore, as would be expected, performance for medium vessel occlusions is significantly lower than for large vessel occlusions. Including medium vessel occlusions in the analysis nearly doubled the number of false negatives (15 vs 28). The sensitivity of 65% for large and medium vessel occlusions, which are increasingly considered a target for mechanical thrombectomy, highlights the on-going requirement for specialist review of the CT angiogram (and the requirement for on-going software development).

    The values derived from Brainomix e-CTP and syngo.via were positively correlated. The positive correlation was similar to that reported in a study comparing outputs from RAPID AI and Brainomix.15 In contrast to the other components of e-Stroke, the validation of e-CTP is limited by the lack of a readily available ground truth. While comparison of the core ischaemia with contemporaneous diffusion-weighted imaging is possible, this is rarely employed at our centre. Alternatively, the core infarct could be compared with the final infarct volume in patients where complete recanalisation was achieved soon after the CTP. There were too few patients who met these criteria in this study cohort for this to be an option.

    A balance must be met between suppressing an output from e-Stroke due to artefact and providing a dependable output. The processing success rate for e-ASPECTS (96.5%) and e-CTA (97.7%) is higher than reported previously (61—89.5%).14 21 22 This likely reflects the homogeneous imaging acquisition at our centre, which has been optimised for use with Brainomix e-Stroke. In contrast, higher failure rates may have been anticipated in prior studies where imaging data was acquired from historical multi-centre trials, often with suboptimal image quality and slice thickness.

    The findings from this study are relevant to clinical practice in the UK where investment in AI-based imaging analysis systems is being encouraged at a national level.23 Particularly in centres without immediate 24/7 neuroradiology support, the short processing time and automated analysis of e-Stroke can aid in the early identification of patients who may be candidates for mechanical thrombectomy. Theoretically, this would reduce the time to referral and transfer (ie, reduced door-in-door-out time) to a comprehensive stroke centre for a mechanical thrombectomy, which in turn would improve clinical outcomes. However, our data show that e-Stroke—at the current level of performance—cannot be used in isolation to select cases for active management on a thrombectomy pathway. Clinicians should ensure that all imaging for patients who are eligible for time-critical treatment is still formally reviewed by a radiologist and that cases are not ‘stood down’ for thrombectomy treatment consideration based purely on the results of an automated imaging analysis.

    This study has limitations that should be considered. First, we did not compare the performance of Brainomix with a blinded neuroradiologist, which has been reported previously. Rather, the goal of this study was to assess as accurately as possible the true performance of Brainomix in a real-world setting, which requires a firmer assessment of the ground truth. As discussed above, the evaluation of CTP is limited by the lack of a clear ground truth; future studies would ideally compare CTP values with those derived from ‘hyperacute’ MRI or the final infarct volume in those where complete recanalisation was achieved. Our results are based on a tertiary hyperacute stroke centre, where patients routinely undergo video triage before transfer; performance of an algorithm may vary in other centres where the incidence of stroke is lower. A major component of Brainomix e-Stroke is the ability to communicate and transfer images between centres; the value of this functionality was not assessed in this study. Lastly, while the performance of Brainomix software is similar to that reported for other software products,24 direct comparison of Brainomix e-Stroke with other competing products would ideally be performed. Such a comparison, using identical cohorts, would offer valueable information to centres deciding which clinical decision support tool to implement.

    Conclusion

    e-Stroke is a fast and reliable software package that can analyse CT imaging in the acute stroke setting. However, with wide variation in the accuracy of the individual components of e-Stroke, it remains an adjunct in the interpretation of acute stroke CT imaging and cannot be solely relied on for clinical decision making.

    Data availability statement

    No data are available. Not applicable.

    Ethics statements

    Patient consent for publication

    Acknowledgments

    RJS is part funded by the UCLH/UCL Biomedical Research Centre.

    References

    1. ↵
      1. Badhiwala JH,
      2. Nassiri F,
      3. Alhazzani W, et al
      . Endovascular Thrombectomy for acute ischemic stroke: A meta-analysis. JAMA 2015;314:1832–43. doi:10.1001/jama.2015.13767
      OpenUrlCrossRefPubMed
    2. ↵
      1. Goyal M,
      2. Menon BK,
      3. van Zwam WH, et al
      . Endovascular Thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. The Lancet 2016;387:1723–31. doi:10.1016/S0140-6736(16)00163-X
      OpenUrlCrossRef
    3. ↵
      1. Alawieh A,
      2. Vargas J,
      3. Fargen KM, et al
      . Impact of procedure time on outcomes of Thrombectomy for stroke. J Am Coll Cardiol 2019;73:879–90. doi:10.1016/j.jacc.2018.11.052
      OpenUrlFREE Full Text
    4. ↵
      1. Heit JJ,
      2. Mlynash M,
      3. Christensen S, et al
      . What predicts poor outcome after successful Thrombectomy in late time windows J NeuroIntervent Surg 2021;13:421–5. doi:10.1136/neurintsurg-2020-016125
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Herweh C,
      2. Ringleb PA,
      3. Rauch G, et al
      . Performance of E-ASPECTS software in comparison to that of stroke physicians on assessing CT scans of acute ischemic stroke patients. Int J Stroke 2016;11:438–45. doi:10.1177/1747493016632244
      OpenUrlCrossRefPubMed
    6. ↵
      1. Brinjikji W,
      2. Abbasi M,
      3. Arnold C, et al
      . E-ASPECTS software improves Interobserver agreement and accuracy of interpretation of aspects score. Interv Neuroradiol 2021;27:781–7. doi:10.1177/15910199211011861
      OpenUrlPubMed
    7. ↵
      1. Mikhail P,
      2. Le MGD,
      3. Mair G
      . Computational image analysis of Nonenhanced computed tomography for acute ischaemic stroke: A systematic review. J Stroke Cerebrovasc Dis 2020;29:104715.:S1052-3057(20)30091-4. doi:10.1016/j.jstrokecerebrovasdis.2020.104715
      OpenUrl
    8. ↵
      1. Yu AC,
      2. Mohajer B,
      3. Eng J
      . External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review. Radiol Artif Intell 2022;4:e210064. doi:10.1148/ryai.210064
    9. ↵
      1. Hoelter P,
      2. Muehlen I,
      3. Goelitz P, et al
      . Automated ASPECT scoring in acute ischemic stroke: comparison of three software tools. Neuroradiology 2020;62:1231–8. doi:10.1007/s00234-020-02439-3
      OpenUrlCrossRefPubMed
    10. ↵
      1. Goebel J,
      2. Stenzel E,
      3. Guberina N, et al
      . Automated ASPECT rating: comparison between the frontier ASPECT score software and the Brainomix software. Neuroradiology 2018;60:1267–72. doi:10.1007/s00234-018-2098-x
      OpenUrlCrossRefPubMed
    11. ↵
      1. Austein F,
      2. Wodarg F,
      3. Jürgensen N, et al
      . Automated versus manual imaging assessment of early ischemic changes in acute stroke: comparison of two software packages and expert consensus. Eur Radiol 2019;29:6285–92. doi:10.1007/s00330-019-06252-2
      OpenUrlCrossRefPubMed
    12. ↵
      1. Neuhaus A,
      2. Seyedsaadat SM,
      3. Mihal D, et al
      . Region-specific agreement in ASPECTS estimation between Neuroradiologists and E-ASPECTS software. J NeuroIntervent Surg 2020;12:720–4. doi:10.1136/neurintsurg-2019-015442
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. Barber PA,
      2. Demchuk AM,
      3. Zhang J, et al
      . Validity and reliability of a quantitative computed tomography score in predicting outcome of Hyperacute stroke before thrombolytic therapy. The Lancet 2000;355:1670–4. doi:10.1016/S0140-6736(00)02237-6
      OpenUrl
    14. ↵
      1. Mair G,
      2. White P,
      3. Bath PM, et al
      . External validation of E-ASPECTS software for interpreting brain CT in stroke. Ann Neurol 2022;92:943–57. doi:10.1002/ana.26495
      OpenUrlCrossRefPubMed
    15. ↵
      1. Mallon DH,
      2. Taylor EJR,
      3. Vittay OI, et al
      . Comparison of automated ASPECTS, large vessel occlusion detection and CTP analysis provided by Brainomix and Rapidai in patients with suspected ischaemic stroke. J Stroke Cerebrovasc Dis 2022;31:106702.:S1052-3057(22)00396-2. doi:10.1016/j.jstrokecerebrovasdis.2022.106702
      OpenUrl
    16. ↵
      1. Albers GW,
      2. Marks MP,
      3. Kemp S, et al
      . Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. N Engl J Med 2018;378:708–18. doi:10.1056/NEJMoa1713973
      OpenUrlCrossRefPubMed
    17. ↵
      1. Guberina N,
      2. Dietrich U,
      3. Radbruch A, et al
      . Detection of early infarction signs with machine learning-based diagnosis by means of the Alberta stroke program early CT score (ASPECTS) in the clinical routine. Neuroradiology 2018;60:889–901. doi:10.1007/s00234-018-2066-5
      OpenUrlCrossRefPubMed
    18. ↵
      1. Nagel S,
      2. Sinha D,
      3. Day D, et al
      . E-ASPECTS software is non-inferior to Neuroradiologists in applying the ASPECT score to computed tomography scans of acute ischemic stroke patients. Int J Stroke 2017;12:615–22. doi:10.1177/1747493016681020
      OpenUrlCrossRefPubMed
    19. ↵
      1. Sundaram VK,
      2. Goldstein J,
      3. Wheelwright D, et al
      . Automated ASPECTS in acute ischemic stroke: A comparative analysis with CT perfusion. AJNR Am J Neuroradiol 2019;40:2033–8. doi:10.3174/ajnr.A6303
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Sarraj A,
      2. Pujara DK,
      3. Campbell BC
      . Trial of Endovascular Thrombectomy for large ischemic strokes. N Engl J Med 2023;389:89–90.:10.1056/NEJMc2305915#sa3. doi:10.1056/NEJMc2305915
      OpenUrl
    21. ↵
      1. Mair G,
      2. White P,
      3. Bath PM, et al
      . Accuracy of artificial intelligence software for CT angiography in stroke. Ann Clin Transl Neurol 2023;10:1072–82. doi:10.1002/acn3.51790
      OpenUrl
    22. ↵
      1. Nagel S,
      2. Wang X,
      3. Carcel C, et al
      . Clinical utility of electronic Alberta stroke program early computed tomography score software in the ENCHANTED trial database. Stroke 2018;49:1407–11. doi:10.1161/STROKEAHA.117.019863
      OpenUrlAbstract/FREE Full Text
    23. ↵
      National Clinical Guideline for Stroke for the UK and Ireland 2023, Available: www.strokeguideline.org
    24. ↵
      1. Wardlaw JM,
      2. Mair G,
      3. von Kummer R, et al
      . Accuracy of automated computer-aided diagnosis for stroke imaging: A critical evaluation of current evidence. Stroke 2022;53:2393–403. doi:10.1161/STROKEAHA.121.036204
      OpenUrlCrossRef

    Supplementary material

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors DHM: conceptualisation, methodology, data curation, formal analysis, visualisation, writing—original draft, writing—review and editing. MF, EB, CM, AM, CLI: data curation, writing—review and editing. TY, PC, RS, DD: conceptualisation, methodology, writing—review and editing. DHM is the study guarantor.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests DHM has received consultancy fees from Brainomix for work related to algorithm development.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.