Skip to main content
SpringerLink
Log in

MMA: metadata supported multi-variate attention for onset detection and prediction

  • Published:
Data Mining and Knowledge Discovery Aims and scope Submit manuscript

Abstract

Deep learning has been applied successfully in sequence understanding and translation problems, especially in univariate, unimodal contexts, where large number of supervision data are available. The effectiveness of deep learning in more complex (multi-modal, multi-variate) contexts, where supervision data is rare, however, is generally not satisfactory. In this paper, we focus on improving detection and prediction accuracy in precisely such contexts – in particular, we focus on the problem of predicting seizure onsets relying on multi-modal (EEG, ICP, ECG, and ABP) sensory data streams, some of which (such as EEG) are inherently multi-variate due to the placement of multiple sensors to capture spatial distribution of the relevant signals. In particular, we note that multi-variate time series often carry robust, spatio-temporally localized features that could help predict onset events. We further argue that such features can be used to support implementation of metadata supported multivariate attention (or MMA) mechanisms that help significantly improve the effectiveness of neural networks architectures. In this paper, we use the proposed MMA approach to develop a multi-modal LSTM-based neural network architecture to tackle seizure onset detection and prediction tasks relying on EEG, ICP, ECG, and ABP data streams. We experimentally evaluate the proposed architecture under different scenarios – the results illustrate the effectiveness of the proposed attention mechanism, especially compared against other metadata driven competitors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. The EEG real world dataset is imbalanced, time steps having seizure-positive labels are \(7 \%\) of the total at best and \(<2\%\) at the worst (the labels are provided by expert physicians; see Sect.  3.1.1 for dataset details).

  2. In experiments reported in Sect. 3, the descriptor vector length is 128.

  3. Since the healthcare data is HIPAA protected, we make the code available. Also we have publicly available multi-modal COVID, traffic, Bitcoin and S &P index datasets and code for reproduction of results at https://rb.gy/umbzt8.

  4. Provided by NSF testbed “Chameleon: A Large-Scale Re-configurable Experimental Environment for Cloud Research”.

  5. https://finance.yahoo.com/quote/BTC-USD?p=BTC-USD

References

  • Ali ÖG, Arıtürk U (2014) Dynamic churn prediction framework with more effective use of rare event data: The case of private banking. Expert Syst Appl 41(17):7889–7903

    Article  Google Scholar 

  • Alotaiby TN, Alshebeili SA, Alotaibi FM, Alrshoud SR (2017) Epileptic seizure prediction using CSP and LDA for scalp EEG signals. Computat Intell Neurosci. https://doi.org/10.1155/2017/1240323

    Article  Google Scholar 

  • Ando S, Huang CY (2017) Deep over-sampling framework for classifying imbalanced data. In: Joint european conference on machine learning and knowledge discovery in databases. Springer, pp. 770–785

  • Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  • Bao F, Deng Y, Kong Y, Ren Z, Suo J, Dai Q (2019) Learning deep landmarks for imbalanced classification. IEEE Trans Neural Netw Learn Syst 31(8):2691–2704

    Article  PubMed  Google Scholar 

  • Bogaerts T, Masegosa AD, Angarita-Zapata JS, Onieva E, Hellinckx P (2020) A graph CNN-LSTM neural network for short and long-term traffic forecasting based on trajectory data. Transport Res Part C: Emerg Technol 112:62–77

    Article  Google Scholar 

  • Cheng J, Dong L, Lapata M (2016) Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733

  • Cheon S-P, Kim S, Lee S-Y, Lee C-B (2009) Bayesian networks based rare event prediction with sensor data. Knowl-Based Syst 22(5):336–343

    Article  Google Scholar 

  • Ding K, Gupta PK, Diaz-Arrastia R (2016) Epilepsy after traumatic brain injury. Translational research in traumatic brain injury

  • Dong Y, Jiang X, Zhou H, Lin Y, Shi Q (2021) Sr2cnn: zero-shot learning for signal recognition. IEEE Trans Signal Process 69:2316–2329

    Article  ADS  MathSciNet  Google Scholar 

  • Dumanis SB, French JA, Bernard C, Worrell GA, Fureman BE (2017) Seizure forecasting from idea to reality outcomes of the my seizure gauge epilepsy innovation institute workshop. Eneuro. https://doi.org/10.1523/ENEURO.0349-17.2017

    Article  PubMed  PubMed Central  Google Scholar 

  • England H (2018) Highways agency network journey time and traffic flow data. Guildford, UK, Highways England

  • Epilepsy foundation, michigan. https://epilepsymichigan.org/page.php?id=358

  • Fan M, Chou C-A (2018) Detecting abnormal pattern of epileptic seizures via temporal synchronization of EEG signals. IEEE Trans Biomed Eng 66(3):601–608

    Article  PubMed  Google Scholar 

  • Freedman D, Pisani R, Purves R (2020) Statistics: Fourth international student edition. WW Nort Co Httpswww Amaz ComStatistics-Fourth-Int-Stud-Free Accessed 22

  • Garg Y, Candan KS (2021) Sdma: Saliency-driven mutual cross attention for multi-variate time series. In: 2020 25th International conference on pattern recognition (ICPR). IEEE, pp. 7242–7249

  • Garg Y, Candan KS (2021) Xm2a: Multi-scale multi-head attention with cross-talk for multi-variate time series analysis. In: 2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 151–157. IEEE

  • Graves A, Graves A (2012) Long short-term memory. Supervised sequence labelling with recurrent neural networks, 37–45

  • Hain D, Jurowetzki R (2020) Introduction to rare-event predictive modeling for inferential statisticians–a hands-on application in the prediction of breakthrough patents. arXiv preprint arXiv:2003.13441

  • Islam M, Tanaka T, Iimura Y, Mitsuhashi T, Sugano H, Wang D, Molla M, Islam K et al (2020) Statistical features in high-frequency bands of interictal ieeg work efficiently in identifying the seizure onset zone in patients with focal epilepsy. Entropy 22(12):1415

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  • Jiang Y, Wu D, Deng Z, Qian P, Wang J, Wang G, Chung F-L, Choi K-S, Wang S (2017) Seizure classification from eeg signals using transfer learning, semi-supervised learning and tsk fuzzy system. IEEE Trans Neural Syst Rehabil Eng 25(12):2270–2284

    Article  PubMed  Google Scholar 

  • Khan SH, Hayat M, Bennamoun M, Sohel FA, Togneri R (2017) Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Trans Neural Netw Learn Syst 29(8):3573–3587

    PubMed  Google Scholar 

  • Khattar A, Quadri S (2022) Camm: cross-attention multimodal classification of disaster-related tweets. IEEE Access 10:92889–92902

    Article  Google Scholar 

  • Kim J, Jeong J, Shin J (2020) M2m: Imbalanced classification via major-to-minor translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13896–13905

  • Kirik S, Çatak Z (2021) Vitamin b12 deficiency observed in children with first afebrile seizures. Cureus. https://doi.org/10.7759/cureus.13745

    Article  PubMed  PubMed Central  Google Scholar 

  • Klem GH, Lüders HO, Jasper H, Elger C et al (1999) The ten-twenty electrode system of the international federation. Electroencephalogr Clin Neurophysiol 52(3):3–6

    CAS  Google Scholar 

  • Li J, Fong S, Mohammed S, Fiaidhi J, Chen Q, Tan Z (2016) Solving the under-fitting problem for decision tree algorithms by incremental swarm optimization in rare-event healthcare classification. J Med Imag Health Inform 6(4):1102–1110

    Article  Google Scholar 

  • Li J, Liu L-S, Fong S, Wong RK, Mohammed S, Fiaidhi J, Sung Y, Wong KK (2017) Adaptive swarm balancing algorithms for rare-event prediction in imbalanced healthcare data. PLoS ONE 12(7):0180830

    Article  Google Scholar 

  • Li G, Lee CH, Jung JJ, Youn YC, Camacho D (2020) Deep learning for EEG data analytics: a survey. Concurr Comput: Pract Exper 32(18):5199

    Article  Google Scholar 

  • Li J, Fong S, Hu S, Chu VW, Wong RK, Mohammed S, Dey N (2017) Rare event prediction using similarity majority under-sampling technique. In: International Conference on Soft Computing in Data Science. Springer, pp. 23–39

  • Liu S, Poccia SR, Candan KS, Sapino ML, Wang X (2018) Robust multi-variate temporal features of multi-variate time series. ACM Trans Multim Comp, Commun Appl (TOMM) 14(1):7

    Google Scholar 

  • Liu L, Chen W, Cao G (2016) Prediction of neonatal amplitude-integrated eeg based on lstm method. In: 2016 IEEE International conference on bioinformatics and biomedicine (BIBM). IEEE, pp. 497–500

  • Li Y, Yu R, Shahabi C, Liu Y (2017) Diffusion convolutional recurrent neural network: data-driven traffic forecasting. arXiv preprint arXiv:1707.01926

  • Lun X, Yu Z, Chen T, Wang F, Hou Y (2020) A simplified CNN classification method for MI-EEG via the electrode pairs signals. Front Hum Neurosci 14:338

    Article  PubMed  PubMed Central  Google Scholar 

  • Luong M-T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025

  • Natu M, Bachute M, Gite S, Kotecha K, Vidyarthi A (2022) Review on epileptic seizure prediction: machine learning and deep learning approaches. Comput Math Method Med. https://doi.org/10.1155/2022/7751263

    Article  Google Scholar 

  • Ozcan AR, Erturk S (2019) Seizure prediction in scalp EEG using 3d convolutional neural networks with an image-based approach. IEEE Trans Neural Syst Rehabil Eng 27(11):2284–2293

    Article  PubMed  Google Scholar 

  • Palatucci M, Pomerleau D, Hinton GE, Mitchell TM (2009) Zero-shot learning with semantic output codes. In: Advances in neural information processing systems, pp. 1410–1418

  • Park Y, Luo L, Parhi KK, Netoff T (2011) Seizure prediction with spectral power of EEG using cost-sensitive support vector machines. Epilepsia 52(10):1761–1770

    Article  PubMed  Google Scholar 

  • Patel KS, Zhao M, Ma H, Schwartz TH (2013) Imaging preictal hemodynamic changes in neocortical epilepsy. Neurosurg Focus 34(4):10

    Article  Google Scholar 

  • Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830

    MathSciNet  Google Scholar 

  • Peng M, Zhang Q, Xing X, Gui T, Huang X, Jiang Y-G, Ding K, Chen Z (2019) Trainable undersampling for class-imbalance learning. In: Proceedings of the AAAI conference on artificial intelligence, vol. 33, pp. 4707–4714

  • Pinto M, Coelho T, Leal A, Lopes F, Dourado A, Martins P, Teixeira C (2022) Interpretable EEG seizure prediction using a multiobjective evolutionary algorithm. Sci Rep 12(1):1–15

    Article  Google Scholar 

  • Population, population change, and estimated components of population change: April 1, 2010 to july 1, 2019 (nst-est2019-alldata). https://www.census.gov/data/tables/time-series/demo/popest/2010s-national-total.html

  • Ramgopal S, Thome-Souza S, Jackson M, Kadish NE, Fernández IS, Klehm J, Bosl W, Reinsberger C, Schachter S, Loddenkemper T (2014) Seizure detection, seizure prediction, and closed-loop warning systems in epilepsy. Epilep Behav 37:291–307

    Article  Google Scholar 

  • Ranjan C, Reddy M, Mustonen M, Paynabar K, Pourak K (2018) Dataset: rare event classification in multivariate time series. arXiv preprint arXiv:1809.10717

  • Ravindranath M, Candan KS, Sapino ML (2020) M2nn: rare event inference through multi-variate multi-scale attention. In: 2020 IEEE International conference on smart data services (SMDS). IEEE, pp. 53–62

  • Shahsavari S, McKelvey T, Ritzen CE, Rydenhag B (2011) Cerebrovascular mechanical properties and slow waves of intracranial pressure in TBI patients. IEEE Trans Biomed Eng 58(7):2072–2082

    Article  PubMed  Google Scholar 

  • Singh K, Malhotra J (2022) Two-layer LSTM network-based prediction of epileptic seizures using EEG spectral features. Compl Intell Syst. https://doi.org/10.1007/s40747-021-00627-z

    Article  Google Scholar 

  • Socher R, Ganjoo M, Manning CD, Ng A (2013) Zero-shot learning through cross-modal transfer. In: Advances in neural information processing systems, pp. 935–943

  • Stacey WC (2018) Seizure prediction is possible-now let’s make it practical. EBioMedicine 27:3–4

    Article  PubMed  PubMed Central  Google Scholar 

  • Storti SF, Del Felice A, Formaggio E, Boscolo Galazzo I, Bongiovanni LG, Cerini R, Fiaschi A, Manganotti P (2015) Spatial and temporal EEG-FMRI changes during preictal and postictal phases in a patient with posttraumatic epilepsy. Clin EEG Neurosci 46(3):247–252

    Article  PubMed  Google Scholar 

  • The new york times coronavirus (covid-19) cases and deaths in the united states. https://data.humdata.org/dataset/nyt-covid-19-data

  • Tibshirani R (1996) Regression shrinkage and selection via the lasso. J Roy Stat Soc: Ser B (Methodol) 58(1):267–288

    MathSciNet  Google Scholar 

  • Tong Y (2014) Seizures caused by pyridoxine (vitamin b6) deficiency in adults: a case report and literature review. Intract Rae Dis Res 3(2):52–56

    Google Scholar 

  • Tsiouris KM, Pezoulas VC, Zervakis M, Konitsiotis S, Koutsouris DD, Fotiadis DI (2018) A long short-term memory deep learning network for the prediction of epileptic seizures using EEG signals. Comput Biol Med 99:24–37

    Article  PubMed  Google Scholar 

  • Usman SM, Usman M, Fong S (2017) Epileptic seizures prediction using machine learning methods. Comput Math Meth Med. https://doi.org/10.1155/2017/9074759

    Article  MathSciNet  Google Scholar 

  • Usman SM, Khalid S, Aslam MH (2020) Epileptic seizures prediction using deep learning techniques. IEEE Access 8:39998–40007

    Article  Google Scholar 

  • Valderrama M, Nikolopoulos S, Adam C, Navarro V, Le Van Quyen M (2010) Patient-specific seizure prediction using a multi-feature and multi-modal EEG-ECG classification. In: XII Mediterranean conference on medical and biological engineering and computing 2010. Springer, pp. 77–80

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp. 5998–6008

  • Wang H-S, Kuo M (2007) Vitamin b\(\tilde{}\) 6 related epilepsy during childhood. Chang Gung Med J 30(5):396

    PubMed  Google Scholar 

  • Xiu Z, Tao C, Gao M, Davis C, Goldstein BA, Henao R (2021) Variational disentanglement for rare event modeling. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 10469–10477

  • Xu D, Zhang Z, Shi J (2022) Training data selection by categorical variables for better rare event prediction in multiple products production line. Electronics 11(7):1056

    Article  Google Scholar 

  • Yu B, Yin H, Zhu Z (2017) Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.04875

  • Zhu J, Xia Y, Wu L, He D, Qin T, Zhou W, Li H, Liu T-Y (2020) Incorporating bert into neural machine translation. arXiv preprint arXiv:2002.06823

Download references

Acknowledgements

Thanks to Dr Stephen Foldes and Austin Jacobson of Phoenix Children’s, Phoenix, AZ for their support with the work. Results presented in this paper were obtained using the Chameleon testbed supported by the National Science Foundation.

Funding

The work is primarily supported by Department Of Defense Grant No. W81XWH-19-1-0514. Some aspects of the research is also supported by NSF Grant Nos. #1909555, #2026860, #1827757.

Author information

Authors and Affiliations

  1. SCAI, Arizona State University, Tempe, AZ, USA

    Manjusha Ravindranath & K. Selçuk Candan

  2. Dipartimento di Informatica, University of Torino, Turin, Italy

    Maria Luisa Sapino

  3. Neurology, Phoenix Children’s, Phoenix, AZ, USA

    Brian Appavu

Authors
  1. Manjusha Ravindranath
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. K. Selçuk Candan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maria Luisa Sapino
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brian Appavu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manjusha Ravindranath.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

For data collection and analysis, the research is approved by the Phoenix Children’s Institutional Review Board (IRB #19-284).

Informed consent

For retrospective analysis as ours, informed consent was not required by the the Phoenix Children’s Institutional Review Board.

Additional information

Responsible editor: Panagiotis Papapetrou.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ravindranath, M., Candan, K.S., Sapino, M.L. et al. MMA: metadata supported multi-variate attention for onset detection and prediction. Data Min Knowl Disc (2024). https://doi.org/10.1007/s10618-024-01008-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10618-024-01008-z

Keywords

Search

Navigation