Skip to main content
SpringerLink
Log in

Explaining deep multi-class time series classifiers

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Explainability helps users trust deep learning solutions for time series classification. However, existing explainability methods for multi-class time series classifiers focus on one class at a time, ignoring relationships between the classes. Instead, when a classifier is choosing between many classes, an effective explanation must show what sets the chosen class apart from the rest. We now formalize this notion, studying the open problem of class-specific explainability for deep time series classifiers, a challenging and impactful problem setting. We design a novel explainability method, DEMUX, which learns saliency maps for explaining deep multi-class time series classifiers by adaptively ensuring that its explanation spotlights the regions in an input time series that a model uses specifically to its predicted class. DEMUX adopts a gradient-based approach composed of three interdependent modules that combine to generate consistent, class-specific saliency maps that remain faithful to the classifier’s behavior yet are easily understood by end users. We demonstrate that DEMUX outperforms nine state-of-the-art alternatives on seven popular datasets when explaining two types of deep time series classifiers. We analyze runtime performance, show the impacts of hyperparameter selection, and introduce a detailed study of perturbation methods for time series. Further, through a case study, we demonstrate that DEMUX’s explanations indeed highlight what separates the predicted class from the others in the eyes of the classifier.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Algorithm 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Dixon MF, London J (2020) Financial forecasting with \(\alpha \)-RNNs: a time series modeling approach. Front Appl Math Stat

  2. Fulcher BD, Jones NS (2017) hctsa: A computational framework for automated time-series phenotyping using massive feature extraction. Cell Syst 5(5):527–5313

    Article  CAS  PubMed  Google Scholar 

  3. Che Z, Purushotham S, Cho K, Sontag DA, Liu Y (2018) Recurrent neural networks for multivariate time series with missing values. Sci Rep 8

  4. Li G, Choi B, Xu J, Bhowmick SS, Chun K-P, Wong GL (2021) Shapenet: A shapelet-neural network approach for multivariate time series classification. In: AAAI

  5. Tonekaboni S, Joshi S, Duvenaud D, Goldenberg A (2020) What went wrong and when? Instance-wise feature importance for time-series models. Adv Neural Inf Process Syst

  6. Schlegel U, Vo DL, Keim DA, Seebacher D (2021) Ts-mule: Local interpretable model-agnostic explanations for time series forecast models. In: PKDD/ECML Workshops

  7. Parvatharaju PS, Doddaiah R, Hartvigsen T, Rundensteiner EA (2021) Learning saliency maps to explain deep time series classifiers. In: Proceedings of the 30th ACM international conference on information & knowledge management

  8. Crabbe J, van der Schaar M (2021) Explaining time series predictions with dynamic masks. In: ICML

  9. Bento J, Saleiro P, Cruz AF, Figueiredo MAT, Bizarro P (2020) Timeshap: explaining recurrent models through sequence perturbations. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining

  10. Guidotti R, Monreale A, Spinnato F, Pedreschi D, Giannotti F (2020) Explaining any time series classifier. In: International conference on cognitive machine intelligence

  11. Guillemé M, Masson V, Rozé L, Termier A (2019) Agnostic local explanation for time series classification. In: ICTAI, pp 432–439

  12. Ribeiro MT, Singh S, Guestrin C (2016) “Why should I trust you?”: explaining the predictions of any classifier. CoRR arXiv:1602.04938

  13. Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. In: NIPS

  14. Ismail AA, Gunady M, Bravo HC, Feizi S (2020) Benchmarking deep learning interpretability in time series predictions. In: Proceedings of the 34th international conference on neural information processing systems. NIPS’20. Curran Associates Inc., Red Hook

  15. Shimoda W, Yanai K (2016) Distinct class-specific saliency maps for weakly supervised semantic segmentation. In: ECCV

  16. Kaur H, Nori H, Jenkins S, Caruana R, Wallach H, Wortman Vaughan J (2020) Interpreting interpretability: Understanding data scientists’ use of interpretability tools for machine learning. In: Proceedings of the 2020 CHI conference on human factors in computing systems

  17. Ribeiro A, Ribeiro M, Paixão G, Oliveira D, Gomes P, Canazart A, Ferreira P, Andersson C, Macfarlane P (2020) Automatic diagnosis of the 12-lead ECG using a deep neural network. Nat Commun 11(1)

  18. Mujkanovic F, Doskoč V, Schirneck M, Schäfer P, Friedrich T (2020) timeXplain—A framework for explaining the predictions of time series classifiers. arXiv e-prints 2007.07606 [cs.LG]

  19. Fong R, Vedaldi A (2017) Interpretable explanations of black boxes by meaningful perturbation. In: ICCV, pp 3449–3457

  20. Kumar IE, Venkatasubramanian S, Scheidegger CE, Friedler SA (2020) Problems with shapley-value-based explanations as feature importance measures. In: ICML

  21. Fong R, Patrick M, Vedaldi A (2019) Understanding deep networks via extremal perturbations and smooth masks. In: ICCV, pp. 2950–2958

  22. Doddaiah R, Parvatharaju P, Rundensteiner E, Hartvigsen T (2022) Class-specific explainability for deep time series classifiers. In: 2022 IEEE International conference on data mining (ICDM)

  23. Kelany OE, Aly S, Ismail MA (2020) Deep learning model for financial time series prediction. In: 2020 14th International conference on innovations in information technology (IIT)

  24. Mehdiyev N, Lahann J, Emrich A, Enke D, Fettke P, Loos P (2017) Time series classification using deep learning for process planning: a case from the process industry. Proc Comput Sci 114:242–249

    Article  Google Scholar 

  25. Zhang Y, Zhang Z, Zhang Y, Bao J, Zhang Y, Deng H (2019) Human activity recognition based on motion sensor using u-net. IEEE Access 7:75213–75226

    Article  Google Scholar 

  26. Schlegel U, Arnout H, El-Assady M, Oelke D, Keim DA (2019) Towards a rigorous evaluation of xai methods on time series. In: 2019 IEEE/CVF International conference on computer vision workshop (ICCVW), pp 4197–4201

  27. Zhang H, Cissé M, Dauphin YN, Lopez-Paz D (2018) mixup: Beyond empirical risk minimization. In: 6th International conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, conference track proceedings. OpenReview.net

  28. Satopaa V, Albrecht JR, Irwin DE, Raghavan B (2011) Finding a “kneedle” in a haystack: Detecting knee points in system behavior. In: 2011 31st international conference on distributed computing systems workshops

  29. Schaul T, Quan J, Antonoglou I, Silver D (2016) Prioritized experience replay. CoRR arXiv:1511.05952

  30. Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller MA (2013) Playing Atari with deep reinforcement learning. arXiv arXiv:1312.5602

  31. Gisler C, Ridi A, Zufferey D, Khaled OA, Hennebert J (2013) Appliance consumption signature database and recognition test protocols. In: 2013 8th International workshop on systems, signal processing and their applications (WoSSPA), pp 336–341

  32. Chen Y, Keogh E, Hu B, Begum N, Bagnall A, Mueen A, Batista G (2015) The UCR time series classification archive. www.cs.ucr.edu/~eamonn/time_series_data/

  33. Roverso D (2000) Multivariate temporal classification by windowed wavelet decomposition and recurrent neural networks

  34. Baldridge AM, Hook SJ, Grove CI, Rivera G (2009) The aster spectral library version 2.0. Remote Sens Environ 113:711–715. https://doi.org/10.1016/j.rse.2008.11.007

    Article  ADS  Google Scholar 

  35. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng C-K, Stanley HE (2000) Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation 101(23):215–20

    Article  Google Scholar 

  36. Al-Jowder O, Kemsley EK, Wilson RH (1997) Mid-infrared spectroscopy and authenticity problems in selected meats. Food Chem 59(2):195–201. https://doi.org/10.1016/S0308-8146(96)00289-0

    Article  CAS  Google Scholar 

  37. Fawaz HI, Forestier G, Weber J, Idoumghar L, Muller P-A (2018) Deep learning for time series classification: a review. Data Min Knowl Disc 33:917–963

    Article  MathSciNet  Google Scholar 

  38. Wang Z, Yan W, Oates T (2017) Time series classification from scratch with deep neural networks: a strong baseline. IJCNN

  39. Charnes A, Golany B, Keane M, Rousseau J (1988) Extremal principle solutions of games in characteristic function form: core, Chebychev and Shapley value generalizations. Econometrics of planning and efficiency. Springer, Dordrecht, pp 123–133

    Chapter  Google Scholar 

  40. Petsiuk V, Das A, Saenko K (2018) Rise: randomized input sampling for explanation of black-box models. CoRR arXiv:1806.07421

  41. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. CoRR arXiv:1412.6980

  42. Biewald L (2020) Experiment tracking with weights and biases. Software available from wandb.com. https://www.wandb.com/

  43. Hama N, Mase M, Owen AB (2022) Deletion and insertion tests in regression models

Download references

Author information

Author notes

  1. Ramesh Doddaiah and Prathyush S. Parvatharaju contributed equally to this work.

Authors and Affiliations

  1. Data Science, WPI, 100 Institute Rd, Worcester, MA, 01609, USA

    Ramesh Doddaiah, Prathyush S. Parvatharaju & Elke Rundensteiner

  2. CSAIL, MIT, 77 Massachusetts Ave, Cambridge, MA, 02139, USA

    Thomas Hartvigsen

Authors
  1. Ramesh Doddaiah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Prathyush S. Parvatharaju
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elke Rundensteiner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thomas Hartvigsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Ramesh Doddaiah and Prathyush Parvatharaju contributed equally to the main manuscript text. All authors reviewed the manuscript.

Corresponding author

Correspondence to Ramesh Doddaiah.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Doddaiah, R., Parvatharaju, P.S., Rundensteiner, E. et al. Explaining deep multi-class time series classifiers. Knowl Inf Syst (2024). https://doi.org/10.1007/s10115-024-02073-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10115-024-02073-y

Keywords

Search

Navigation