Survival analysis using deep learning with medical imaging

Samantha Morrison; Constantine Gatsonis; Ani Eloyan; Jon Arni Steingrimsson

doi:10.1515/ijb-2022-0113

Published online by De Gruyter June 14, 2023

Survival analysis using deep learning with medical imaging

Samantha Morrison , Constantine Gatsonis , Ani Eloyan and Jon Arni Steingrimsson

From the journal The International Journal of Biostatistics

https://doi.org/10.1515/ijb-2022-0113

Showing a limited preview of this publication:

Abstract

There is widespread interest in using deep learning to build prediction models for medical imaging data. These deep learning methods capture the local structure of the image and require no manual feature extraction. Despite the importance of modeling survival in the context of medical data analysis, research on deep learning methods for modeling the relationship of imaging and time-to-event data is still under-developed. We provide an overview of deep learning methods for time-to-event outcomes and compare several deep learning methods to Cox model based methods through the analysis of a histology dataset of gliomas.

Keywords: convolutional neural networks; Doubly Robust estimation; survival analysis; time-to-event outcomes

Corresponding author: Jon Arni Steingrimsson, Department of Biostatistics, School of Public Health, Brown University, Providence, RI, USA, E-mail: jon_steingrimsson@brown.edu

Funding source: National Institute of General Medical Sciences

Award Identifier / Grant number: U54GM115677

Funding source: National Cancer Institute

Award Identifier / Grant number: U10CA180794

Award Identifier / Grant number: U10CA180820

Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
Research funding: This work was supported by the National Institute of General Medical Sciences (U54GM115677) and National Cancer Institute (U10CA180794, U10CA180820).
Conflict of interest statement: The authors declare no conflicts of interest regarding this article.

References

1. Klein, JP, Moeschberger, ML. Survival analysis: techniques for censored and truncated data. New York: Springer Science & Business Media; 2006.Search in Google Scholar

2. Zhao, L. Deep neural networks for predicting restricted mean survival times. Bioinformatics 2020;36:5672–7. https://doi.org/10.1093/bioinformatics/btaa1082.Search in Google Scholar PubMed PubMed Central

3. Sun, Y, Chiou, SH, Wang, MC. Roc-guided survival trees and ensembles. Biometrics 2020;76:1177–89. https://doi.org/10.1111/biom.13213.Search in Google Scholar PubMed PubMed Central

4. Bou-Hamad, I, Larocque, D, Ben-Ameur, H. A review of survival trees. Stat Surv 2011;5:44–71. https://doi.org/10.1214/09-ss047.Search in Google Scholar

5. Cui, Y, Zhu, R, Kosorok, M. Tree based weighted learning for estimating individualized treatment rules with censored data. Electron J Stat 2017;11:3927. https://doi.org/10.1214/17-ejs1305.Search in Google Scholar

6. Steingrimsson, JA, Morrison, S. Deep learning for survival outcomes. Stat Med 2020;39:2339–49. https://doi.org/10.1002/sim.8542.Search in Google Scholar PubMed PubMed Central

7. Litjens, G, Kooi, T, Bejnordi, BE, Setio, AAA, Ciompi, F, Ghafoorian, M, et al.. A survey on deep learning in medical image analysis. Med Image Anal 2017;42:60–88. https://doi.org/10.1016/j.media.2017.07.005.Search in Google Scholar PubMed

8. Mobadersany, P, Yousefi, S, Amgad, M, Gutman, DA, Barnholtz-Sloan, JS, Vega, JEV, et al.. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci USA 2018;115:E2970–79. https://doi.org/10.1073/pnas.1717139115.Search in Google Scholar PubMed PubMed Central

9. Li, H, Boimel, P, Janopaul-Naylor, J, Zhong, H, Xiao, Y, Ben-Josef, E, Fan, Y. Deep convolutional neural networks for imaging data based survival analysis of rectal cancer. In: 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019). IEEE; 2019:846–9 pp.10.1109/ISBI.2019.8759301Search in Google Scholar PubMed PubMed Central

10. Zhu, X, Yao, J, Huang, J. Deep convolutional neural network for survival analysis with pathological images. In: 2016 IEEE international conference on bioinformatics and biomedicine (BIBM). IEEE; 2016:544–7 pp.10.1109/BIBM.2016.7822579Search in Google Scholar

11. Szegedy, C, Liu, W, Jia, Y, Sermanet, P, Reed, S, Anguelov, D, et al.. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2015:1–9 pp.10.1109/CVPR.2015.7298594Search in Google Scholar

12. Pedano, N, Flanders, A, Scarpace, L, Mikkelsen, T, Eschbacher, J, Hermes, B, Ostrom, Q. Radiology data from the cancer genome atlas low grade glioma [tcga-lgg] collection. The Cancer Imaging Archive; 2016. Available from: https://www.cancerimagingarchive.net/.Search in Google Scholar

13. Clark, K, Vendt, B, Smith, K, Freymann, J, Kirby, J, Koppel, P, et al.. The cancer imaging archive (tcia): maintaining and operating a public information repository. J Digit Imag 2013;26:1045–57. https://doi.org/10.1007/s10278-013-9622-7.Search in Google Scholar PubMed PubMed Central

14. Goodfellow, I, Bengio, Y, Courville, A. Deep Learning. MIT Press; 2016. Available from: http://www.deeplearningbook.org.Search in Google Scholar

15. LeCun, Y, Bengio, Y. Convolutional networks for images, speech, and time series. In: The handbook of brain theory and neural networks. Cambridge: MIT Press; 1995, 3361:1995 p.Search in Google Scholar

16. LeCun, Y, Bottou, L, Bengio, Y, Haffner, P. Gradient-based learning applied to document recognition. Proc IEEE 1998;86:2278–324. https://doi.org/10.1109/5.726791.Search in Google Scholar

17. Caterini, AL, Chang, DE. Deep neural networks in a mathematical framework. New York: Springer; 2018.10.1007/978-3-319-75304-1Search in Google Scholar

18. Murphy, SA, Van der Vaart, AW. On profile likelihood. J Am Stat Assoc 2000;95:449–65. https://doi.org/10.1080/01621459.2000.10474219.Search in Google Scholar

19. Lao, J, Chen, Y, Li, ZC, Li, Q, Zhang, J, Liu, J, et al.. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci Rep 2017;7:10353. https://doi.org/10.1038/s41598-017-10649-8.Search in Google Scholar PubMed PubMed Central

20. Friedman, J, Hastie, T, Tibshirani, R. Regularization paths for generalized linear models via coordinate descent. J Stat Software 2010;33:1. https://doi.org/10.18637/jss.v033.i01.Search in Google Scholar

21. Aerts, HJ, Velazquez, ER, Leijenaar, RT, Parmar, C, Grossmann, P, Carvalho, S, et al.. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun 2014;5:1–9. https://doi.org/10.1038/ncomms5006.Search in Google Scholar PubMed PubMed Central

22. Davatzikos, C, Rathore, S, Bakas, S, Pati, S, Bergman, M, Kalarot, R, et al.. Cancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome. J Med Imag 2018;5:011018. https://doi.org/10.1117/1.jmi.5.1.011018.Search in Google Scholar

23. Pati, S, Singh, A, Rathore, S, Gastounioti, A, Bergman, M, Ngo, P, et al.. The cancer imaging phenomics toolkit (captk): technical overview. In: International MICCAI brainlesion workshop. Springer; 2019:380–94 pp.10.1007/978-3-030-46643-5_38Search in Google Scholar PubMed PubMed Central

24. Faraggi, D, Simon, R. A neural network model for survival data. Stat Med 1995;14:73–82. https://doi.org/10.1002/sim.4780140108.Search in Google Scholar PubMed

25. Katzman, JL, Shaham, U, Cloninger, A, Bates, J, Jiang, T, Kluger, Y. DeepSurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC Med Res Methodol 2018;18:1–12. https://doi.org/10.1186/s12874-018-0482-1.Search in Google Scholar

26. Yousefi, S, Amrollahi, F, Amgad, M, Dong, C, Lewis, JE, Song, C, et al.. Predicting clinical outcomes from large scale cancer genomic profiles with deep survival models. Sci Rep 2017;7:11707. https://doi.org/10.1038/s41598-017-11817-6.Search in Google Scholar

27. Steingrimsson, JA, Diao, L, Strawderman, RL. Censoring unbiased regression trees and ensembles. J Am Stat Assoc 2019;114:370–83. https://doi.org/10.1080/01621459.2017.1407775.Search in Google Scholar

28. Steingrimsson, JA, Diao, L, Molinaro, AM, Strawderman, RL. Doubly robust survival trees. Stat Med 2016;35:3595–612. https://doi.org/10.1002/sim.6949.Search in Google Scholar

29. Tajbakhsh, N, Shin, JY, Gurudu, SR, Hurst, RT, Kendall, CB, Gotway, MB, et al.. Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans Med Imag 2016;35:1299–312. https://doi.org/10.1109/tmi.2016.2535302.Search in Google Scholar

30. Simonyan, K, Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.Search in Google Scholar

31. Ruder, S. An overview of gradient descent optimization algorithms. 2016. arXiv preprint arXiv:1609.04747.Search in Google Scholar

32. Graf, E, Schmoor, C, Sauerbrei, W, Schumacher, M. Assessment and comparison of prognostic classification schemes for survival data. Stat Med 1999;18:2529–45. https://doi.org/10.1002/(sici)1097-0258(19990915/30)18:17/18<2529::aid-sim274>3.0.co;2-5.10.1002/(SICI)1097-0258(19990915/30)18:17/18<2529::AID-SIM274>3.0.CO;2-5Search in Google Scholar

33. LeBlanc, M, Crowley, J. Survival trees by goodness of split. J Am Stat Assoc 1993;88:457–67. https://doi.org/10.1080/01621459.1993.10476296.Search in Google Scholar

34. Lostritto, K, Strawderman, RL, Molinaro, AM. A partitioning deletion/substitution/addition algorithm for creating survival risk groups. Biometrics 2012;68:1146–56. https://doi.org/10.1111/j.1541-0420.2012.01756.x.Search in Google Scholar

35. Hou, L, Samaras, D, Kurc, TM, Gao, Y, Davis, JE, Saltz, JH. Patch-based convolutional neural network for whole slide tissue image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016:2424–33 pp.10.1109/CVPR.2016.266Search in Google Scholar

36. Zhu, X, Yao, J, Zhu, F, Huang, J. Wsisa: making survival prediction from whole slide histopathological images. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2017:7234–42 pp.10.1109/CVPR.2017.725Search in Google Scholar

37. Ribeiro, MT, Singh, S, Guestrin, C. “Why should I trust you?” Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining; 2016:1135–44 pp.10.1145/2939672.2939778Search in Google Scholar

38. Wagner, J, Kohler, JM, Gindele, T, Hetzel, L, Wiedemer, JT, Behnke, S. Interpretable and fine-grained visual explanations for convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2019:9097–107 pp.10.1109/CVPR.2019.00931Search in Google Scholar

Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/ijb-2022-0113).

Received: 2022-09-14

Accepted: 2023-02-24

Published Online: 2023-06-14