Abstract
Background
Referral of patients with heart failure (HF) who are at high mortality risk for specialist evaluation is recommended. Yet, most tools for identifying such patients are difficult to implement in electronic health record (EHR) systems.
Objective
To assess the performance and ease of implementation of Machine learning Assessment of RisK and EaRly mortality in Heart Failure (MARKER-HF), a machine-learning model that uses structured data that is readily available in the EHR, and compare it with two commonly used risk scores: the Seattle Heart Failure Model (SHFM) and Meta‐Analysis Global Group in Chronic (MAGGIC) Heart Failure Risk Score.
Design
Retrospective, cohort study.
Participants
Data from 6764 adults with HF were abstracted from EHRs at a large integrated health system from 1/1/10 to 12/31/19.
Main measures
One-year survival from time of first cardiology or primary care visit was estimated using MARKER-HF, SHFM, and MAGGIC. Discrimination was measured by the area under the receiver operating curve (AUC). Calibration was assessed graphically.
Key results
Compared to MARKER-HF, both SHFM and MAGGIC required a considerably larger amount of data engineering and imputation to generate risk score estimates. MARKER-HF, SHFM, and MAGGIC exhibited similar discriminations with AUCs of 0.70 (0.69–0.73), 0.71 (0.69–0.72), and 0.71 (95% CI 0.70–0.73), respectively. All three scores showed good calibration across the full risk spectrum.
Conclusions
These findings suggest that MARKER-HF, which uses readily available clinical and lab measurements in the EHR and required less imputation and data engineering than SHFM and MAGGIC, is an easier tool to identify high-risk patients in ambulatory clinics who could benefit from referral to a HF specialist.
Graphical Abstract
Similar content being viewed by others
Data Availability
Data will be made available upon reasonable request and with appropriate data use agreements in place.
References
Tsao CW, Aday AW, Almarzooq ZI et al (2023) Heart disease and stroke statistics-2023 update: a report from the American Heart Association. Circulation 147:e93–e621
Heidenreich PA, Bozkurt B, Aguilar D et al (2022) 2022 AHA/ACC/HFSA guideline for the management of heart failure: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation 145:e895–e1032
Morris AA, Khazanie P, Drazner MH, et al (2021) Guidance for timely and appropriate referral of patients with advanced heart failure: a scientific statement from the American Heart Association. Circulation 0:CIR0000000000001016
Kao DP, Trinkley KE, Lin C-T (2020) Heart failure management innovation enabled by electronic health records. JACC Heart Fail 8:223–233
Spratt SE, Pereira K, Granger BB et al (2017) Assessing electronic health record phenotypes against gold-standard diagnostic criteria for diabetes mellitus. J Am Med Inform Assoc 24:e121–e128
Slotwiner DJ (2016) Electronic health records and cardiac implantable electronic devices: new paradigms and efficiencies. J Interv Card Electrophysiol 47:29–35
Polubriaginof F, Salmasian H, Albert DA, Vawdrey DK (2018) Challenges with collecting smoking status in electronic health records. AMIA Annu Symp Proc 2017:1392–1400
Gothe H, Rajsic S, Vukicevic D et al (2019) Algorithms to identify COPD in health systems with and without access to ICD coding: a systematic review. BMC Health Serv Res 19:737–737
Adler ED, Voors AA, Klein L et al (2020) Improving risk prediction in heart failure using machine learning. Eur J Heart Fail 22:139–147
Greenberg B, Adler E, Campagnari C, Yagil A (2021) A machine learning risk score predicts mortality across the spectrum of left ventricular ejection fraction. Eur J Heart Fail 23:995–999
Williams BA, Agarwal S (2018) Applying the Seattle Heart Failure Model in the office setting in the era of electronic medical records. Circ J 82:724–731
Rich JD, Burns J, Freed BH et al (2018) Meta-Analysis Global Group in Chronic (MAGGIC) heart failure risk score: validation of a simple tool for the prediction of morbidity and mortality in heart failure with preserved ejection fraction. J Am Heart Assoc 7:e009594
Pocock SJ, Ariti CA, McMurray JJ et al (2013) Predicting survival in heart failure: a risk score based on 39 372 patients from 30 studies. Eur Heart J 34:1404–1413
Ng Fat Hing N, MacIver J, Chan D et al (2018) Utility of the Seattle Heart Failure Model for palliative care referral in advanced ambulatory heart failure. BMJ Support Palliat Care. https://doi.org/10.1136/bmjspcare-2018-001626
May HT, Horne BD, Levy WC et al (2007) Validation of the Seattle Heart Failure Model in a community-based heart failure population and enhancement by adding B-type natriuretic peptide. Am J of Cardiol 100:697–700
Li S, Marcus P, Nunez J et al (2019) Validity of the Seattle Heart Failure Model after heart failure hospitalization. ESC Heart Fail 6:509–515
Levy WC, Mozaffarian D, Linker DT et al (2006) The Seattle Heart Failure Model. Circulation 113:1424–1433
Khanam SS, Choi E, Son JW et al (2018) Validation of the MAGGIC (Meta-Analysis Global Group in Chronic Heart Failure) heart failure risk score and the effect of adding natriuretic peptide for predicting mortality after discharge in hospitalized patients with heart failure. PLoS One 13:e0206380
Allen LA, Matlock DD, Shetterly SM et al (2017) Use of risk models to predict death in the next year among individual ambulatory patients with heart failure. JAMA Cardiol 2:435–441
Starren JB, Winter AQ, Lloyd-Jones DM (2015) Enabling a learning health system through a unified enterprise data warehouse: the experience of the Northwestern University Clinical and Translational Sciences (NUCATS) Institute. Clin Transl Sci 8:269–271
Bavishi A, Bruce M, Ning H et al (2020) Predictive accuracy of heart failure-specific risk equations in an electronic health record-based cohort. Circ Heart Fail 13:e007462
Chronic Conditions Warehouse Condition Categories. www2.ccwdata.org. 2021. Accessed on 23 Jun 2023 at https://www2.ccwdata.org/web/guest/condition-categories
Lloyd-Jones DM (2010) Cardiovascular risk prediction. Circulation 121:1768–1777
DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Machine learning in Python. J Mach Learn Res 12:2825–28030
Davidson-Pilon C. Lifelines, survival analysis in Python (v0.26.4). 2021. Accessed 10 Novn 2022 at https://doi.org/10.5281/zenodo.5745573
Wehbe RM, Khan SS, Shah SJ, Ahmad FS (2020) Predicting high-risk patients and high-risk outcomes in heart failure. Heart Fail Clin 16:387–407
Okada A, Kaneko H, Konishi M et al (2023) A machine-learning-based prediction of non-home discharge among acute heart failure patients. Clin Res Cardiol. https://doi.org/10.1007/s00392-023-02209-0
Raparelli V, Romiti GF, Di Teodoro G et al (2023) A machine-learning based bio-psycho-social model for the prediction of non-obstructive and obstructive coronary artery disease. Clin Res Cardiol 112:1263–1277
Bisson A, Lemrini Y, El-Bouri W et al (2023) Prediction of incident atrial fibrillation in post-stroke patients using machine learning: a French nationwide study. Clin Res Cardiol 112:815–823
Maddox TM, Albert NM, Borden WB et al (2017) The learning healthcare system and cardiovascular care: a scientific statement from the American Heart Association. Circulation 135:e826–e857
Herr JJ, Ravichandran A, Sheikh FH et al (2021) Practices of referring patients to advanced heart failure centers. J Card Fail 27(11):1251–1259
Larsson J, Kristensen SL, Madelaire C et al (2021) Socioeconomic disparities in referral for invasive hemodynamic evaluation for advanced heart failure: a nationwide cohort study. Circ Heart Fail 4(10):e008662
Di Tanna GL, Wirtz H, Burrows KL, Globe G (2020) Evaluating risk prediction models for adults with heart failure: a systematic literature review. PLoS One 15:e0224135
Rahimi K, Bennett D, Conrad N et al (2014) Risk prediction in patients with heart failure. JACC Heart Fail 2:440–446
Dettling A, Weimann J, Sundermeyer J et al (2024) Association of systemic inflammation with shock severity, 30-day mortality, and therapy response in patients with cardiogenic shock. Clin Res Cardiol 113:324–335
Prokopidis K, Triantafyllidis KK, Kechagias KS, et al (2023) Are sarcopenia and its individual components linked to all-cause mortality in heart failure? A systematic review and meta-analysis. Clin Res Cardiol. [Epub ahead of print] https://doi.org/10.1007/s00392-023-02360-8
Göbel S, Braun AS, Hahad O, et al (2024) Etiologies and predictors of mortality in an all-comer population of patients with non-ischemic heart failure. Clin Res Cardiol. [Epub ahead of print] https://doi.org/10.1007/s00392-023-02354-6
Ghassemi M, Oakden-Rayner L, Beam AL (2021) The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health 3:e745–e750
Guo LL, Pfohl SR, Fries J et al (2021) Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine. Appl Clin Inform 12:808–815
Finlayson SG, Subbaswamy A, Singh K et al (2021) The clinician and dataset shift in artificial intelligence. N Engl J Med 385:283–286
Ghassemi M, Naumann T, Schulam P et al (2020) A review of challenges and opportunities in machine learning for health. AMIA Jt Summits Transl Sci Proc 2020:191–200
Kashyap S, Morse KE, Patel B, Shah NH (2021) A survey of extant organizational and computational setups for deploying predictive models in health systems. J Am Med Inform Assoc 28:2445–2450
Ahmad T, Yamamoto Y, Biswas A et al (2021) REVeAL-HF: design and rationale of a pragmatic randomized controlled trial embedded within routine clinical practice. JACC Heart Fail 9:409–419
Ahmad T, Desai NR, Yamamoto Y et al (2022) Alerting clinicians to 1-year mortality risk in patients hospitalized with heart failure: the REVEAL-HF randomized clinical trial. JAMA Cardiol 7(9):905–912
Funding
Dr. Ahmad was supported by grants from the Agency for Healthcare Research and Quality (K12HS026385), National Institutes of Health/National Heart, Lung, and Blood Institute (K23HL155970), and the American Heart Association (AHA number 856917). Research reported in this publication was supported, in part, by the National Institutes of Health’s National Center for Advancing Translational Sciences (UL1TR001422).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Dr. Ahmad has received consulting fees from Teladoc Livongo and Pfizer outside the submitted work. Dr. Petito received research support from Omron Healthcare Co. Ltd. outside the submitted work. The remaining authors declare no financial or non-financial competing interests.
Disclaimer
The statements presented in this work are solely the responsibility of the author(s) and do not necessarily represent the official views of the National Institutes of Health or the American Heart Association.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ahmad, F.S., Hu, T.L., Adler, E.D. et al. Performance of risk models to predict mortality risk for patients with heart failure: evaluation in an integrated health system. Clin Res Cardiol (2024). https://doi.org/10.1007/s00392-024-02433-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00392-024-02433-2