Clinical paperDevelopment and validation of the SARICA score to predict survival after return of spontaneous circulation in out of hospital cardiac arrest using an interpretable machine learning framework
Introduction
Out-of-hospital cardiac arrest (OHCA) is a major healthcare issue worldwide, with high incidence rates and poor outcomes.1., 2. In the United States, the disability-adjusted life years (DALY) contributed by OHCA was ranked fifth, a testament to its huge health burden.3 Globally, the median estimated incidence for OHCA attended by the Emergency Medical Services (EMS) was 82.1 per 100,000 person-years,2 while the percentage survival to discharge was only 7.0%.2
Advanced hospital treatment modalities that can improve outcomes for post-OHCA patients are available.4 However, it is not feasible to offer these treatments to all OHCA patients as they can be resource-intensive, and unethical to provide to patients in whom they are futile.5 There is thus an unmet need for accurate prognostication tools to assist clinical decision-making and inform rational allocation of healthcare resources, such as intensive care unit (ICU) beds for OHCA patients at the Emergency Department (ED). Additionally, for families and caregivers, objective and accurate information on a patient’s outlook at an early stage can facilitate communication,6 by eliciting appropriate expectations, and where necessary, closure.
Although key predictors of OHCA survival have been identified,7 the prediction of OHCA survival in current practice remains reliant on clinical gestalt which can be widely variable, with no simple prediction tool available for routine use. A recent systematic review of existing prognostication tools by Gue et al. identified several existing OHCA prognostication risk scores with good predictive ability.8., 9., 10., 11. However, these scores are limited by difficulty in computation, predictors that might be subjected to recall bias or may not be static over time, including some that are not readily available at the time the patient is in the ED.8 These drawbacks make it challenging to recommend their routine use in clinical practice.
Recently, machine learning (ML) prediction models have been developed and deployed to prognosticate OHCA with high accuracy.12., 13. Compared to usual care methods and clinical prediction tools based on conventional statistical models, ML has been shown to have better discriminatory abilities in several different diagnostic and prognostic predictions.14 However, these ML models require extensive data points to function optimally and are technically difficult to deploy in existing hospital infrastructure.15 The “black box” approach of developing ML prognostication models is also a concern for clinicians and thus discourages them from fully trusting the score in actual practice16 - especially in high-stakes clinical decisions. To overcome the complexities and difficulties in interpretation with ML models, AutoScore,17 an easy-to-use, automatic, ML-based clinical score generator was created. Autoscore has been used to develop simple point-based clinical scores to predict short-term mortality risk at ED triage18 and inpatient acute kidney injury,19 with predictive ability that outperforms existing benchmark scores. Autoscore has not been tested on the OHCA population.
The aim of this study was to develop and internally validate Survival after Return of spontaneous circulation in Cardiac Arrest (SARICA], a simple point-based clinical prediction rule, using objective predictors typically available at ED admission to prognosticate 30-day survival of OHCA patients. AutoScore, which is an ML-based algorithm, was used to systematically develop SARICA through a transparent and interpretable framework.
Section snippets
Data source and study setting
Singapore is a densely-populated urban city-state with a population of 5.7 million and a population density of 7,810 per km2.20 A population-based survey in 2010 showed that 31.4% of responders had ever been trained in cardiopulmonary resuscitation (CPR), 10.7% had ever been trained in automated external defibrillation, while 9.6% and 3.7% possessed valid certificates for these skills, respectively.21 Emergency Medical Services (EMS) are provided by the Singapore Civil Defence Force, which
Baseline characteristics of study population
18,359 OHCA cases occurred between April 2010 and December 2018, of which 18,097 were attended by EMS. 2 cases were excluded due to missing data, and 12,125 cases were excluded due to ROSC not attained. Finally, 5970 cases qualified for analysis. The population flow diagram shown in Fig. 1 demonstrates the selection of study participants and subsequent segmentation into training, validation, and testing cohorts.
Baseline characteristics of all included OHCA cases were summarised in Table 1. 855
Discussion
In this study, using data from the large-scale multi-centre cardiac arrest registry in Singapore and the AutoScore framework, we developed and internally validated the SARICA score, a 10-point clinical score based on 3 variables readily available at ED admission, to predict OHCA patient survival post-ROSC. The predictive ability of our score performed reasonably well, with an AUC of 0.87 when evaluated on our testing cohort.
Conclusion
We developed and internally validated the SARICA score to accurately predict survival of OHCA patients with ROSC at the time of admission. SARICA is clinically practical and developed using an interpretable machine-learning framework. SARICA has unknown generalizability pending external validation studies.
Ethical approval and Consent to participate
Not applicable.
Consent for publication
All authors have approved the manuscript and agree with its submission to Resuscitation.
Availability of supporting data
Not applicable.
Funding
A.F.W.H was supported by the Estate of Tan Sri Khoo Teck Puat (Khoo Clinical Scholars Programme), Khoo Pilot Award (KP/2019/0034), Duke-NUS Medical School and National Medical Research Council (NMRC/CS_Seedfd/012/2018). This work was supported by the National Research Foundation of Singapore through the Virtual Singapore Program (Grant No. NRF2017VSG-AT3DCM001-031) and National Medical Research Council, Clinician Scientist Awards, Singapore (NMRC/CSA/024/2010 and NMRC/CSA/0049/2013).
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (32)
- et al.
Out-of-hospital cardiac arrest: current concepts
Lancet
(2018) - et al.
Global incidences of out-of-hospital cardiac arrest and survival rates: Systematic review of 67 prospective studies
Resuscitation
(2010) - et al.
Out-of-hospital cardiac arrest: A systematic review of current risk scores to predict survival
Am Heart J
(2021) - et al.
Prediction rules for estimating neurologic outcome following out-of-hospital cardiac arrest
Resuscitation
(2004) - et al.
Deep-learning-based out-of-hospital cardiac arrest prognostic system to predict clinical outcomes
Resuscitation
(2019) - et al.
National population based survey on the prevalence of first aid, cardiopulmonary resuscitation and automated external defibrillator skills in Singapore
Resuscitation
(2013) - et al.
Outcomes for out-of-hospital cardiac arrests across 7 countries in Asia: The Pan Asian Resuscitation Outcomes Study (PAROS)
Resuscitation
(2015) A prediction survival model for out-of-hospital cardiopulmonary resuscitations
J Crit Care
(2011)- et al.
A prediction tool for initial out-of-hospital cardiac arrest survivors
Resuscitation
(2014) - et al.
Effect of age on prehospital cardiac resuscitation outcome
Am J Emerg Med
(1995)
Outcome and characteristics of out-of-hospital cardiac arrest according to location of arrest: A report from a large-scale, population-based study in Osaka, Japan
Resuscitation
Gender and survival after sudden cardiac arrest: A systematic review and meta-analysis
Resuscitation
Trend analysis of disability-adjusted life years following adult out-of-hospital cardiac arrest in the United States: A study from the CARES Surveillance Group
Resuscitation
Advances in the hospital management of patients following an out of hospital cardiac arrest
Heart
Transportation during and after cardiac arrest: who, when, how and where?
Curr Opin Crit Care
Revising the Critical Care Family Needs Inventory for the Emergency Department
J Adv Nurs
Cited by (17)
Artificial intelligence in emergency medicine. A systematic literature review
2023, International Journal of Medical InformaticsArtificial intelligence to support out-of-hospital cardiac arrest care: A scoping review
2023, Resuscitation PlusTherapeutic hypothermia in patients after cardiac arrest: A systematic review and meta-analysis of randomized controlled trials
2023, American Journal of Emergency MedicineValidation of the CaRdiac Arrest Survival Score (CRASS) for predicting good neurological outcome after out-of-hospital cardiac arrest in an Asian emergency medical service system
2022, ResuscitationCitation Excerpt :The most important indicators of successful resuscitation are overall survival and survival with favorable neurological outcomes, therefore early and accurate prediction of such outcomes are key to efficient OHCA management.8–10. Numerous prognostic models have been developed to predict survival,8,9,11 neurologic recovery,12 and the return of spontaneous circulation (ROSC).13,14 These models served as clinical scoring tools to provide valuable support in decision-making regarding the proper allocation of resources, optimal treatment and communication with families.9