Elsevier

Resuscitation

Volume 170, January 2022, Pages 126-133
Resuscitation

Clinical paper
Development and validation of the SARICA score to predict survival after return of spontaneous circulation in out of hospital cardiac arrest using an interpretable machine learning framework

https://doi.org/10.1016/j.resuscitation.2021.11.029Get rights and content

Abstract

Background

Accurate and timely prognostication of patients with out-of-hospital cardiac arrest (OHCA) who achieved the return of spontaneous circulation (ROSC) is crucial in clinical decision-making, resource allocation, and communications with next-of-kins. We aimed to develop the Survival After ROSC in Cardiac Arrest (SARICA), a practical clinical decision tool to predict survival in OHCA patients who attained ROSC.

Methods

We utilized real-world Singapore data from the population-based Pan-Asian Resuscitation Outcomes Study between 2010–2018. We excluded patients without ROSC. The dataset was segmented into training (60%), validation (20%) and testing (20%) cohorts. The primary endpoint was survival (to 30-days or hospital discharge). AutoScore, an interpretable machine-learning based clinical score generation algorithm, was used to develop SARICA. Candidate factors were chosen based on objective demographic and clinical factors commonly available at the time of admission. Performance of SARICA was evaluated based on receiver-operating curve (ROC) analyses.

Results

5970 patients were included, of which 855 (14.3%) survived. A three-variable model was determined to be most parsimonious. Prehospital ROSC, age, and initial heart rhythm were identified for inclusion via random forest selection. Finally, SARICA consisted of these 3 variables and ranged from 0 to 10 points, achieving an area under the ROC (AUC) of 0.87 (95% confidence interval: 0.84–0.90) within the testing cohort.

Conclusion

We developed and internally validated the SARICA score to accurately predict survival of OHCA patients with ROSC at the time of admission. SARICA is clinically practical and developed using an interpretable machine-learning framework. SARICA has unknown generalizability pending external validation studies.

Introduction

Out-of-hospital cardiac arrest (OHCA) is a major healthcare issue worldwide, with high incidence rates and poor outcomes.1., 2. In the United States, the disability-adjusted life years (DALY) contributed by OHCA was ranked fifth, a testament to its huge health burden.3 Globally, the median estimated incidence for OHCA attended by the Emergency Medical Services (EMS) was 82.1 per 100,000 person-years,2 while the percentage survival to discharge was only 7.0%.2

Advanced hospital treatment modalities that can improve outcomes for post-OHCA patients are available.4 However, it is not feasible to offer these treatments to all OHCA patients as they can be resource-intensive, and unethical to provide to patients in whom they are futile.5 There is thus an unmet need for accurate prognostication tools to assist clinical decision-making and inform rational allocation of healthcare resources, such as intensive care unit (ICU) beds for OHCA patients at the Emergency Department (ED). Additionally, for families and caregivers, objective and accurate information on a patient’s outlook at an early stage can facilitate communication,6 by eliciting appropriate expectations, and where necessary, closure.

Although key predictors of OHCA survival have been identified,7 the prediction of OHCA survival in current practice remains reliant on clinical gestalt which can be widely variable, with no simple prediction tool available for routine use. A recent systematic review of existing prognostication tools by Gue et al. identified several existing OHCA prognostication risk scores with good predictive ability.8., 9., 10., 11. However, these scores are limited by difficulty in computation, predictors that might be subjected to recall bias or may not be static over time, including some that are not readily available at the time the patient is in the ED.8 These drawbacks make it challenging to recommend their routine use in clinical practice.

Recently, machine learning (ML) prediction models have been developed and deployed to prognosticate OHCA with high accuracy.12., 13. Compared to usual care methods and clinical prediction tools based on conventional statistical models, ML has been shown to have better discriminatory abilities in several different diagnostic and prognostic predictions.14 However, these ML models require extensive data points to function optimally and are technically difficult to deploy in existing hospital infrastructure.15 The “black box” approach of developing ML prognostication models is also a concern for clinicians and thus discourages them from fully trusting the score in actual practice16 - especially in high-stakes clinical decisions. To overcome the complexities and difficulties in interpretation with ML models, AutoScore,17 an easy-to-use, automatic, ML-based clinical score generator was created. Autoscore has been used to develop simple point-based clinical scores to predict short-term mortality risk at ED triage18 and inpatient acute kidney injury,19 with predictive ability that outperforms existing benchmark scores. Autoscore has not been tested on the OHCA population.

The aim of this study was to develop and internally validate Survival after Return of spontaneous circulation in Cardiac Arrest (SARICA], a simple point-based clinical prediction rule, using objective predictors typically available at ED admission to prognosticate 30-day survival of OHCA patients. AutoScore, which is an ML-based algorithm, was used to systematically develop SARICA through a transparent and interpretable framework.

Section snippets

Data source and study setting

Singapore is a densely-populated urban city-state with a population of 5.7 million and a population density of 7,810 per km2.20 A population-based survey in 2010 showed that 31.4% of responders had ever been trained in cardiopulmonary resuscitation (CPR), 10.7% had ever been trained in automated external defibrillation, while 9.6% and 3.7% possessed valid certificates for these skills, respectively.21 Emergency Medical Services (EMS) are provided by the Singapore Civil Defence Force, which

Baseline characteristics of study population

18,359 OHCA cases occurred between April 2010 and December 2018, of which 18,097 were attended by EMS. 2 cases were excluded due to missing data, and 12,125 cases were excluded due to ROSC not attained. Finally, 5970 cases qualified for analysis. The population flow diagram shown in Fig. 1 demonstrates the selection of study participants and subsequent segmentation into training, validation, and testing cohorts.

Baseline characteristics of all included OHCA cases were summarised in Table 1. 855

Discussion

In this study, using data from the large-scale multi-centre cardiac arrest registry in Singapore and the AutoScore framework, we developed and internally validated the SARICA score, a 10-point clinical score based on 3 variables readily available at ED admission, to predict OHCA patient survival post-ROSC. The predictive ability of our score performed reasonably well, with an AUC of 0.87 when evaluated on our testing cohort.

Conclusion

We developed and internally validated the SARICA score to accurately predict survival of OHCA patients with ROSC at the time of admission. SARICA is clinically practical and developed using an interpretable machine-learning framework. SARICA has unknown generalizability pending external validation studies.

Ethical approval and Consent to participate

Not applicable.

Consent for publication

All authors have approved the manuscript and agree with its submission to Resuscitation.

Availability of supporting data

Not applicable.

Funding

A.F.W.H was supported by the Estate of Tan Sri Khoo Teck Puat (Khoo Clinical Scholars Programme), Khoo Pilot Award (KP/2019/0034), Duke-NUS Medical School and National Medical Research Council (NMRC/CS_Seedfd/012/2018). This work was supported by the National Research Foundation of Singapore through the Virtual Singapore Program (Grant No. NRF2017VSG-AT3DCM001-031) and National Medical Research Council, Clinician Scientist Awards, Singapore (NMRC/CSA/024/2010 and NMRC/CSA/0049/2013).

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (32)

  • T. Iwami et al.

    Outcome and characteristics of out-of-hospital cardiac arrest according to location of arrest: A report from a large-scale, population-based study in Osaka, Japan

    Resuscitation

    (2006)
  • W. Bougouin et al.

    Gender and survival after sudden cardiac arrest: A systematic review and meta-analysis

    Resuscitation

    (2015)
  • R.A. Coute et al.

    Trend analysis of disability-adjusted life years following adult out-of-hospital cardiac arrest in the United States: A study from the CARES Surveillance Group

    Resuscitation

    (2020)
  • J.P. Nolan et al.

    Advances in the hospital management of patients following an out of hospital cardiac arrest

    Heart

    (2012)
  • A.F.W. Ho et al.

    Transportation during and after cardiac arrest: who, when, how and where?

    Curr Opin Crit Care

    (2021)
  • B. Redley et al.

    Revising the Critical Care Family Needs Inventory for the Emergency Department

    J Adv Nurs

    (2004)
  • Cited by (17)

    • Validation of the CaRdiac Arrest Survival Score (CRASS) for predicting good neurological outcome after out-of-hospital cardiac arrest in an Asian emergency medical service system

      2022, Resuscitation
      Citation Excerpt :

      The most important indicators of successful resuscitation are overall survival and survival with favorable neurological outcomes, therefore early and accurate prediction of such outcomes are key to efficient OHCA management.8–10. Numerous prognostic models have been developed to predict survival,8,9,11 neurologic recovery,12 and the return of spontaneous circulation (ROSC).13,14 These models served as clinical scoring tools to provide valuable support in decision-making regarding the proper allocation of resources, optimal treatment and communication with families.9

    View all citing articles on Scopus
    1

    Equal contribution.

    2

    Equal contribution.

    View full text