Demystifying machine learning for mortality prediction

With interest, we read the article by Banoei et al. [1] on machine learning (ML) models to predict mortality among COVID-19 patients. They refer to other studies that failed to predict mortality using ‘conventional statistical analysis’, after which they present a linear ML model as a better suited method for such complex medical problems. We feel such a claim creates an image around ML as an alternative technique that offers solutions where statistical modeling fails. However, ML and statistical modeling are tightly interwoven. Indeed, there is no consensus on whether or how to differentiate between the two [2, 3], e.g., the approach Banoei and colleagues present (Partial Least Squares) could easily be considered a ‘statistical analysis’ as well. A recent systematic review [4] differentiates ML from statistical models based on how ‘automatically’ these models learn and found no difference in discriminative performance. This raises the question why efforts are made to differentiate between statistical modeling and ML, as it does not provide insights into which prediction models work for which kind of problems. We advocate it is more important to demystify ML by emphasizing its connections to statistical models most clinicians are already familiar with, as it may help in setting reasonable expectations for the potential clinical benefit ML could bring. Towards demystifying ML, good reporting of methodology and findings is essential. Due to unclear or incomplete reporting, one may draw wrong conclusions and miss out on opportunities to learn. For instance, Banoei and colleagues are unclear on how frequently measured predictors (like SpO2) were aggregated to one value. We encourage to follow a unified approach for reporting, e.g. the TRIPOD guidelines [5]. Moreover, it is unfortunately still common that the intended use of prediction models is unclear, whereas it has implications for choices in the model development. Likewise, Banoei and colleagues present several models (for mortality prediction and patient clustering) and suggest that these can ‘aid in clinical decision making and resource allocation’, without further specification. We strongly encourage to develop models with an explicit intended use in mind, enabling fair judgement of their clinical relevance. Altogether, we join Banoei and colleagues in their belief that ML models hold a lot of promise as valuable tools in the modern ICU. However, we advise against differentiating them from statistical models and advocate proper reporting about the methodology and intended use.

Authors’ response

Machine learning-based methods vs. conventional statistical methods for studying mortality outcome

• Mohammad M. Banoei,
• Roshan Dinparastisaleh,
• Ali Vaeli Zadeh &
• Mehdi Mirsaeidi 

1. Department of Critical Care Medicine, University of Calgary, Alberta, Canada

   Mohammad M. Banoei

2. Department of Biological Science, University of Calgary, Alberta, Canada

   Mohammad M. Banoei

3. Division of Pulmonary and Critical Care Medicine, Johns Hopkins University, Baltimore, MD, 21218, USA

   Roshan Dinparastisaleh

4. Division of Pulmonary and Critical Care, Miami VA Medical Center, Miami, FL, USA

   Ali Vaeli Zadeh

5. Division of Pulmonary and Critical Care and Sleep Medicine, Department of Medicine, College of Medicine, University of Florida, Jacksonville, FL, USA

   Mehdi Mirsaeidi

In response to the Letter to the Editor by J.M. Smit et al., we certainly agree that machine learning (ML)-based methods can be used interchangeably with multivariate data analysis (MVA) and multivariate prediction models (MPMs) [6]. Although ML approaches like partial least square (PLS), statistically inspired modification of PLS (SIMPLS), random forest (RF), Support Vector Machine (SVM), and artificial neural network (ANN) are considered statistical methods, they are notably different from conventional statistical methods (CSM) [7]. Considering the advantages of ML methods, ML approaches have contributed significantly to the early detection, tracing, diagnosis, prognosis and clinical trials of COVID-19 that have been more functional to support researchers in confronting the coronavirus pandemic [8]. Several studies previously have shown that ML can be more appropriate than CSM for the clinical datasets. ML algorithms have proven better able to stratify COVID-19 patients and mortality risk [9], identify high-risk patients with COVID-19 [10].

It is generally believed that there is no single ML method superior to others. The SIMPLS method was the only prediction model used in our current study. SIMPLS has remarkable advantages including a lower risk of overfitting, a high level of interpretation, a high level of variable selection, easy implementation when compared with RF, SVM and ANN methods [6]. SIMPLS can easily account for batch processing and a high degree of correlation (multicollinearity) between and among variables of large datasets. SIMPLS fit the outcome responses with nominal, continuous, and polynomial data type, interaction and categorial effects, and provides strong visualization when compared with other ML and CSM. In our study, SIMPLS was successfully applied to recognize the most differentiating variables involved in the prediction of COVID-19 mortality. Other statistical methods that are not considered prediction models, such as principal component analysis (PCA) and latent class analysis (LCA) were established based on the findings of SIMPLS to identify patients at the highest risk of dying.

In our study, the importance of ML-based model was its ability to predict patient mortality using variables measured at the time of admission, although these variables were frequently measured during the patients’ hospitalization for other purposes.

Our study mainly focused on the importance of ML in clinical application, and the recognition of the most important variables contributing to mortality from COVID-19, instead of describing the details of ML that may be extraneous for clinicians.