Abstract

Objective

The development of machine learning (ML) algorithms to address a variety of issues faced in clinical practice has increased rapidly. However, questions have arisen regarding biases in their development that can affect their applicability in specific populations. We sought to evaluate whether studies developing ML models from electronic health record (EHR) data report sufficient demographic data on the study populations to demonstrate representativeness and reproducibility.

Materials and Methods

We searched PubMed for articles applying ML models to improve clinical decision-making using EHR data. We limited our search to papers published between 2015 and 2019.

Results

Across the 164 studies reviewed, demographic variables were inconsistently reported and/or included as model inputs. Race/ethnicity was not reported in 64%; gender and age were not reported in 24% and 21% of studies, respectively. Socioeconomic status of the population was not reported in 92% of studies. Studies that mentioned these variables often did not report if they were included as model inputs. Few models (12%) were validated using external populations. Few studies (17%) open-sourced their code. Populations in the ML studies include higher proportions of White and Black yet fewer Hispanic subjects compared to the general US population.

Discussion

The demographic characteristics of study populations are poorly reported in the ML literature based on EHR data. Demographic representativeness in training data and model transparency is necessary to ensure that ML models are deployed in an equitable and reproducible manner. Wider adoption of reporting guidelines is warranted to improve representativeness and reproducibility.

中文翻译：

---

使用电子健康记录报告人口统计数据和机器学习模型的代表性。

 抽象的

 客观的

用于解决临床实践中面临的各种问题的机器学习 (ML) 算法的发展迅速增长。然而，关于它们发展中的偏差的问题已经出现，这些偏差可能会影响它们在特定人群中的适用性。我们试图评估根据电子健康记录 (EHR) 数据开发机器学习模型的研究是否报告了有关研究人群的足够的人口统计数据，以证明代表性和可重复性。

 材料和方法

我们在 PubMed 中搜索了应用 ML 模型来利用 EHR 数据改进临床决策的文章。我们将搜索范围限制在 2015 年至 2019 年期间发表的论文。

 结果

在审查的 164 项研究中，人口统计变量的报告和/或作为模型输入的内容不一致。 64% 的人未报告种族/民族；分别有 24% 和 21% 的研究未报告性别和年龄。 92% 的研究未报告人口的社会经济状况。提到这些变量的研究通常不会报告它们是否被纳入模型输入。很少有模型 (12%) 使用外部人群进行验证。很少有研究 (17%) 开源他们的代码。与美国总人口相比，机器学习研究中的人群包括较高比例的白人和黑人，但西班牙裔受试者较少。

 讨论

基于 EHR 数据的 ML 文献中对研究人群的人口统计特征的报道很少。训练数据的人口代表性和模型透明度对于确保机器学习模型以公平和可重复的方式部署是必要的。有必要更广泛地采用报告指南，以提高代表性和可重复性。