Background

Reinforcement learning (RL) is a computational approach to understanding and automating goal-directed learning and decision-making. It is designed for problems which include a learning agent interacting with its environment to achieve a goal. For example, blood glucose (BG) control in diabetes mellitus (DM), where the learning agent and its environment are the controller and the body of the patient respectively. RL algorithms could be used to design a fully closed-loop controller, providing a truly personalized insulin dosage regimen based exclusively on the patient’s own data.

Objective

In this review we aim to evaluate state-of-the-art RL approaches to designing BG control algorithms in DM patients, reporting successfully implemented RL algorithms in closed-loop, insulin infusion, decision support and personalized feedback in the context of DM.

Methods

An exhaustive literature search was performed using different online databases, analyzing the literature from 1990 to 2019. In a first stage, a set of selection criteria were established in order to select the most relevant papers according to the title, keywords and abstract. Research questions were established and answered in a second stage, using the information extracted from the articles selected during the preliminary selection.

Results

The initial search using title, keywords, and abstracts resulted in a total of 404 articles. After removal of duplicates from the record, 347 articles remained. An independent analysis and screening of the records against our inclusion and exclusion criteria defined in Methods section resulted in removal of 296 articles, leaving 51 relevant articles. A full-text assessment was conducted on the remaining relevant articles, which resulted in 29 relevant articles that were critically analyzed. The inter-rater agreement was measured using Cohen Kappa test, and disagreements were resolved through discussion.

Conclusions

The advances in health technologies and mobile devices have facilitated the implementation of RL algorithms for optimal glycemic regulation in diabetes. However, there exists few articles in the literature focused on the application of these algorithms to the BG regulation problem. Moreover, such algorithms are designed for control tasks as BG adjustment and their use have increased recently in the diabetes research area, therefore we foresee RL algorithms will be used more frequently for BG control in the coming years. Furthermore, in the literature there is a lack of focus on aspects that influence BG level such as meal intakes and physical activity (PA), which should be included in the control problem. Finally, there exists a need to perform clinical validation of the algorithms.

中文翻译：

---

强化学习在糖尿病血糖控制中的应用：系统评价。

背景

强化学习 (RL) 是一种用于理解和自动化目标导向学习和决策的计算方法。它专为包括学习代理与其环境交互以实现目标的问题而设计。例如，糖尿病 (DM) 中的血糖 (BG) 控制，其中学习代理及其环境分别是患者的控制器和身体。RL 算法可用于设计一个完全闭环的控制器，提供完全基于患者自己数据的真正个性化的胰岛素给药方案。

目标

在这篇综述中，我们旨在评估在 DM 患者中设计 BG 控制算法的最先进 RL 方法，报告在闭环、胰岛素输注、决策支持和 DM 背景下的个性化反馈中成功实施的 RL 算法。

方法

使用不同的在线数据库进行了详尽的文献搜索，分析了 1990 年至 2019 年的文献。在第一阶段，建立了一套选择标准，以便根据标题、关键词和摘要选择最相关的论文。研究问题在第二阶段确定并回答，使用从初步选择期间选择的文章中提取的信息。

结果

使用标题、关键字和摘要进行初始搜索，结果总共有 404 篇文章。从记录中删除重复项后，保留了 347 篇文章。根据我们在方法部分定义的纳入和排除标准对记录进行独立分析和筛选，结果删除了 296 篇文章，留下 51 篇相关文章。对剩余的相关文章进行了全文评估，对 29 篇相关文章进行了批判性分析。评价者间的一致性采用 Cohen Kappa 检验，分歧通过讨论解决。

结论

健康技术和移动设备的进步促进了 RL 算法的实施，以实现糖尿病患者的最佳血糖调节。然而，文献中很少有文章专注于将这些算法应用于 BG 调节问题。此外，此类算法专为 BG 调整等控制任务而设计，最近它们在糖尿病研究领域的使用有所增加，因此我们预计 RL 算法将在未来几年更频繁地用于 BG 控制。此外，在文献中缺乏对影响 BG 水平的方面的关注，例如膳食摄入量和体力活动 (PA)，这应该包括在控制问题中。最后，需要对算法进行临床验证。