The state-of-the-art reinforcement learning (RL) techniques have made innumerable advancements in robot control, especially in combination with deep neural networks (DNNs), known as deep reinforcement learning (DRL). In this article, instead of reviewing the theoretical studies on RL, which were almost fully completed several decades ago, we summarize some state-of-the-art techniques added to commonly used RL frameworks for robot control. We mainly review bioinspired robots (BIRs) because they can learn to locomote or produce natural behaviors similar to animals and humans. With the ultimate goal of practical applications in real world, we further narrow our review scope to techniques that could aid in sim-to-real transfer. We categorized these techniques into four groups: 1) use of accurate simulators; 2) use of kinematic and dynamic models; 3) use of hierarchical and distributed controllers; and 4) use of demonstrations. The purposes of these four groups of techniques are to supply general and accurate environments for RL training, improve sampling efficiency, divide and conquer complex motion tasks and redundant robot structures, and acquire natural skills. We found that, by synthetically using these techniques, it is possible to deploy RL on physical BIRs in actuality.

中文翻译：

---

用于仿生机器人强化学习的模拟到真实迁移技术的调查

最先进的强化学习 (RL) 技术在机器人控制方面取得了无数进步，特别是与深度神经网络 (DNN)（称为深度强化学习 (DRL)）相结合。在本文中，我们没有回顾几十年前几乎完全完成的强化学习理论研究，而是总结了一些添加到常用的机器人控制强化学习框架中的最先进技术。我们主要回顾仿生机器人（BIR），因为它们可以学习运动或产生类似于动物和人类的自然行为。为了实现现实世界中实际应用的最终目标，我们进一步将审查范围缩小到可以帮助模拟到现实的迁移的技术。我们将这些技术分为四组：1）使用精确的模拟器；2）运动学和动力学模型的使用；3）采用分层和分布式控制器；4) 使用演示。这四组技术的目的是为强化学习训练提供通用且准确的环境，提高采样效率，分治复杂的运动任务和冗余的机器人结构，并获得自然技能。我们发现，通过综合使用这些技术，实际上可以在物理 BIR 上部署 RL。