This paper proposes to use deep reinforcement learning for the simulation of physics-based musculoskeletal models of both healthy subjects and transfemoral prostheses’ users during normal level-ground walking. The deep reinforcement learning algorithm is based on the proximal policy optimization approach in combination with imitation learning to guarantee a natural walking gait while reducing the computational time of the training. Firstly, the optimization algorithm is implemented for the OpenSim model of a healthy subject and validated with experimental data from a public data-set. Afterwards, the optimization algorithm is implemented for the OpenSim model of a generic transfemoral prosthesis’ user, which has been obtained by reducing the number of muscles around the knee and ankle joints and, specifically, by keeping only the uniarticular ones. The model of the transfemoral prosthesis’ user shows a stable gait, with a forward dynamic comparable to the healthy subject’s, yet using higher muscles’ forces. Even though the computed muscles’ forces could not be directly used as control inputs for muscle-like linear actuators due to their pattern, this study paves the way for using deep reinforcement learning for the design of the control architecture of transfemoral prostheses.

中文翻译：

---

在正常行走过程中对健康受试者和经股假肢使用者进行基于物理的肌肉骨骼模拟的深度强化学习

本文建议使用深度强化学习对健康受试者和经股假肢使用者在正常水平地面行走过程中基于物理的肌肉骨骼模型进行仿真。深度强化学习算法基于近端策略优化方法并结合模仿学习，以确保自然的步态，同时减少训练的计算时间。首先，针对健康受试者的OpenSim模型实施优化算法，并使用来自公共数据集的实验数据对其进行验证。此后，针对通用经股假体使用者的OpenSim模型实施了优化算法，该算法是通过减少膝盖和踝关节周围的肌肉数量，特别是仅保留单关节的肌肉来获得的。经股假体使用者的模型显示出稳定的步态，其前进动力与健康受试者的相当，但使用了更高的肌肉力量。尽管由于其模式无法将计算出的肌肉力直接用作类似肌肉的线性致动器的控制输入，但这项研究为将深度强化学习用于经股假体的控制结构设计铺平了道路。