Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.

强化学习（RL）的递归神经网络（RNN）已显示出明显的优势，例如，解决了依赖于记忆的任务和元学习。但是，在改进RNN架构和理解潜在的神经机制以提高性能方面所做的工作很少。在本文中，我们为RL提出了一种新颖的多时间尺度随机RNN。实验结果表明，该网络可以自主学习抽象目标，并可以在具有挑战性的连续控制任务中使用内部动力学自行开发动作层次。此外，我们表明，与适应新任务相比，网络的自我发展的组合能力比从头开始时更快，可以更快地重新学习。