We introduce Adaptive Procedural Task Generation (APT-Gen), an approach to progressively generate a sequence of tasks as curricula to facilitate reinforcement learning in hard-exploration problems. At the heart of our approach, a task generator learns to create tasks from a parameterized task space via a black-box procedural generation module. To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the similarity to the target tasks. Through adversarial training, the task similarity is adaptively estimated by a task discriminator defined on the agent's experiences, allowing the generated tasks to approximate target tasks of unknown parameterization or outside of the predefined task space. Our experiments on grid world and robotic manipulation task domains show that APT-Gen achieves substantially better performance than various existing baselines by generating suitable tasks of rich variations.

中文翻译：

---

困难探索问题的自适应程序任务生成

我们引入了自适应程序任务生成 (APT-Gen)，这是一种逐步生成一系列任务作为课程的方法，以促进困难探索问题中的强化学习。在我们方法的核心，任务生成器学习通过黑盒程序生成模块从参数化任务空间创建任务。为了在没有学习进度的直接指标的情况下实现课程学习，我们建议通过平衡代理在生成任务中的表现和与目标任务的相似性来训练任务生成器。通过对抗性训练，任务相似性由定义在代理经验上的任务鉴别器自适应地估计，允许生成的任务逼​​近未知参数化或预定义任务空间之外的目标任务。