Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system. In this work, we study these challenges in the context of sequential supervised learning with an emphasis on recurrent neural networks. To evaluate the models in the lifelong learning setting, we propose a curriculum-based, simple, and intuitive benchmark where the models are trained on tasks with increasing levels of difficulty. To measure the impact of catastrophic forgetting, the model is tested on all the previous tasks as it completes any task. As a step toward developing true lifelong learning systems, we unify gradient episodic memory (a catastrophic forgetting alleviation approach) and Net2Net (a capacity expansion approach). Both models are proposed in the context of feedforward networks, and we evaluate the feasibility of using them for recurrent networks. Evaluation on the proposed benchmark shows that the unified model is more suitable than the constituent models for lifelong learning setting.

中文翻译：

---

为终身学习训练循环神经网络

灾难性遗忘和容量饱和是任何参数终身学习系统的核心挑战。在这项工作中，我们在顺序监督学习的背景下研究这些挑战，重点是循环神经网络。为了评估终身学习环境中的模型，我们提出了一个基于课程的、简单且直观的基准，其中模型在难度不断增加的任务上进行训练。为了衡量灾难性遗忘的影响，该模型在完成任何任务时会在所有先前的任务上进行测试。作为开发真正终身学习系统的一步，我们统一了梯度情景记忆（一种灾难性遗忘缓解方法）和 Net2Net（一种容量扩展方法）。这两种模型都是在前馈网络的背景下提出的，我们评估了将它们用于循环网络的可行性。对拟议基准的评估表明，统一模型比组成模型更适合终身学习设置。