Continual Learning has been often framed as the problem of training a model in a sequence of tasks. In this regard, Neural Networks have been attested to forget the solutions to previous task as they learn new ones. Yet, modelling human life-long learning does not necessarily require any crisp notion of tasks. In this work, we propose a benchmark based on language modelling in a multilingual and multidomain setting that prescinds of any explicit delimitation of training examples into distinct tasks, and propose metrics to study continual learning and catastrophic forgetting in this setting. Then, we introduce a simple Product of Experts learning system that performs strongly on this problem while displaying interesting properties, and investigate its merits for avoiding forgetting.

中文翻译：

---

交替语言和域的类不可知的持续学习

持续学习通常被认为是在一系列任务中训练模型的问题。在这方面，已经证明神经网络在学习新任务时会忘记先前任务的解决方案。然而，模拟人类终身学习并不一定需要任何清晰的任务概念。在这项工作中，我们提出了一个基于多语言和多域环境中的语言建模的基准，该基准将训练示例的任何明确划分为不同的任务，并提出了在这种环境中研究持续学习和灾难性遗忘的指标。然后，我们介绍了一个简单的专家产品学习系统，该系统在显示有趣属性的同时在这个问题上表现出色，并研究了它在避免遗忘方面的优点。