Current deep learning architectures suffer from catastrophic forgetting, a failure to retain knowledge of previously learned classes when incrementally trained on new classes. The fundamental roadblock faced by deep learning methods is that deep learning models are optimized as "black boxes," making it difficult to properly adjust the model parameters to preserve knowledge about previously seen data. To overcome the problem of catastrophic forgetting, we propose utilizing an alternative "white box" architecture derived from the principle of rate reduction, where each layer of the network is explicitly computed without back propagation. Under this paradigm, we demonstrate that, given a pre-trained network and new data classes, our approach can provably construct a new network that emulates joint training with all past and new classes. Finally, our experiments show that our proposed learning algorithm observes significantly less decay in classification performance, outperforming state of the art methods on MNIST and CIFAR-10 by a large margin and justifying the use of "white box" algorithms for incremental learning even for sufficiently complex image data.

中文翻译：

---

通过降低速率进行增量学习

当前的深度学习体系结构遭受灾难性的遗忘，当在新课程上进行增量培训时，无法保留先前学习的课程的知识。深度学习方法面临的根本障碍是，深度学习模型被优化为“黑匣子”，这使得很难适当地调整模型参数以保留有关先前看到的数据的知识。为了克服灾难性遗忘的问题，我们建议使用一种从速率降低原理中衍生出来的“白盒”体系结构，该体系中网络的每一层都经过显式计算而没有反向传播。在这种范式下，我们证明，给定一个预先训练的网络和新的数据类别，我们的方法可以证明可以构建一个新的网络，该网络可以模拟所有过去和新班级的联合培训。最后，我们的实验表明，我们提出的学习算法在分类性能上观察到的衰减要小得多，在MNIST和CIFAR-10上的性能远远优于现有方法，并证明即使在足够的情况下也可以使用“白盒”算法进行增量学习复杂的图像数据。