In this paper, we propose a novel ensembling technique for deep neural networks, which is able to drastically reduce the required memory compared to alternative approaches. In particular, we propose to extract multiple sub-networks from a single, untrained neural network by solving an end-to-end optimization task combining differentiable scaling over the original architecture, with multiple regularization terms favouring the diversity of the ensemble. Since our proposal aims to detect and extract sub-structures, we call it Structured Ensemble. On a large experimental evaluation, we show that our method can achieve higher or comparable accuracy to competing methods while requiring significantly less storage. In addition, we evaluate our ensembles in terms of predictive calibration and uncertainty, showing they compare favourably with the state-of-the-art. Finally, we draw a link with the continual learning literature, and we propose a modification of our framework to handle continuous streams of tasks with a sub-linear memory cost. We compare with a number of alternative strategies to mitigate catastrophic forgetting, highlighting advantages in terms of average accuracy and memory.

在本文中，我们为深度神经网络提出了一种新颖的集成技术，与替代方法相比，它能够大大减少所需的内存。特别是，我们建议通过解决端到端优化任务，从单个未经训练的神经网络中提取多个子网络，该任务将原始架构的可微缩放与多个正则化项相结合，有利于集成的多样性。由于我们的提议旨在检测和提取子结构，因此我们称其为Structured Ensemble. 在大型实验评估中，我们表明我们的方法可以达到比竞争方法更高或相当的准确度，同时需要的存储空间要少得多。此外，我们在预测校准和不确定性方面评估了我们的集成，表明它们与最先进的技术相比具有优势。最后，我们与持续学习文献建立了联系，我们建议修改我们的框架，以处理具有亚线性内存成本的连续任务流。我们与许多减轻灾难性遗忘的替代策略进行了比较，突出了平均准确性和记忆力方面的优势。