In recent years, ensemble methods have shown sterling performance and gained popularity in visual tasks. However, the performance of an ensemble is limited by the paucity of diversity among the models. Thus, to enrich the diversity of the ensemble, we present the distillation approach—learning from experts (LFEs). Such method involves a novel knowledge distillation (KD) method that we present, specific expert learning (SEL), which can reduce class selectivity and improve the performance on specific weaker classes and overall accuracy. Through SEL, models can acquire different knowledge from distinct networks with various areas of expertise, and a highly diverse ensemble can be obtained afterward. Our experimental results demonstrate that, on CIFAR-10, the accuracy of the ResNet-32 increases 0.91% with SEL, and that the ensemble trained by SEL increases accuracy by 1.13%. Compared to state-of-the-art approaches, for example, DML only improves accuracy by 0.3% and 1.02% on single ResNet-32 and the ensemble, respectively. Furthermore, our proposed architecture also can be applied to ensemble distillation (ED), which applies KD on the ensemble model. In conclusion, our experimental results show that our proposed SEL not only improves the accuracy of a single classifier but also boosts the diversity of the ensemble model.

中文翻译：

---

特定专家学习：通过知识蒸馏丰富集成多样性

近年来，集成方法表现出出色的性能，并在视觉任务中得到普及。然而，集成的性能受到模型之间缺乏多样性的限制。因此，为了丰富集成的多样性，我们提出了蒸馏方法——向专家学习 (LFEs)。这种方法涉及我们提出的一种新颖的知识蒸馏（KD）方法，即特定专家学习（SEL），它可以降低类选择性并提高特定较弱类的性能和整体准确性。通过 SEL，模型可以从具有不同专业领域的不同网络中获取不同的知识，之后可以获得高度多样化的集成。我们的实验结果表明，在 CIFAR-10 上，ResNet-32 的精度随着 SEL 提高了 0.91%，并且由 SEL 训练的集成将准确度提高了 1.13%。例如，与最先进的方法相比，DML 仅将单个 ResNet-32 和集成的精度分别提高 0.3% 和 1.02%。此外，我们提出的架构也可以应用于集成蒸馏（ED），它将 KD 应用于集成模型。总之，我们的实验结果表明，我们提出的 SEL 不仅提高了单个分类器的准确性，而且还提高了集成模型的多样性。