Adversarial training deep neural networks often experience serious overfitting problem. Recently, it is explained that the overfitting happens because the sample complexity of training data is insufficient to generalize robustness. In traditional machine learning, one way to relieve overfitting from the lack of data is to use ensemble methods. However, adversarial training multiple networks is extremely expensive. Moreover, we found that there is a dilemma on choosing target model to generate adversarial examples. Optimizing attack to the members of ensemble will be suboptimal attack to the ensemble and incurs covariate shift, while attack to ensemble will weaken the members and lose the benefit from ensembling. In this paper, we propose adversarial training with Stochastic weight average (SWA); while performing adversarial training, we aggregate the temporal weight states in the trajectory of training. By adopting SWA, the benefit of ensemble can be gained without tremendous computational increment and without facing the dilemma. Moreover, we further improved SWA to be adequate to adversarial training. The empirical results on CIFAR-10, CIFAR-100 and SVHN show that our method can improve the robustness of models.

中文翻译：

---

具有随机权重的对抗训练

对抗性训练深度神经网络经常遇到严重的过拟合问题。最近，有人解释过拟合的发生是因为训练数据的样本复杂度不足以泛化鲁棒性。在传统的机器学习中，缓解因缺乏数据而过度拟合的一种方法是使用集成方法。然而，对抗性训练多个网络非常昂贵。此外，我们发现在选择目标模型来生成对抗性示例时存在两难选择。对集成成员的优化攻击将是对集成的次优攻击并导致协变量偏移，而对集成的攻击会削弱成员并失去集成的好处。在本文中，我们提出了随机权重平均（SWA）的对抗训练；在进行对抗训练时，我们聚合了训练轨迹中的时间权重状态。通过采用 SWA，可以在没有巨大的计算增量和不面临困境的情况下获得集成的好处。此外，我们进一步改进了 SWA 以适应对抗性训练。CIFAR-10、CIFAR-100 和 SVHN 的实证结果表明，我们的方法可以提高模型的鲁棒性。