Abstract

Data used in deep learning is notoriously problematic. For example, data are usually combined from diverse sources, rarely cleaned and vetted thoroughly, and sometimes corrupted on purpose. Intentional corruption that targets the weak spots of algorithms has been studied extensively under the label of “adversarial attacks.” In contrast, the arguably much more common case of corruption that reflects the limited quality of data has been studied much less. Such “random” corruptions are due to measurement errors, unreliable sources, convenience sampling, and so forth. These kinds of corruption are common in deep learning, because data are rarely collected according to strict protocols—in strong contrast to the formalized data collection in some parts of classical statistics. This article concerns such corruption. We introduce an approach motivated by very recent insights into median-of-means and Le Cam’s principle, we show that the approach can be readily implemented, and we demonstrate that it performs very well in practice. In conclusion, we believe that our approach is a very promising alternative to standard parameter training based on least-squares and cross-entropy loss.

摘要

众所周知，深度学习中使用的数据存在问题。例如，数据通常来自不同来源，很少经过彻底清理和审查，有时还会被故意破坏。针对算法弱点的故意破坏已在“对抗性攻击”的标签下得到广泛研究。相比之下，反映数据质量有限的腐败案例可以说更为常见，但研究却少得多。这种“随机”损坏是由于测量误差、不可靠的来源、便利抽样等原因造成的。这些类型的损坏在深度学习中很常见，因为数据很少根据严格的协议收集——这与经典统计某些部分的形式化数据收集形成鲜明对比。这篇文章关注的是这种腐败。我们介绍了一种方法，该方法的动机是最近对均值中值和 Le Cam 原理的见解，我们证明了该方法可以很容易地实施，并且我们证明它在实践中表现非常好。总之，我们相信我们的方法是基于最小二乘和交叉熵损失的标准参数训练的一种非常有前途的替代方法。