Previous research evaluating deep learning (DL) classifiers has often used top-1/top-5 accuracy. However, the accuracy of DL classifiers is unstable in that it often changes significantly when retested on imperfect or adversarial images. This paper adds to the small but fundamental body of work on benchmarking the robustness of DL classifiers on imperfect images by proposing a two-dimensional metric, consisting of mean accuracy and coefficient of variation, to measure the robustness of DL classifiers. Spearman's rank correlation coefficient and Pearson's correlation coefficient are used and their independence evaluated. A statistical plot we call mCV is presented which aims to help visualize the robustness of the performance of DL classifiers across varying amounts of imperfection in tested images. Finally, we demonstrate that defective images corrupted by two-factor corruption could be used to improve the robustness of DL classifiers. All source codes and related image sets are shared on a website (http://www.animpala.com) to support future research projects.

中文翻译：

---

对深度学习分类器进行基准测试：超越准确性

先前评估深度学习（DL）分类器的研究经常使用top-1 / top-5准确性。但是，DL分类器的准确性不稳定，因为在对不完整或对抗性图像进行重新测试时，它经常会发生重大变化。本文通过提出一个由均值精度和变异系数组成的二维度量来测量DL分类器的鲁棒性，为在不完美图像上对DL分类器的鲁棒性进行基准测试的工作量很小但基础很广。使用Spearman秩相关系数和Pearson相关系数并评估其独立性。提出了一个称为mCV的统计图，该图旨在帮助可视化DL分类器在测试图像中不同缺陷量上的性能鲁棒性。最后，我们证明了由两因素破坏破坏的缺陷图像可用于提高DL分类器的鲁棒性。所有源代码和相关图像集都在网站（http://www.animpala.com）上共享，以支持未来的研究项目。