Deep Learning (DL) components are routinely integrated into software systems that need to perform complex tasks such as image or natural language processing. The adequacy of the test data used to test such systems can be assessed by their ability to expose artificially injected faults (mutations) that simulate real DL faults. In this paper, we describe an approach to automatically generate new test inputs that can be used to augment the existing test set so that its capability to detect DL mutations increases. Our tool DeepMetis implements a search based input generation strategy. To account for the non-determinism of the training and the mutation processes, our fitness function involves multiple instances of the DL model under test. Experimental results show that \tool is effective at augmenting the given test set, increasing its capability to detect mutants by 63% on average. A leave-one-out experiment shows that the augmented test set is capable of exposing unseen mutants, which simulate the occurrence of yet undetected faults.

中文翻译：

---

DeepMetis：增强深度学习测试集以提高其变异分数

深度学习 (DL) 组件通常集成到需要执行复杂任务（例如图像或自然语言处理）的软件系统中。用于测试此类系统的测试数据的充分性可以通过它们暴露模拟真实深度学习故障的人工注入故障（突变）的能力来评估。在本文中，我们描述了一种自动生成新测试输入的方法，该输入可用于扩充现有测试集，从而提高其检测 DL 突变的能力。我们的工具 DeepMetis 实现了基于搜索的输入生成策略。为了说明训练和变异过程的不确定性，我们的适应度函数涉及被测 DL 模型的多个实例。实验结果表明 \tool 在增加给定的测试集方面是有效的，将其检测突变体的能力平均提高了 63%。留一法实验表明，增强的测试集能够暴露看不见的突变体，模拟尚未检测到的故障的发生。