Software testing is often hindered where it is impossible or impractical to determine the correctness of the behaviour or output of the software under test (SUT), a situation known as the oracle problem. An example of an area facing the oracle problem is automatic image classification, using machine learning to classify an input image as one of a set of predefined classes. An approach to software testing that alleviates the oracle problem is metamorphic testing (MT). While traditional software testing examines the correctness of individual test cases, MT instead examines the relations amongst multiple executions of test cases and their outputs. These relations are called metamorphic relations (MRs): if an MR is found to be violated, then a fault must exist in the SUT. This paper examines the problem of classifying images containing visually hidden markers called Artcodes, and applies MT to verify and enhance the trained classifiers. This paper further examines two MRs, Separation and Occlusion, and reports on their capability in verifying the image classification using one-way analysis of variance (ANOVA) in conjunction with three other statistical analysis methods: t-test (for unequal variances), Kruskal–Wallis test, and Dunnett’s test. In addition to our previously-studied classifier, that used Random Forests, we introduce a new classifier that uses a support vector machine, and present its MR-augmented version. Experimental evaluations across a number of performance metrics show that the augmented classifiers can achieve better performance than non-augmented classifiers. This paper also analyses how the enhanced performance is obtained.

当无法或不切实际地确定被测软件 (SUT) 的行为或输出的正确性时，软件测试通常会受到阻碍，这种情况称为预言机问题. 面临 oracle 问题的一个例子是自动图像分类，使用机器学习将输入图像分类为一组预定义类中的一个。一种缓解预言机问题的软件测试方法是变形测试 (MT)。传统的软件测试检查单个测试用例的正确性，而 MT 则检查测试用例的多次执行及其输出之间的关系。这些关系称为变形关系 (MR)：如果发现违反了 MR，则 SUT 中必须存在故障。本文研究了包含视觉隐藏标记的图像分类问题，称为Artcodes，并应用 MT 来验证和增强训练好的分类器。本文进一步检查了两个 MR，分离和遮挡，并报告了它们使用单​​向方差分析 (ANOVA) 结合其他三种统计分析方法验证图像分类的能力：t 检验（用于不等方差）、Kruskal – Wallis 检验和 Dunnett 检验。除了我们之前研究的使用随机森林的分类器之外，我们还引入了一个使用支持向量机的新分类器，并展示了它的 MR 增强版本。多个性能指标的实验评估表明，增强分类器可以实现比非增强分类器更好的性能。本文还分析了如何获得增强的性能。