Test oracles differentiate between the correct and incorrect system behavior. Automation of test oracles for visual output systems mainly involves image comparison, where a snapshot of the output is compared with respect to a reference image. Hereby, the captured snapshot can be subject to variations such as scaling and shifting. These variations lead to incorrect evaluations. Existing approaches employ computer vision techniques to address a specific set of variations. In this article, we introduce ADVISOR, an adjustable framework for test oracle automation of visual output systems. It allows the use of a flexible combination and configuration of computer vision techniques. We evaluated a set of valid configurations with a benchmark dataset collected during the tests of commercial digital TV systems. Some of these configurations achieved up to 3% better overall accuracy with respect to state-of-the-art tools. Further, we observed that there is no configuration that reaches the best accuracy for all types of image variations. We also empirically investigated the impact of significant parameters. One of them is a threshold regarding image matching score that determines the final verdict. This parameter is automatically tuned by offline training. We evaluated runtime performance as well. Results showed that differences among the ADVISOR configurations and state-of-the-art tools are in the order of seconds per image comparison.

中文翻译：

---

ADVISOR：用于测试 Oracle 视觉输出系统自动化的可调框架

测试预言机区分正确和不正确的系统行为。视觉输出系统测试预言机的自动化主要涉及图像比较，其中将输出的快照与参考图像进行比较。因此，捕获的快照可能会发生变化，例如缩放和移位。这些变化会导致错误的评估。现有方法采用计算机视觉技术来解决一组特定的变化。在本文中，我们介绍了 ADVISOR，一个用于测试可视化输出系统的预言机自动化的可调框架。它允许使用计算机视觉技术的灵活组合和配置。我们使用在商业数字电视系统测试期间收集的基准数据集评估了一组有效配置。与最先进的工具相比，其中一些配置的整体精度提高了 3%。此外，我们观察到对于所有类型的图像变化都没有达到最佳精度的配置。我们还根据经验研究了重要参数的影响。其中之一是关于决定最终判决的图像匹配分数的阈值。该参数由离线训练自动调整。我们还评估了运行时性能。结果表明，ADVISOR 配置和最先进的工具之间的差异在每个图像比较的数量级上。我们还根据经验研究了重要参数的影响。其中之一是关于决定最终判决的图像匹配分数的阈值。该参数由离线训练自动调整。我们还评估了运行时性能。结果表明，ADVISOR 配置和最先进的工具之间的差异在每个图像比较的数量级上。我们还根据经验研究了重要参数的影响。其中之一是关于决定最终判决的图像匹配分数的阈值。该参数由离线训练自动调整。我们还评估了运行时性能。结果表明，ADVISOR 配置和最先进的工具之间的差异在每个图像比较的数量级上。