We present an empirical comparison of three test generation techniques, namely, Combinatorial Testing (CT), Random Testing (RT) and Adaptive Random Testing (ART), under different test scenarios. This is the first study in the literature to account for the (more realistic) testing setting in which the tester may not have complete information about the parameters and constraints that pertain to the system, and to account for the challenge posed by faults (in terms of failure rate). Our study was conducted on nine real-world programs under a total of 1683 test scenarios (combinations of available parameter and constraint information and failure rate). The results show significant differences in the techniques’ fault detection ability when faults are hard to detect (failure rates are relatively low). CT performs best overall; no worse than any other in 98 percent of scenarios studied. ART enhances RT, and is comparable to CT in 96 percent of scenarios, but its computational cost can be up to 3.5 times higher than CT when the program is highly constrained. Additionally, when constraint information is unavailable for a highly-constrained program, a large random test suite is as effective as CT or ART, yet its computational cost of test generation is significantly lower than that of other techniques.

中文翻译：

---

组合测试、随机测试和自适应随机测试的实证比较

我们在不同的测试场景下对三种测试生成技术进行了实证比较，即组合测试 (CT)、随机测试 (RT) 和自适应随机测试 (ART)。这是文献中第一项考虑（更现实的）测试设置的研究，其中测试人员可能没有关于与系统相关的参数和约束的完整信息，并说明故障带来的挑战（就故障率）。我们的研究是在总共 1683 个测试场景（可用参数和约束信息以及故障率的组合）下对九个真实世界的程序进行的。结果表明，当故障难以检测（故障率相对较低）时，这些技术的故障检测能力存在显着差异。CT 整体表现最佳；在所研究的 98% 的情景中，并不比其他任何情况差。ART 增强了 RT，在 96% 的场景中与 CT 相当，但在程序高度受限的情况下，其计算成本可能比 CT 高出 3.5 倍。此外，当高度约束的程序无法获得约束信息时，大型随机测试套件与 CT 或 ART 一样有效，但其测试生成的计算成本明显低于其他技术。