As incorporating software testing into programming assignments becomes routine, educators have begun to assess not only the correctness of students’ software, but also the adequacy of their tests. In practice, educators rely on code coverage measures, though its shortcomings are widely known. Mutation analysis is a stronger measure of test adequacy, but it is too costly to be applied beyond the small programs developed in introductory programming courses. We demonstrate how to adapt mutation analysis to provide rapid automated feedback on software tests for complex projects in large programming courses. We study a dataset of 1389 student software projects ranging from trivial to complex. We begin by showing that although the state-of-the-art in mutation analysis is practical for providing rapid feedback on projects in introductory courses, it is prohibitively expensive for the more complex projects in subsequent courses. To reduce this cost, we use a statistical procedure to select a subset of mutation operators that maintains accuracy while minimizing cost. We show that with only 2 operators, costs can be reduced by a factor of 2–3 with negligible loss in accuracy. Finally, we evaluate our approach on open-source software and report that our findings may generalize beyond our educational context.

随着将软件测试合并到编程作业中成为日常工作，教育者不仅开始评估学生软件的正确性，而且还评估他们的测试的充分性。在实践中，尽管其缺点广为人知，但教育工作者仍依赖代码覆盖率度量。变异分析是检验测试充分性的更有效方法，但成本太高，无法应用于入门编程课程中开发的小程序之外。我们演示了如何使用突变分析为大型编程课程中的复杂项目提供有关软件测试的快速自动化反馈。我们研究了从简单到复杂的1389个学生软件项目的数据集。我们首先显示，尽管最新的突变分析技术可以为入门课程中的项目提供快速反馈，对于后续课程中更复杂的项目而言，这是非常昂贵的。为了降低此成本，我们使用统计程序来选择变异算子的子集，该子集在保持准确性的同时将成本降至最低。我们证明，只有2个运营商，成本可以降低2-3倍，而精度损失可忽略不计。最后，我们评估我们在开源软件上的方法，并报告说我们的发现可能会超出我们的教育背景。