Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem -- both in terms of human and computational effort. This has hindered the adoption of mutation testing as an industry standard. For example, Google has a codebase of two billion lines of code and more than 500,000,000 tests are executed on a daily basis. The traditional approach to mutation testing does not scale to such an environment. To address these challenges, this paper presents a scalable approach to mutation testing based on the following main ideas: (1) Mutation testing is done incrementally, mutating only changed code during code review, rather than the entire code base; (2) Mutants are filtered, removing mutants that are likely to be irrelevant to developers, and limiting the number of mutants per line and per code review process; (3) Mutants are selected based on the historical performance of mutation operators, further eliminating irrelevant mutants and improving mutant quality. Evaluation in a code-review-based setting with more than 24,000 developers on more than 1,000 projects shows that the proposed approach produces orders of magnitude fewer mutants and that context-based mutant filtering and selection improve mutant quality and actionability. Overall, the proposed approach represents a mutation testing framework that seamlessly integrates into the software development workflow and is applicable up to large-scale industrial settings.

中文翻译：

---

大规模实用突变测试

变异分析通过测量测试套件检测小型人为故障的能力（系统地植入测试程序中）来评估其适当性。突变分析被认为是最强的测试充分性标准之一。变异测试建立在变异分析的基础之上，是一种将变异作为测试目标来创建或改进测试套件的测试技术。长期以来，突变测试一直被认为是棘手的，因为可以创建的大量突变代表了一个无法解决的问题-无论是从人力还是在计算方面。这阻碍了将突变测试作为行业标准的采用。例如，Google的代码库包含20亿行代码，每天执行的测试超过500,000,000次。传统的变异测试方法无法适应这样的环境。为了解决这些挑战，本文基于以下主要思想提出了一种可扩展的突变测试方法：（1）突变测试是增量进行的，仅在代码检查过程中对已更改的代码进行突变，而不对整个代码库进行突变；（2）过滤突变体，去除可能与开发者无关的突变体，并限制每行和每个代码审查过程的突变体数量；（3）根据突变算子的历史表现选择突变体，进一步消除不相关的突变体，提高突变体质量。在基于代码审查的环境中进行评估，超过24,000名开发人员参与了超过1个项目的开发工作，000个项目表明，所提出的方法产生的突变体数量减少了几个数量级，并且基于上下文的突变体过滤和选择提高了突变体的质量和可操作性。总体而言，所提出的方法代表了一种突变测试框架，该框架可以无缝集成到软件开发工作流程中，并且适用于大规模工业环境。