Symbolic execution is a powerful technique for bug finding and program testing. It is successful in finding bugs in real-world code. The core reasoning techniques use constraint solving, path exploration, and search, which are also the same techniques used in solving combinatorial problems, e.g., finite-domain constraint satisfaction problems (CSPs). We propose CSP instances as more challenging benchmarks to evaluate the effectiveness of the core techniques in symbolic execution. We transform CSP benchmarks into C programs suitable for testing the reasoning capabilities of symbolic execution tools. From a single CSP P, we transform P depending on transformation choice into different C programs. Preliminary testing with the KLEE, Tracer-X, and LLBMC tools show substantial runtime differences from transformation and solver choice. Our C benchmarks are effective in showing the limitations of existing symbolic execution tools. The motivation for this work is we believe that benchmarks of this form can spur the development and engineering of improved core reasoning in symbolic execution engines.

中文翻译：

---

使用约束问题对符号执行进行基准测试-初步结果

符号执行是用于错误查找和程序测试的强大技术。它可以成功地找到实际代码中的错误。核心推理技术使用约束求解，路径探索和搜索，这也是解决组合问题（例如，有限域约束满足问题（CSP））所使用的相同技术。我们建议使用CSP实例作为更具挑战性的基准，以评估符号执行中核心技术的有效性。我们将CSP基准转换为适合于测试符号执行工具的推理能力的C程序。从单个CSP P，我们根据转换选择将P转换为不同的C程序。使用KLEE，Tracer-X和LLBMC工具进行的初步测试显示，转换和求解器选择之间的运行时差异很大。我们的C基准可以有效地显示现有符号执行工具的局限性。进行这项工作的动机是我们认为，这种形式的基准可以刺激符号执行引擎中改进的核心推理的开发和工程。