Context:

Test-driven development (TDD) is an agile software development approach that has been widely claimed to improve software quality. However, the extent to which TDD improves quality appears to be largely dependent upon the characteristics of the study in which it is evaluated (e.g., the research method, participant type, programming environment, etc.). The particularities of each study make the aggregation of results untenable.

Objectives:

The goal of this paper is to: increase the accuracy and generalizability of the results achieved in isolated experiments on TDD, provide joint conclusions on the performance of TDD across different industrial and academic settings, and assess the extent to which the characteristics of the experiments affect the quality-related performance of TDD.

Method:

We conduct a family of 12 experiments on TDD in academia and industry. We aggregate their results by means of meta-analysis. We perform exploratory analyses to identify variables impacting the quality-related performance of TDD.

Results:

TDD novices achieve a slightly higher code quality with iterative test-last development (i.e., ITL, the reverse approach of TDD) than with TDD. The task being developed largely determines quality. The programming environment, the order in which TDD and ITL are applied, or the learning effects from one development approach to another do not appear to affect quality. The quality-related performance of professionals using TDD drops more than for students. We hypothesize that this may be due to their being more resistant to change and potentially less motivated than students.

Conclusion:

Previous studies seem to provide conflicting results on TDD performance (i.e., positive vs. negative, respectively). We hypothesize that these conflicting results may be due to different study durations, experiment participants being unfamiliar with the TDD process, or case studies comparing the performance achieved by TDD vs. the control approach (e.g., the waterfall model), each applied to develop a different system. Further experiments with TDD experts are needed to validate these hypotheses.

中文翻译：

---

测试驱动开发的一系列实验

语境：

测试驱动开发（TDD）是一种敏捷的软件开发方法，已被广泛认为可以提高软件质量。但是，TDD改善质量的程度似乎很大程度上取决于对其进行评估的研究的特征（例如，研究方法，参与者类型，编程环境等）。每个研究的特殊性使结果的汇总变得站不住脚。

目标：

本文的目的是：提高在TDD上的独立实验中获得的结果的准确性和可概括性，提供有关TDD在不同工业和学术环境下的性能的联合结论，并评估实验特征对影响的程度TDD的质量相关性能。

方法：

我们在学术界和工业界对TDD进行了12个实验系列。我们通过荟萃分析汇总其结果。我们进行探索性分析，以确定影响TDD质量相关性能的变量。

结果：

与TDD相比，TDD新手通过迭代的最新测试开发（即ITL，TDD的反向方法）可以实现更高的代码质量。正在开发的任务在很大程度上决定了质量。编程环境，TDD和ITL的应用顺序或从一种开发方法到另一种开发方法的学习效果似乎都不会影响质量。使用TDD的专业人员在质量方面的表现下降的幅度要比学生下降的幅度更大。我们假设这可能是由于他们比学生更能抵抗变化，并且动机较弱。

结论：

先前的研究似乎在TDD性能上提供了矛盾的结果（即分别为阳性与阴性）。我们假设这些矛盾的结果可能是由于不同的研究持续时间，实验参与者不熟悉TDD流程或将TDD与对照方法（例如瀑布模型）所取得的性能进行比较的案例研究，每个案例都适用于开发不同的系统。需要与TDD专家进行进一步的实验，以验证这些假设。