Evaluating the agreement among technical debt measurement tools: building an empirical benchmark of technical debt liabilities

Abstract

Software teams are often asked to deliver new features within strict deadlines leading developers to deliberately or inadvertently serve “not quite right code” compromising software quality and maintainability. This non-ideal state of software is efficiently captured by the Technical Debt (TD) metaphor, which reflects the additional effort that has to be spent to maintain software. Although several tools are available for assessing TD, each tool essentially checks software against a particular ruleset. The use of different rulesets can often be beneficial as it leads to the identification of a wider set of problems; however, for the common usage scenario where developers or researchers rely on a single tool, diverse estimates of TD and the identification of different mitigation actions limits the credibility and applicability of the findings. The objective of this study is two-fold: First, we evaluate the degree of agreement among leading TD assessment tools. Second, we propose a framework to capture the diversity of the examined tools with the aim of identifying few “reference assessments” (or class/file profiles) representing characteristic cases of classes/files with respect to their level of TD. By extracting sets of classes/files exhibiting similarity to a selected profile (e.g., that of high TD levels in all employed tools) we establish a basis that can be used either for prioritization of maintenance activities or for training more sophisticated TD identification techniques. The proposed framework is illustrated through a case study on fifty (50) open source projects and two programming languages (Java and JavaScript) employing three leading TD tools.

Appendix

In this Appendix we report a sensitivity analysis conducted to examine the variability of the classes belonging to the Max-Ruler archetype in terms of their a-coefficients. After finding the archetypes and expressing the original data points with their a-coefficients, the new transformed data are of special type. Specifically, the new variables are non-negative and their sum is fixed, equal to 1. These data are called compositional or proportional data and the new variables have an inherent correlation which raises new problems. Indeed, classical statistical methodologies should not be applied (Aitchison, 1982), since their principles are violated. Actually, there is a whole class of theories, methods and tools for analyzing such data, belonging to a special branch of statistics, the Compositional Data Analysis (CoDA) introduced by Aitchison (Aitchison, 1982). These methods have been used in the context of software engineering (Chatzipetrou et al., 2015; 2010).

Describing briefly, the sample space of the compositional data (a-coefficients in our case) is a simplex of the form ${\sum}_{j=1}^k{a}_{ij}=1$ with a_ij ≥ 0 and i = 1, …, n, as it has been already defined in Eq. (4) in the manuscript. In our study, we adopted some certain suggested CoDA methodologies and specifically the centered log-ratio transformation (clr) and an imputation method for zeros which cause problems in the analysis, known as the multiplicative replacement strategy (Martín-Fernández et al., 2003).

After this preprocessing, we evaluated for each Max-Ruler archetype and their corresponding classes, a global measure of spread proposed by Pawlowsky-Glahn and Egozcue (Pawlowsky-Glahn and Egozcue, 2001), namely the metric standard deviation (MSD) that can be computed via the following formula

$$ MSD=\sqrt{\frac{1}{k-1} MVAR(X)} $$

(6)

where $MVAR(X)=\frac{1}{m-1}{\sum}_{i=1}^m{d}^2\left({\mathbf{x}}_m,\overline{\mathbf{x}}\right)$ represents the metric variance and m represents the number of classes for the Max-Ruler archetype.

Appendix Table 8 summarizes the MSD values for the set of the experiments conducted through sensitivity analysis following a similar approach to the analysis of percentages of high-TD classes presented above. The general intuition from the inspection of the estimated MSD values is that the spread of the classes increases as the value of the threshold a increases only for JavaScript projects, whereas the spread seems to be generally the same for Java projects. Indeed, the findings of the two LME models (beyond model vs. the model without the interaction term) indicated a statistically significant difference χ² = 39.786, p < 0.001, and thus, the interaction term should be retained in the final model. This practically means that the spread depends on the combination of the levels of the two examined factors (Threshold and Language). To this regard, the post-hoc analysis through Tukey’s HSD test did not reveal a statistically significant difference for any pair-wise comparison conducted on the levels of factor Threshold for Java projects. In contrast, there were noted statistically significant differences for specific levels of factor Threshold for JavaScript projects forming four overlapping homogenous groups that are A = {0.60, 0.65, 0.70}, B = {0.65, 0.70, 0.75}, C = {0.70, 0.75, 0.80} and D = {0.80, 0.85, 0.90}.

Table 8 Estimated mean MSD with 95% CI for each threshold value a (sensitivity analysis)