Regression test case prioritization by code combinations coverage

Highlights

• We propose a new coverage criterion, code combinations coverage.

• We propose a new approach, code combinations coverage based prioritization (CCCP).

• We report on empirical studies to investigate the performance of CCCP.

• We provide some practical guidelines for testers when using CCCP.

Abstract

Regression test case prioritization (RTCP) aims to improve the rate of fault detection by executing more important test cases as early as possible. Various RTCP techniques have been proposed based on different coverage criteria. Among them, a majority of techniques leverage code coverage information to guide the prioritization process, with code units being considered individually, and in isolation. In this paper, we propose a new coverage criterion, code combinations coverage, that combines the concepts of code coverage and combination coverage. We apply this coverage criterion to RTCP, as a new prioritization technique, code combinations coverage based prioritization (CCCP). We report on empirical studies conducted to compare the testing effectiveness and efficiency of CCCP with four popular RTCP techniques: total, additional, adaptive random, and search-based test prioritization. The experimental results show that even when the lowest combination strength is assigned, overall, the CCCP fault detection rates are greater than those of the other four prioritization techniques. The CCCP prioritization costs are also found to be comparable to the additional test prioritization technique. Moreover, our results also show that when the combination strength is increased, CCCP provides higher fault detection rates than the state-of-the-art, regardless of the levels of code coverage.

Introduction

Modern software systems continuously evolve due to the fixing of detected bugs, the adding of new functionalities, and the refactoring of system architecture. Regression testing is conducted to ensure that the changed source code does not introduce new defects. However, it can become expensive to run an entire regression test suite because its size naturally increases during software maintenance and evolution: In an industrial case reported by Rothermel et al. (1999), for example, the execution time for running the entire test suite could become several weeks.

Regression test case prioritization (RTCP) has become one of the most effective approaches to reduce the overheads in regression testing (Li et al., 2007, Jiang et al., 2009, Mei et al., 2012, Saha et al., 2015, Ledru et al., 2009). RTCP techniques reorder the execution sequence of regression test cases, aiming to execute those test cases more likely to detect faults (according to some award function) as early as possible (Hao et al., 2016, Hao et al., 2014, Zhang et al., 2009).

Traditional RTCP techniques (Rothermel et al., 1999, Zhang et al., 2013a, Wang et al., 2017) usually use code coverage criteria to guide the prioritization process. Intuitively speaking, a code coverage criterion indicates the percentage of some code units (e.g. statements) covered by a test case. The expectation is that test cases with higher code coverage value have a greater chance of detecting faults (Zhu et al., 1997). Because of this, a goal of maximizing code coverage has been incorporated into various RTCP techniques, including greedy strategies (Rothermel et al., 1999). Given a coverage criterion (e.g., method, branch, or statement coverage), the total strategy selects the next test case with greatest absolute coverage, whereas the additional strategy selects the one with greatest coverage of code units not already covered by the prioritized test cases. Furthermore, Li et al. (2007) proposed two search-based RTCP techniques (a hill-climbing strategy and a genetic strategy) to explore the search space (the set of all permutations of the test cases) to find a sequence with a better fault detection rate. Jiang et al. (2009) investigated adaptive random techniques (Huang et al., 2019) to prioritize test cases using code coverage criteria. In an attempt to bridge the gap between the two greedy strategies, Zhang et al. (2013a) proposed a unified approach based on the fault detection probability for each test case (referred to as a p value).

In this paper, we propose a new coverage criterion, code combinations coverage, that combines the concepts of code coverage (Zhu et al., 1997) and combination coverage (Nie and Leung, 2011): Given a set of regression test cases $T$, each test case is first transferred to an equally-sized tuple. Each position in this tuple is a binary value representing whether the corresponding item (such as branch, statement, or method) is covered by this test case. In other words, $T$ is represented by a set of abstract test cases with binary values $T^{\prime }$. The code combinations coverage of $T$ is measured by the traditional combination coverage of $T^{\prime }$. We apply this new coverage criterion to RTCP, proposing a new prioritization technique: code combinations coverage based prioritization (CCCP).

We conducted empirical studies on 14 versions of four Java programs, and 30 versions of five real-world Unix utility programs. Our goal was to investigate the testing effectiveness and efficiency of CCCP compared with four widely-used RTCP techniques — total, additional, adaptive random, and search-based test prioritization. The results show that when the lowest combination strength is assigned, overall, our approach has better fault detection rates than the other four test prioritization techniques. It not only achieves comparable testing efficiency to additional, but also requires much less prioritization time than the adaptive random and search-based techniques. In addition, while the code coverage granularity does not impact on the testing effectiveness of CCCP, the test case granularity does significantly impact on it. Furthermore, when the combination strength is increased, CCCP provides better fault detection rates than all other RTCP techniques, regardless of the level of code coverage.

The main contributions of this paper are:

• We propose a new coverage criterion called code combinations coverage that combines the concepts of code coverage and combination coverage.

• We apply code combinations coverage to RTCP, leading to a new prioritization technique called code combinations coverage based prioritization (CCCP).

• We report on empirical studies conducted to investigate the test effectiveness and efficiency of CCCP compared to four widely-used prioritization techniques, and also analyze the impact of code coverage granularity and test case granularity on the effectiveness of CCCP.

• We provide some guidelines for how to choose the combination strength and code-coverage level for CCCP, under different testing scenarios.

The rest of this paper is organized as follows: Section 2 presents some background information. Section 3 introduces the proposed approach. Section 4 presents the research questions, and explains details of the empirical study. Section 5 provides the detailed results of the study and answers the research questions. Section 6 discusses some related work, and Section 7 concludes this paper, including highlighting some potential future work.