Skip to main content
SpringerLink
Log in

Automating system test case classification and prioritization for use case-driven testing in product lines

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Product Line Engineering (PLE) is a crucial practice in many software development environments where software systems are complex and developed for multiple customers with varying needs. At the same time, many development processes are use case-driven and this strongly influences their requirements engineering and system testing practices. In this paper, we propose, apply, and assess an automated system test case classification and prioritization approach specifically targeting system testing in the context of use case-driven development of product families. Our approach provides: (i) automated support to classify, for a new product in a product family, relevant and valid system test cases associated with previous products, and (ii) automated prioritization of system test cases using multiple risk factors such as fault-proneness of requirements and requirements volatility in a product family. Our evaluation was performed in the context of an industrial product family in the automotive domain. Results provide empirical evidence that we propose a practical and beneficial way to classify and prioritize system test cases for industrial product lines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. For accessing the tool, see: https://sntsvv.github.io/PUMconf/.

  2. In PL use case diagrams, use cases are connected to variation points with an include dependency.

References

  • Al-Hajjaji M, Thum T, Meinicke J, Lochau M, Saake G (2014) Similarity-based prioritization in software product-line testing. In: SPLC’14, pp 197–206

  • Al-Hajjaji M, Thum T, Lochau M, Meinicke J, Saake G (2016) Effective product-line testing using similarity-based product prioritization. Softw Syst Model

  • Al-Hajjaji M, Kruger J, Schulze S, Leich T, Saake G (2017a) Efficient product-line testing using cluster-based product prioritization. In: AST’17, pp 16–22

  • Al-Hajjaji M, Lity S, Lachmann R, Thum T, Schaefer I, Saake G (2017b) Delta-oriented product prioritization for similarity-based product-line testing. In: VACE’17, pp 34–40

  • Arafeen MJ, Do H (2013) Test case prioritization using requirements-based clustering. In: ICST’13, pp 312–321

  • Arrieta A, Wang S, Sagardui G, Etxeberria L (2016) Test case prioritization of configurable cyber-physical systems with weight-based search algorithms. In: GECCO’16, pp 1053–1060

  • Arrieta A, Sagardui G, Etxeberria L, Zander J (2017) Automatic generation of test system instances for configurable cyber-physical sytems. Softw Qual J 25(3):1041–1083

    Google Scholar 

  • Arrieta A, Wang S, Sagardui G, Etxeberria L (2019) Search-based test case prioritization for simulation-based testing of cyber-physical system product lines. J Syst Softw 149:1–34

    Google Scholar 

  • Baller H, Lity S, Lochau M, Schaefer I (2014) Multi-objective test suite optimization for incremental product family testing. In: ICST’14, pp 303–312

  • Bertolino A, Gnesi S (2003) PLUTO: a test methodology for product families. In: PFE’03, pp 181–197

  • Binkley D (1997) Semantics guided regression test cost reduction. IEEE Trans Softw Eng 23(8):498–516

    Google Scholar 

  • Briand LC, Labiche Y, He S (2009) Automating regression test selection based on uml designs. Inf Softw Technol 51:16–30

    Google Scholar 

  • Bühne S, Halmans G, Pohl K (2003) Modeling dependencies between variation points in use case diagrams. In: REFSQ’03, pp 59–69

  • Cabral I, Cohen MB, Rothermel G (2010) Improving the testing and testability of software product lines. In: SPLC’10, pp 241–255

  • Chen Y, Probert RL, Sims DP (2002) Specification-based regression test selection with risk analysis. In: CASCON’02

  • Clarke D, Helvensteijn M, Schaefer I (2010) Abstract delta modeling. In: GPCE’10, pp 13–22

  • Coleman D, Holland P, Kaden N, Klema V, Peters SC (1980) A system of subroutines for iteratively reweighted least squares computations. ACM Trans Math Softw 6(3):327–336

    MATH  Google Scholar 

  • Da Mota Silveira Neto PA, do Carmo Machado I, McGregor JD, de Almeida ES, de Lemos Meira SR (2011) A systematic mapping study of software product line testing. Inf Softw Technol 53:407–423

    Google Scholar 

  • Devroey X, Perrouin G, Cordy M, Schobbens PY, Legay A, Heymans P (2014) Towards statistical prioritization for software product lines testing. In: VaMoS’14, pp 1–7

  • Devroey X, Perrouin G, Cordy M, Samih H, Legay A, Schobbens PY, Heymans P (2017) Statistical prioritization for software product line testing: an experience report. Softw Syst Model 16(1):153–171

    Google Scholar 

  • Do H (2016) Recent advances in regression testing techniques. In: Advances in computers, vol 103

  • Do H, Rothermel G (2006) On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans Softw Eng 32(9):733–752

    Google Scholar 

  • do Carmo Machado I, Mcgregor JD, Cavalcanti YC, De Almeida ES (2014) On strategies for testing software product lines: a systematic literature review. Inf Softw Technol 56(10):1183–1199

    Google Scholar 

  • Dukaczewski M, Schaefer I, Lachmann R, Lochau M (2013) Requirements-based delta-oriented spl testing. In: PLEASE’13, pp 49–52

  • Eclipse EMF (2018) https://eclipse.org/modeling/emf/

  • IEE (2018) (International Electronics & Engineering) S.A. http://www.iee.lu/

  • Engström E (2013) Supporting decisions on regression test scoping in a software product line context—from evidence to practice. Ph.D. thesis, Lund University

  • Engström E, Runeson P (2011) Software product line testing—a systematic mapping study. Inf Softw Technol 53:2–13

    Google Scholar 

  • Engström E, Runeson P, Skoglund M (2010) A systematic review on regression test selection techniques. Inf Softw Technol 52(1):14–30

    Google Scholar 

  • Engström E, Runeson P, Ljung A (2011) Improving regression testing transparency and efficiency with history-based prioritization—an industrial case study. In: ICST’11, pp 367–376

  • Ensan A, Bagheri E, Asadi M, Gasevic D, Biletskiy Y (2011) Goal-oriented test case selection and prioritization for product line feature models. In: ITNG’11, pp 291–298

  • Geppert B, Li J, Weiss DM (2004) Towards generating acceptance tests for product lines. Springer, Berlin, pp 35–48

    Google Scholar 

  • Gonzales-Sanchez A, Piel E, Abreu R, Gross HG, van Gemund AJ (2011) Prioritizing tests for software fault diagnosis. Softw Pract Exp 41 (10):1105–1129

    Google Scholar 

  • Haidry S, Miller T (2013) Using dependency structures for prioritization of functional test suites. IEEE Trans Softw Eng 39(2):258–275

    Google Scholar 

  • Hajri I, Goknil A, Briand LC, Stephany T (2015) Applying product line use case modeling in an industrial automotive embedded system: lessons learned and a refined approach. In: 338–347

  • Hajri I, Goknil A, Briand LC, Stephany T (2016) PUMConf: a tool to configure product specific use case and domain models in a product line. In: SIGSOFT FSE’16, pp 1008–1012

  • Hajri I, Goknil A, Briand LC, Stephany T (2017a) Incremental reconfiguration of product specific use case models for evolving configuration decisions. In: REFSQ’17, pp 3–21

  • Hajri I, Goknil A, Briand LC (2017b) A change management approach in product lines for use case-driven development and testing. In: Poster Session—REFSQ’17

  • Hajri I, Goknil A, Briand LC, Stephany T (2018a) Change impact analysis for evolving configuration decisions in product line use case models. J Syst Softw 139:211–237

  • Hajri I, Goknil A, Briand LC, Stephany T (2018b) Configuring use case models in product families. Softw Syst Model 17(3):939–971

  • Halmans G, Pohl K (2003) Communicating the variability of a software-product family to customers. Softw Syst Model 22(1):15–36

    Google Scholar 

  • Harrold MJ, Jones JA, Li T, Liang D (2001) Regression test selection for java software. In: OOPSLA’01

  • Hemmati H, Briand L, Arcuri A, Ali S (2010) An enhanced test case selection approach for model-based testing: an industrial case study. In: FSE’10, pp 267–276

  • Hemmati H, Fang Z, Mantyla MV, Adams B (2017) Prioritizing manual test cases in rapid release environments. Softw Test Verif Reliab 27(6):e1609

    Google Scholar 

  • Henard C, Papadakis M, Perrouin G, Klein J, Heymans P, Traon YL (2014) Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans Softw Eng 40(7):650–670

    Google Scholar 

  • Henry J, Henry S (1993) Quantitative assessment of the software maintenance process and requirements volatility. In: CSC’93, pp 346–351

  • Hosmer DW Jr, Lemeshow S, Sturdivant RX (2013) Applied logistic regression. Wiley, New Jersey

    MATH  Google Scholar 

  • ISO (2018) ISO-26262: road vehicles—functional safety

  • Johansen MF, Haugen Ø, Fleurey F (2011) A survey of empirics of strategies for software product line testing. In: ICSTW’11. IEEE, pp 266–269

  • Kamsties E, Pohl K, Reis S, Reuys A (2004) Testing variabilities in use case models. In: PFE’03, pp 6–18

  • Khatibsyarbini M, Isa MA, Jawawi DN, Tumeng R (2018) Test case prioritization approaches in regression testing: A systematic literature review. Inf Softw Technol 93:74–93

    Google Scholar 

  • Knapp A, Roggenbach M, Schlingloff BH (2014) On the use of test cases in model-based software product line development. In: SPLC’14, pp 247–251

  • Korel B, Koutsogiannakis G, Tahat LH (2008) Application of system models in regression test suite prioritization. In: ICSM’08, pp 247–256

  • Krishnamoorthi R, Mary SSA (2009) Factor oriented requirement coverage based system test case prioritization of new and regression test cases. Inf Softw Technol 51:799–808

    Google Scholar 

  • Kundu D, Sarma M, Sarma D, Mall R (2009) System testing for object-oriented systems with test case prioritization. Softw Test Verif Reliab 19(4):297–333

    Google Scholar 

  • Kung DC, Gao J, Hsia P (1995) Class firewall, test order, and regression testing of object-oriented programs. J Object-Oriented Program 8(2):51–65

    Google Scholar 

  • Lachmann R, Lity S, Lischke S, Beddig S, Schulze S, Schaefer I (2015) Delta-oriented test case prioritization for integration testing of software product lines. In: SPLC’15, pp 81–90

  • Lachmann R, Lity S, Al-Hajjaji M, Furchtegott F, Schaefer I (2016a) Fine-grained test case prioritization for integration testing of delta-oriented software product lines. In: FOSD’16

  • Lachmann R, Nieke M, Seidl C, Schaefer I, Schulze S (2016b) System-level test case prioritization using machine learning. In: ICMLA’16, pp 361–368

  • Lachmann R, Beddig S, Lity S, Schulze S, Schaefer I (2017) Risk-based integration testing of software product lines. In: VaMoS’17, pp 52–59

  • Larman C (2002) Applying UML and patterns: an introduction to object-oriented analysis and design and the unified process. Prentice Hall Professional

  • Lee J, Kang S, Lee D (2012) A survey on software product line testing. In: SPLC’12, pp 31–40

  • Li Z, Harman M, Hierons RM (2007) Search algorithms for regression test prioritization. IEEE Trans Softw Eng 33(4):225–237

    Google Scholar 

  • Lity S, Lochau M, Schaefer I, Goltz U (2012) Delta-oriented model-based spl regression testing. In: PLEASE’12, pp 53–56

  • Lity S, Morbach T, Thum T, Schaefer I (2016) Applying incremental model slicing to product-line regression testing. In: ICSR’16, pp 3–19

  • Lity S, Al-Hajjaji M, Thum T, Schaefer I (2017) Optimizing product orders using graph algorithms for improving incremental product-line analysis. In: VaMoS’17, pp 60–67

  • Lochau M, Lity S, Lachmann R, Schaefer I, Goltz U (2014) Delta-oriented model-based integration testing of large-scale systems. J Syst Softw 91:63–84

    Google Scholar 

  • Mai PX, Goknil A, Shar LK, Pastore F, Briand LC, Shaame S (2018a) Modeling security and privacy requirements: a use case-driven approach. Inf Softw Technol 100:165–182

  • Mai PX, Pastore F, Goknil A, Briand LC (2018b) A natural language programming approach for requirements-based security testing. In: ISSRE’18, pp 58–69

  • Mai PX, Pastore F, Goknil A, Briand LC (2019) MCP: a security testing tool driven by requirements. In: ICSE’19, pp 55–58

  • Malaiya YK, Denton H (1999) Requirements volatility and defect density. In: ISSRE’99, pp 285–298

  • Mirarab S, Ganjali A, Tahvildari L, Li S, Liu W, Morrissey M (2008) A requirement-based software testing framework: an industrial practice. In: ICSM’08, pp 452–455

  • Muccini H (2007) Using model differencing for architecture-level regression testing. In: SEAA’07

  • Muccini H, Dias M, Richardson DJ (2006) Software architecture-based regression testing. J Syst Softw 79:1379–1396

    Google Scholar 

  • Nardo DD, Alshahwan N, Briand L, Labiche Y (2015) Coverage-based regression test case selection, minimization and prioritization: a case study on an industrial system. Softw Test Verif Reliab 25(4):371–396

    Google Scholar 

  • Nebut C, Fleurey F, Traon YL, Jezequel JM (2006a) Automatic test generation: a use case driven approach. IEEE Trans Softw Eng 32 (3):140–155

  • Nebut C, Traon YL, Jezequel JM (2006b) System testing of product families: from requirements to test cases. In: Software Product Lines. Springer, Berlin

  • Oster S, Wübbeke A, Engels G, Schürr A (2011) A survey of model-based software product lines testing. In: Model-based testing for embedded systems, pp 338–381

  • Parejo JA, Sánchez A B, Segura S, Ruiz-Cortés A, Lopez-Herrejon RE, Egyed A (2016) Multi-objective test case prioritization in highly configurable systems: a case study. J Syst Softw 122:287–310

    Google Scholar 

  • Pohl K, Bockle G, van der Linden F (2005) Software product line engineering: foundations, principles, and techniques. Springer, Berlin

    MATH  Google Scholar 

  • Qu X, Acharya M, Robinson B (2011) Impact analysis of configuration changes for test case selection. In: ISSRE’11, pp 140–149

  • Ramesh B, Jarke M (2001) Toward reference models for requirements traceability. IEEE Trans Softw Eng 27(1):58–93

    Google Scholar 

  • Reuys A, Kamsties E, Pohl K, Reis S (2005) Model-based system testing of software product families. In: CAiSE’05, pp 519–534

  • Reuys A, Reis S, Kamsties E, Pohl K (2006) The ScenTED method for testing software product lines. In: Software product lines, pp 479–520

  • Rice JA (2007) Mathematical statistics and data analysis. Thomson Higher Education

  • Rothermel G, Harrold MJ (1996) Analyzing regression test selection techniques. IEEE Trans Softw Eng 22(8):529–551

    Google Scholar 

  • Rothermel G, Harrold MJ (1997) A safe, efficient regression test selection technique. ACM Trans Softw Eng Methodol 6(2):173–210

    Google Scholar 

  • Rothermel G, Harrold MJ, Dedhia J (2000) Regression test selection for C++ software. Softw Test Verif Reliab 10(2):77–109

    Google Scholar 

  • Rothermel G, Untch RH, Chu C, Harrold MJ (2001) Prioritizing test cases for regression testing. IEEE Trans Softw Eng 27(10):929–948

    Google Scholar 

  • RTCA (2018) EUROCAE: DO-178C: software considerations in airborne systems and equipment certification

  • Runeson P, Engström E (2012) Regression testing in software product line engineering. In: Advances in computers, vol 86, pp 223–263

  • Schurr A, Oster S, Markert F (2010) Model-driven software product lines testing: an integrated approach. In: SOFSEM’10, pp 112–131

  • Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45(4):427–437

    Google Scholar 

  • Srikanth H, Banerjee S (2012) Improving test efficiency through system test prioritization. J Syst Softw 85:1176–1187

    Google Scholar 

  • Srikanth H, Williams L, Osborne J (2005) System test case prioritization of new and regression test cases. In: ESEM’05, pp 64–73

  • Srikanth H, Williams L, Osborne J (2014) Towards the prioritization of system test cases. Softw Test Verif Reliab 24:320–337

    Google Scholar 

  • Srikanth H, Hettiarachchi C, Do H (2016) Requirements based test prioritization using risk factors: an industrial study. Inf Softw Technol 69:71–83

    Google Scholar 

  • Stricker V, Metzger A, Pohl K (2010) Avoiding redundant testing in application engineering. In: SPLC’10, pp 226–240

  • Tahat L, Korel B, Harman M, Ural H (2012) Regression test suite prioritization using system models. Softw Test Verif Reliab 22(7):481–506

    Google Scholar 

  • Tevanlinna A, Taina J, Kauppinen R (2004) Product family testing: a survey. ACM SIGSOFT Softw Eng Notes 29(2):12–12

    Google Scholar 

  • The R project (2018) https://www.r-project.org

  • Tonella P, Avesani P, Susi A (2006) Using the case-based ranking methodology for test case prioritization. In: ICSM’06, pp 123–133

  • Uzuncaova E, Garcia D, Khurshid S, Batory DS (2008) Testing software product lines using incremental test generation. In: ISSRE’08, pp 249–258

  • Uzuncaova E, Khurshid S, Batory DS (2010) Incremental test generation for software product lines. IEEE Trans Softw Eng 36(3):309–322

    Google Scholar 

  • Vaysburg B, Tahat LH, Korel B (2002) Dependence analysis in reduction of requirement based test suites. In: ISSTA’02, pp 107–111

  • von Mayrhauser A, Zhang N (1999) Automated regression testing using dbt and sleuth. J Softw Maint 11(2):93–116

    Google Scholar 

  • Wang S, Buchmann D, Ali S, Gotlieb A, Pradhan D, Liaaen M (2014) Multi-objective test prioritization in software product line testing: an industrial case study. In: SPLC’14, pp 32–41

  • Wang C, Pastore F, Goknil A, Briand LC, Iqbal MZZ (2015a) Automatic generation of system test cases from use case specifications. In: ISSTA’15, pp 385–396

  • Wang C, Pastore F, Goknil A, Briand LC, Iqbal MZZ (2015b) UMTG: a toolset to automatically generate system test cases from use case specifications. In: ESEC/SIGSOFT FSE’15, pp 942–945

  • Wang S, Ali S, Gotlieb A, Liaaen M (2016) A systematic test case selection methodology for product lines: results and insights from an industrial case study. Empir Softw Eng 21(4):1586–1622

    Google Scholar 

  • Wang S, Ali S, Gotlieb A, Liaaen M (2017) Automated product line test case selection: industrial case study and controlled experiment. Softw Syst Model 16(2):417–441

    Google Scholar 

  • Wang C, Pastore F, Goknil A, Briand LC (2020) Automatic generation of acceptance test cases from use case specifications: an nlp-based approach. IEEE Trans Softw Eng

  • Wong WE, Horgan JR, London S, Bellcore HA (1997) A study of effective regression testing in practice. In: ISSRE’97, pp 230–238

  • Yang Q, Li JJ, Weiss DM (2009) A survey of coverage-based testing tools. Comput J 52(5):589–597

    Google Scholar 

  • Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Softw Test Verif Reliab 22(2):67–120

    Google Scholar 

  • Yue T, Briand LC, Labiche Y (2013) Facilitating the transition from use case models to analysis models: approach and experiments. ACM Trans Softw Eng Methodol 22:1–38

    Google Scholar 

  • Zhang M, Yue T, Ali S, Selic B, Okariz O, Norgre R, Intxausti K (2018) Specifying uncertainty in use case models. J Syst Softw 144:573–603

    Google Scholar 

Download references

Acknowledgments

We gratefully acknowledge funding from FNR and IEE S.A. Luxembourg, the grants FNR/P10/03 and FNR10045046, the Canada Research Chair program, and the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 694277).

Author information

Authors and Affiliations

  1. SnT Centre, University of Luxembourg, Luxembourg, Luxembourg

    Ines Hajri, Arda Goknil, Fabrizio Pastore & Lionel C. Briand

  2. School of EECS, University of Ottawa, Ottawa, Canada

    Lionel C. Briand

Authors
  1. Ines Hajri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arda Goknil
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fabrizio Pastore
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lionel C. Briand
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabrizio Pastore.

Additional information

Communicated by: Sven Apel

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hajri, I., Goknil, A., Pastore, F. et al. Automating system test case classification and prioritization for use case-driven testing in product lines. Empir Software Eng 25, 3711–3769 (2020). https://doi.org/10.1007/s10664-020-09853-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-020-09853-4

Keywords

Search

Navigation