Model-based testing of software product lines: Mapping study and research roadmap

Highlights

• Community is engaged at mitigating previous known challenges.

• Panorama benefits researchers/practitioners on open issues and remaining challenges.

• Valuable built roadmap for research/practitioners prospective works.

• Remaining challenges are high roads for new research and industry practice.

• Variability is still a huge challenge at model-based testing of product lines.

Abstract

Model-Based Testing (MBT) has been successfully applied to Software Product Lines (SPL). This paper provides a panorama of state-of-the-art on MBT of SPLs. We performed a systematic mapping for answering questions related with domains, approaches, solution types, variability, test case automation, artifacts, and evaluation. We built a roadmap from 44 selected studies. Main obtained results are: Software and Automotive domains are most considered; Black-box testing is widely performed; most studies have fully-automated support; variability is considered in most studies; Finite State Machines is the most used model to test SPLs; Behavioral-based and Scenario-based are the most used models; Case Studies and Experiments are used to evaluate MBT solutions and the majority is performed in industrial environments; traceability is not widely explored for MBT solutions. Furthermore, we provide a roadmap synthesizing studies based on used models, more formal artifacts, supporting tools, variability management, (semi-)automation, and traceability. The roadmap contributes to identify related primary studies based on given artifacts, variability management, tools, automation, and traceability techniques and to identify, from a given primary study, which artifacts, tools, variability management, automation and traceability techniques are related. Therefore, the roadmap serves as a guide to researchers and practitioners on how to model-based test SPLs.

Introduction

Software testing is an essential activity to improve products quality (Garousi and Mntyl, 2016). Due to increasing and important research on software design, models are the main target on software testing on early stages of software development. Design models might be created, for instance, to model software requirements and to describe the software architecture of a system (Ciccozzi, Malavolta, Selic, 2018, Kosar, Bohra, Mernik, 2016). Therefore, Model-Based Testing (MBT) becomes an alternative for software testing to discover faults in software design models, thus, reducing effort and costs in maintenance and evolution of software systems (Bernardino, Rodrigues, Zorzo, Marchezan, 2017, Gurbuz, Tekinerdogan, 2018, Minhas, Masood, Petersen, Nadeem, 2018). Fixing issues in source code, for example, might cost several times more than to fix something in early stage of software development, e.g. in models (van Megen and Meyerhoff, 1995).

MBT provides test cases from a model that describes aspects, usually functional, of the system being tested. These test cases are known as the abstract test suites, and their level of abstraction is closely related to the model abstraction level (Carmo Machado et al., 2014). Some of the advantages of MBT are: i) to begin the generation of test cases earlier in the software development lifecycle; and, ii) to create test cases automatically from such models.

Although MBT has demonstrated its effectiveness in testing single systems, it is a challenge to accomplish it to mass customization approaches, such as in Software Product Lines (SPL) (Pohl, Böckle, van der Linden, 2005, Linden, Schmid, Rommes, 2007). An SPL represents a set of systems sharing common and variable aspects for a given domain (Clements and Northrop, 2001). The SPL engineering is basically composed of two activities: domain engineering, which is responsible for creating reusable assets encompassing respective variabilities; and application engineering, in which such assets are instantiated by resolving variabilities and producing specific SPL products. SPLs are straightforward based on models, such as, features and variability, for representing their similarities and variable aspects (Raatikainen et al., 2019). Thus, models are inheriting parts of an SPL and they are a factor to determine its success. Low quality SPL models generate a higher cost of maintaining a software product (Engström and Runeson, 2011). In addition, testing all potential SPL products from the perspective of the domain engineering core assets is unfeasible, as widely discussed in the existing literature (Carmo Machado, Mcgregor, Cavalcanti, De Almeida, 2014, Lamancha, Polo, Piattini, 2013, Engstrom, Runeson, do Carmo Machado, Almeida, de Lemos Meira, 2011, Silveira Neto, Machado, McGregor, Almeida, Meira, 2011, Silveira Neto, Per Runeson, Almeida, de Lemos Meira, Engstrm, 2011).

One of main challenges in MBT of SPL is related to the particularities of each model, in which it is necessary to provide models within the SPL domain engineering, for later generation of test cases in the SPL application engineering (Carmo Machado, Mcgregor, Cavalcanti, De Almeida, 2014, Lamancha, Polo, Piattini, 2013, Engström, Runeson, 2011). Another great challenge is the SPL variability (Raatikainen, Tiihonen, Mnnist, 2019, Engström, Runeson, 2011, Capilla, Bosch, Kang, 2013). This challenge is due to the large number of specific products an SPL might derive. Therefore, testing all potential SPL specific products might become unfeasible (Carmo Machado, Mcgregor, Cavalcanti, De Almeida, 2014, Engström, Runeson, 2011). In this case, MBT must take into account SPL models with variability for generating test cases, then converting them into artifacts that can be tested and reused later, preserving their characteristics (Lamancha, Usaola, Velthius, 2010, Lamancha, Mateo, de Guzmán, Usaola, Velthius, 2009, Reales, Polo, Caivano, 2011). For this reason, MBT on SPLs is a promising research area.

Although several secondary studies have been published within the SPL testing context (Carmo Machado, Mcgregor, Cavalcanti, De Almeida, 2014, Garousi, Mntyl, 2016, Lamancha, Polo, Piattini, 2013, Engström, Runeson, 2011, Lee, Kang, Lee, 2012, Silveira Neto, Machado, McGregor, Almeida, Meira, 2011), they had a wider scope compared to MBT to SPL. For example, Razak et al. (2017) provide a superficial Systematic Mapping Study (SMS) on SPL, with no further relevant discussions.

Therefore, this paper extends previous works, through an SMS, with a panorama of current literature on MBT for SPL, as well as a roadmap based on such a panorama. Furthermore, this SMS contributes to systematically identify, summarize and evaluate the existing literature in order to find gaps in the area and to position new research and practice activities. Hence, this SMS helps:

• New researchers in a structured understanding of the area by indexing the existing studies and by providing new research directions;

• Practitioners to understand state-of-the-art artifacts tools, variability management techniques, automation, and traceability techniques and their appropriate usage to MBT of SPLs; and

• To provide researchers and practitioners a roadmap on MBT of SPL.

The roadmap, which graphically depicts the analyzed panorama from the SMS, provides readers with the ability to identify which artifacts, tools, techniques and traceability adopted in each of the MBT of SPL stages. Thus, it provides researchers and practitioners a guidance at:

• Identifying which artifacts (class diagram, state machine, sequence diagram, use case diagram, etc), supporting tools, variability management techniques, automation (full, semi, or manual), and traceability techniques have been used for deriving SPL MBT test sequences;

• Given specific artifacts, tools, variability management techniques, automation and traceability techniques, to identify related primary studies that use them; and

• Given a specific primary study, how to identify the artifacts, tools, variability management techniques, automation and traceability techniques it uses.

The first systematic mapping study on SPL testing was performed in 2011 by Engström and Runeson (2011) aimed at surveying existing research to identify useful approaches and needs for future research. In general, a majority of the analyzed studies are of proposal research types (64%). System testing is the largest group with respect to research focus (40%), followed by management (23%). Method contributions are in majority. Although such a mapping study covered general approaches for SPL testing, we can observe ours partially confirmed evidence provided by Engström and Runeson (2011) and add findings specific for MBT of SPLs.

On the contrary of the Engström and Runeson (2011) complains on lack of evidence (in about 90% of studies), our SMS found majority (79.5%) of the studies provide case studies or experiments as proposals evidence methods for MBT of SPLs. From such evidenced studies, most of them (59%) are in the industry sets. This means, for the MBT of SPL specific topic, actual evidence-based works grown in the last years, which is widely desired in this area. In addition, as SPL testing is costly, MBT of SPL researches are investing time and effort to provide more close-to-reality evidence.

Although the SMS of Engström and Runeson (2011) claim testability issues, especially related to the product line architecture (PLA) have an underdeveloped potential to be researched, in our SMS we found most of solution types are focused on testing the PLA (40.9%). We understand the topic of MBT of SPL concentrates most of its activities in early stages of SPL development, thus focusing most on the PLA and its models.

Our SMS found most of studies propose testing approaches (75%) as in Engström and Runeson (2011).

This paper is organized as follows. Section 2 discusses essential background and related work; Section 3 presents the research methodology adopted to conduct this study; Section 4 presents results of our SMS; Section 5 discusses results; Section 6 discusses validity evaluation of this study; Section 7 presents a roadmap of MBT applied to SPL; Section 8 discuss remaining challenges and open issues on MBT of SPL; and Section 9 presents final remarks.