Abstract
Graph Transformation System (GTS) can formally specify the behavioral aspects of complex systems through graph-based contracts. Test suite generation under normal conditions from GTS specifications is a task well-suited to evolutionary algorithms such as Genetic and Particle Swarm Optimization (PSO) metaheuristics. However, testing the vulnerabilities of a system under unexpected events such as invalid inputs is essential. Furthermore, the mentioned global search algorithms tend to make big jumps in the system’s state-space that are not concentrated on particular test goals. In this paper, we extend the HGAPSO approach into a cost-aware Memetic Algorithm (MA) by making small local changes through a proposed local search operator to optimize coverage score and testing costs. Moreover, we test GTS specifications not only under normal events but also under unexpected situations. So, three coverage-based testing strategies are investigated, including normal testing, robustness testing, and a hybrid strategy. The effectiveness of the proposed test generation algorithm and the testing strategies are evaluated through a type of mutation analysis at the model-level. Our experimental results show that (1) the hybrid testing strategy outperforms normal and robustness testing strategies in terms of fault-detection capability, (2) the robustness testing is the most cost-efficient strategy, and (3) the proposed MA with the hybrid testing strategy outperforms the state-of-the-art global search algorithms.
Similar content being viewed by others
References
Aichernig BK, Lorber F, Nickovic D (2013) Model-based mutation testing with timed automata. Graz University of Technology, Graz
Anand S et al (2013) An orchestrated survey of methodologies for automated software test case generation. J Syst Softw 86(8):1978–2001
Araujo W, Briand LC, Labiche Y (2014) On the effectiveness of contracts as test oracles in the detection and diagnosis of functional faults in concurrent object-oriented software. IEEE Trans Software Eng 40(10):971–992
Arqub OA (2017) Adaptation of reproducing kernel algorithm for solving fuzzy Fredholm–Volterra integrodifferential equations. Neural Comput Appl 28(7):1591–1610
Arqub OA et al (2016) Numerical solutions of fuzzy differential equations using reproducing kernel Hilbert space method. Soft Comput 20(8):3283–3302
Arqub OA et al (2017) Application of reproducing kernel algorithm for solving second-order, two-point fuzzy boundary value problems. Soft Comput 21(23):7191–7206
Balci O (1998) Verification, validation, and testing. Handbook of Simulation 10:335–393
Belli F et al (2016) Model-based mutation testing—approach and case studies. Sci Comput Program 120:25–48
Böhmer K, Rinderle-Ma S (2015) A systematic literature review on process model testing: approaches, challenges, and research directions. ArXiv preprint arXiv:1509.04076
de Bruijn V (2013) Model-based testing with graph grammars, Master's thesis, University of Twente, Enschede, The Netherlands
Di Nardo D et al (2015) Evolutionary robustness testing of data processing systems using models and data mutation (T). In: 2015 30th IEEE/ACM international conference on automated software engineering (ASE). IEEE
Dias Neto AC et al (2007) A survey on model-based testing approaches: a systematic review. In: Proceedings of the 1st ACM international workshop on empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM international conference on automated software engineering (ASE) 2007. ACM
Eberhart R, Kennedy J (1995) Particle swarm optimization. In: Proceedings of the IEEE international conference on neural networks. Citeseer
Ehrig H, Ehrig K (2006) Overview of formal concepts for model transformations based on typed attributed graph transformation. Electron Notes Theor Comput Sci 152:3–22
Ehrig H, Rozenberg G, Kreowski H-J (1999) Handbook of graph grammars and computing by graph transformation, vol 3. World Scientific, Singapore
Ehrig H, Prange U, Taentzer G (2004) Fundamental theory for typed attributed graph transformation. In: International conference on graph transformation. Springer
Engels G, Güldali B, Lohmann M (2006) Towards model-driven unit testing. In: International conference on model driven engineering languages and systems. Springer
Ferrer J et al (2015) Search based algorithms for test sequence generation in functional testing. Inf Softw Technol 58:419–432
Fraser G, Arcuri A (2013) Whole test suite generation. IEEE Trans Software Eng 39(2):276–291
Fraser G, Arcuri A, McMinn P (2015) A memetic algorithm for whole test suite generation. J Syst Softw 103:311–327
Gönczy L, Heckel R, Varró D (2007) Model-based testing of service infrastructure components. In: Testing of software and communicating systems. Springer, pp 155–170
Grechanik M, Devanla G (2016) Mutation integration testing. In: 2016 IEEE international conference on software quality, reliability and security (QRS). IEEE
Harman M, Jia Y, Zhang Y (2015) Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th international conference on software testing, verification and validation (ICST). IEEE
Haupt RL, Ellen Haupt S (2004) Practical genetic algorithms, 2nd edition. Wiley, New Jersey
Heckel R (2006) Graph transformation in a nutshell. Electron Notes Theor Comput Sci 148(1):187–198
Heckel R, Mariani L (2004) Component integration testing by graph transformations. In: International conference on computer science, software engineering, information technology, e-business, and applications, Cairo. Citeseer
Heckel R, Khan TA, Machado R (2011) Towards test coverage criteria for visual contracts. In: Proceedings of the Tenth International Workshop on Graph Transformation and Visual Modeling Techniques, Berlin
Hutchison C et al (2018) Robustness testing of autonomy software. In: Proceedings of the 40th international conference on software engineering: software engineering in practice. ACM
ISC Committee (1990) IEEE standard glossary of software engineering terminology (IEEE Std 610.12-1990). Los Alamitos, CA: IEEE Computer Society, 169
Jonsson MBB, Leucker J-PKM, Pretschner A (2005) Model-based testing of reactive systems. Springer, Berlin
Kalaee A, Rafe V (2018) Model-based test suite generation for graph transformation system using model simulation and search-based techniques. Inf Softw Technol 108:1-29
Khan TA, Runge O, Heckel R (2012) Testing against visual contracts: model-based coverage. In: International conference on graph transformation. Springer
Khari M, Kumar P (2017) An extensive evaluation of search-based software testing: a review. Soft Comput 23(6):1933–1946
König B et al (2018) A tutorial on graph transformation. In: Graph transformation, specifications, and nets. Springer, pp 83–104
Liu S (2016) Validating formal specifications using testing-based specification animation. In: Proceedings of the 4th FME workshop on formal methods in software engineering. ACM
Lobo FG, Lima CF (2007) Adaptive population sizing schemes in genetic algorithms. In: Parameter setting in evolutionary algorithms. Springer, pp 185–204
Lobo F, Lima CF, Michalewicz Z (2007) Parameter setting in evolutionary algorithms, vol 54. Springer, Berlin
Louzaoui K, Benlhachmi K (2017) A robustness testing approach for an object oriented model. JCP 12(4):335–353
Ma Y-S, Offutt J, Kwon Y-R (2006) MuJava: a mutation system for Java. In: Proceedings of the 28th international conference on software engineering. ACM
Machado R, Ribeiro L, Heckel R (2015) Rule-based transformation of graph rewriting rules: towards higher-order graph grammars. Theoret Comput Sci 594:1–23
Mattiello-Francisco F et al (2012) InRob: an approach for testing interoperability and robustness of real-time embedded software. J Syst Softw 85(1):3–15
Meyer B (1988) Object-oriented software construction, vol 2. Prentice Hall, New York
Offutt AJ (1992) Investigations of the software testing coupling effect. ACM Trans Softw Eng Methodol (TOSEM) 1(1):5–20
Rafe V (2013) Scenario-driven analysis of systems specified through graph transformations. J Vis Lang Comput 24(2):136–145
Rafe V, Rahmani M, Rashidi K (2013) A survey on coping with the state space explosion problem in model checking. Int Res J Appl Basic Sci 4(6):1379–1384
Rensink A (2003) The GROOVE simulator: a tool for state space generation. In: International workshop on applications of graph transformations with industrial relevance. Springer
Runge O, Khan TA, Heckel R (2013) Test case generation using visual contracts. In: Proceedings of the Tenth International Workshop on Graph Transformation and Visual Modeling Techniques, Berlin
Satpathy M et al (2007) Automatic testing from formal specifications. In: International conference on tests and proofs. Springer
Savary A et al (2015) Model-based robustness testing in event-B using mutation. In: SEFM 2015 collocated workshops. Springer
Schmidt A, Varró D (2003). CheckVML: a tool for model checking visual modeling languages. In: International conference on the unified modeling language. Springer
Strug J (2016) Mutation testing approach to negative testing. J Eng
Taentzer G (2003) AGG: a graph transformation environment for modeling and validation of software. In: International workshop on applications of graph transformations with industrial relevance. Springer
Vos TE et al (2013) Evolutionary functional black-box testing in an industrial setting. Software Qual J 21(2):259–288
Whitley D, Gordon VS, Mathias K (1994) Lamarckian evolution, the Baldwin effect and function optimization. In: International conference on parallel problem solving from nature. Springer
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Bahrampour, A., Rafe, V. Using memetic algorithm for robustness testing of contract-based software models. Artif Intell Rev 54, 877–915 (2021). https://doi.org/10.1007/s10462-020-09881-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-020-09881-y