Abstract
Mutation testing is a well-established but costly technique to assess and improve the fault detection ability of test suites. This technique consists of introducing subtle changes in the code of a program, which are expected to be detected by the designed test cases. Among the strategies conceived to reduce its cost, evolutionary mutation testing (EMT) has been revealed as a promising approach to select a subset of the whole set of mutants based on a genetic algorithm (GA). However, like any other metaheuristic approach, EMT’s execution depends on a set of parameters (both classical of GAs and context-specific ones), so different configurations can greatly vary its performance. Currently, it is difficult to clarify what are the best values for those parameters by applying manual parameter tuning and whether new design choices could improve its effectiveness with other combinations of values. The experience carried out in this paper applying iterated racing, a well-known automated configuration algorithm, reveals that EMT’s performance has been undervalued in previous studies; the new configuration found by iterated racing was able to enhance EMT’s results in all C++ object-oriented programs used in the experiments. This study also confirms alternative design choices as convenient options to improve EMT in this context, namely, detecting and penalizing equivalent mutants by means of Trivial Compiler Equivalence, and learning which mutation operators produced live mutants in the past generations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Please note that the term mutation operator has a different meaning in the context of mutation testing and in the context of GAs. To avoid confusion, the former will be referred to as operator and the latter as mutation operator from now on.
References
Balera, J.M., & de Santiago Júnior, V.A. (2019). A systematic mapping addressing hyper-heuristics within search-based software testing. Information and Software Technology, 114, 176–189. https://doi.org/10.1016/j.infsof.2019.06.012.
Banzi, A.S., Nobre, T., Pinheiro, G.B., Árias, J.C.G., Pozo, A., & Vergilio, S.R. (2012). Selecting mutation operators with a multiobjective approach. Expert Systems with Applications, 39(15), 12131–12142. https://doi.org/10.1016/j.eswa.2012.04.041.
Barbosa, E.F., Maldonado, J.C., & Vincenzi, A.M.R. (2001). Toward the determination of sufficient mutant operators for C. Software Testing, Verification and Reliability, 11(2), 113–136. https://doi.org/10.1002/stvr.226.
Boubeta-Puig, J., García-Domínguez, A., & Medina-Bulo, I. (2011). Analogies and differences between mutation operators for WS-BPEL 2.0 and other languages. In Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, Berlin, Germany, p 398–407. print ISBN: 978-1-4577-0019-4, DOI https://doi.org/10.1109/ICSTW.2011.52.
Budd, T.A. (1980). Mutation analysis of program test data. PhD thesis, Yale University.
Coy, S., Golden, B., Runger, G., & Wasil, E. (2001). Using experimental design to find effective parameter settings for heuristics. Journal of Heuristics, 7 (1), 77–97. https://doi.org/10.1023/A:1026569813391.
Delgado-Pérez, P., & Medina-Bulo, I. (2018). Search-based mutant selection for efficient test suite improvement: evaluation and results. Information and Software Technology, 104, 130–143. https://doi.org/10.1016/j.infsof.2018.07.011.
Delgado-Pérez, P., & Segura, S. (2019). Study of trivial compiler equivalence on C++ object-oriented mutation operators. In Proceedings of the 34th ACM/SIGAPP symposium on applied computing, ACM, New York, NY, USA, SAC ’19, pp 2224–2230, DOI https://doi.org/10.1145/3297280.3297499.
Delgado-Pérez, P., Medina-Bulo, I., & Núñez, M. (2017). Using evolutionary mutation testing to improve the quality of test suites. In 2017 IEEE Congress on Evolutionary Computation (CEC), pp 596–603, DOI https://doi.org/10.1109/CEC.2017.7969365.
Delgado-Pérez, P., Medina-Bulo, I., Palomo-Lozano, F., García-Domínguez, A., & Domínguez-Jiménez, J.J. (2017). Assessment of class mutation operators for C++ with the MuCPP mutation system. Information and Software Technology, 81, 169–184. https://doi.org/10.1016/j.infsof.2016.07.002.
Delgado-Pérez, P., Rose, L.M., & Medina-Bulo, I. (2019). Coverage-based quality metric of mutation operators for test suite improvement. Software Quality Journal, 27(2), 823–859. https://doi.org/10.1007/s11219-018-9425-7.
Dolphin. (2019). Dolphin. https://www.kde.org/applications/system/dolphin.
de Oliveira, A.A.L., Camilo-Junior, C.G., & Vincenzi, A.M.R. (2013). A coevolutionary algorithm to automatic test case selection and mutant in mutation testing. In IEEE Congress on Evolutionary Computation, 2013, pp 829–836, DOI https://doi.org/10.1109/CEC.2013.6557654.
Domínguez-Jiménez, J., Estero-Botaro, A., García-Domínguez, A., & Medina-Bulo, I. (2009). GAMera: an automatic mutant generation system for WS-BPEL compositions. In Eshuis, R., Grefen, P., & Papadopoulos, G.A. (Eds.) Proceedings of the 7th IEEE European conference on web services, IEEE computer society press, Eindhoven, The Netherlands, pp 97–106.
Domínguez-Jiménez, J., Estero-Botaro, A., García-Domínguez, A., & Medina-Bulo, I. (2011). Evolutionary mutation testing. Information and Software Technology, 53(10), 1108–1123. https://doi.org/10.1016/j.infsof.2011.03.008.
Eiben, A.E., & Smith, J.E. (2015). Introduction to evolutionary computing, 2nd edn. Berlin: Springer Publishing Company Incorporated.
Estero-Botaro, A., Palomo-Lozano, F., & Medina-Bulo, I. (2010). Quantitative evaluation of mutation operators for WS-BPEL compositions. In Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), 2010, pp 142–150, DOI https://doi.org/10.1109/ICSTW.2010.36.
Goldberg, D.E. (1989). Genetic algorithms in search, optimization and machine learning, 1st edn. Boston: Addison-Wesley Longman Publishing Co., Inc.
Gutiérrez-Madroñal, L., García-Domínguez, A., & Medina-Bulo, I. (2019). Evolutionary mutation testing for IoT with recorded and generated events. Software: Practice and Experience, 49(4), 640–672. https://doi.org/10.1002/spe.2629.
Hutter, F., Hoos, H.H., Leyton-Brown, K., & Stützle, T. (2009). ParamILS: an automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36(1), 267–306.
Jia, Y., & Harman, M. (2008). Constructing subtle faults using higher order mutation testing. In Eighth IEEE International Working Conference on Source Code Analysis and Manipulation, 2008, pp 249–258, DOI https://doi.org/10.1109/SCAM.2008.36.
Khari, M., & Kumar, P. (2019). An extensive evaluation of search-based software testing: a review. Soft Computing, 23, 1933–1946. https://doi.org/10.1007/s00500-017-2906-y.
Lima, J.A.P., & Vergilio, S.R. (2017). A multi-objective optimization approach for selection of second order mutant generation strategies. In Proceedings of the 2Nd Brazilian symposium on systematic and automated software testing, ACM, New York, NY, USA, SAST, pp 6:1–6:10. https://doi.org/10.1145/3128473.3128479.
López-Ibáñez, M., Dubois-Lacoste, J., Cáceres, L.P., Birattari, M., & Stützle, T. (2016). The irace package: iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3, 43–58. https://doi.org/10.1016/j.orp.2016.09.002.
Madeyski, L., Orzeszyna, W., Torkar, R., & Jozala, M. (2014). Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Transactions on Software Engineering, 40(1), 23–42. https://doi.org/10.1109/TSE.2013.44.
Matrix TCL Pro. (2019). TCL Pro. http://www.techsoftpl.com/matrix/download.php.
McMinn, P. (2004). Search-based software test data generation: a survey. Software Testing, Verification and Reliability, 14(2), 105–156. https://doi.org/10.1002/stvr.v14:2.
MuParser. (2019). MuParser. https://beltoforion.de/article.php?a=muparser.
MySQL Server. (2019). MySQL Server. https://github.com/mysql/mysql-server.
Papadakis, M., & Malevris, N. (2011). Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing. Software Quality Journal, 19(4), 691–723. https://doi.org/10.1007/s11219-011-9142-y.
Papadakis, M., Jia, Y., Harman, M., & Le Traon, Y. (2015). Trivial compiler equivalence: a large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In Proceedings of the 37th international conference on software engineering - Volume 1, IEEE Press, Piscataway, NJ, USA, ICSE’15, pp 936–946, DOI https://doi.org/10.1109/ICSE.2015.103.
Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y.L., & Harman, M. (2019). Chapter six - mutation testing advances: an analysis and survey. Advances in Computers, 112, 275–378. https://doi.org/10.1016/bs.adcom.2018.03.015.
Zhang, Q., Sun, J., & Tsang, E. (2005). An evolutionary algorithm with guided mutation for the maximum clique problem. IEEE Transactions on Evolutionary Computation, 9(2), 192–200. https://doi.org/10.1109/TEVC.2004.840835.
QtDOM. (2019). QtDOM. https://github.com/qtproject/qtbase/tree/dev/src/xml/dom.
Rodrigues, D.S., Delamaro, M.E., Corrêa, C.G., & Nunes, F.L.S. (2018). Using genetic algorithms in test data generation: a critical systematic mapping. ACM Computing Surveys, 51(2), 41:1–41:23. https://doi.org/10.1145/3182659.
Schwarz, B., Schuler, D., & Zeller, A. (2011). Breeding high-impact mutations. In Proceedings of the 4th IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2011, pp 382–387, DOI https://doi.org/10.1109/ICSTW.2011.56.
Segura, S., Hierons, R.M., Benavides, D., & Ruiz-Cortés, A. (2011). Mutation testing on an object-oriented framework: an experience report. Information and Software Technology, 53(10), 1124–1136. https://doi.org/10.1016/j.infsof.2011.03.006. special Section on Mutation Testing.
Silva, R.A., do Rocio Senger de Souza, S., & de Souza, P.S.L. (2016). A systematic review on search based mutation testing. Information and Software Technology.
Tinyxml2. (2019). Tinyxml2. https://github.com/leethomason/tinyxml2.
Wu, F., Nanavati, J., Harman, M., Jia, Y., & Krinke, J. (2017). Memory mutation testing. Information and Software Technology, 81, 97–111. https://doi.org/10.1016/j.infsof.2016.03.002.
XmlRPC++. (2019). XmlRPC++. http://xmlrpcpp.sourceforge.net/.
Funding
This work was partially supported by the European Commission (European Regional Development Fund - ERDF), the Spanish Ministry of Science, Innovation and Universities under projects RTI2018-093608-B-C33 and TIN2017-88213-R, the excellence network RED2018-102472-T, the University of Málaga, and Consejería de Economía y Conocimiento de la Junta de Andalucía (grant number UMA18-FEDERJA-003)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Delgado-Pérez, P., Chicano, F. Evaluation of alternative design choices for evolutionary mutation testing by means of automated configuration. Software Qual J 30, 101–126 (2022). https://doi.org/10.1007/s11219-020-09530-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-020-09530-1