Skip to main content
SpringerLink
Log in

Assessing exception handling testing practices in open-source libraries

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Modern programming languages (e.g., Java and C#) provide features to separate error-handling code from regular code, seeking to enhance software comprehensibility and maintainability. Nevertheless, the way exception handling (EH) code is structured in such languages may lead to multiple, different, and complex control flows, which may affect the software testability. Previous studies have reported that EH code is typically neglected, not well tested, and its misuse can lead to reliability degradation and catastrophic failures. However, little is known about the relationship between testing practices and EH testing effectiveness. In this exploratory study, we (i) measured the adequacy degree of EH testing concerning code coverage (instruction, branch, and method) criteria; and (ii) evaluated the effectiveness of the EH testing by measuring its capability to detect artificially injected faults (i.e., mutants) using 7 EH mutation operators. Our study was performed using test suites of 27 long-lived Java libraries from open-source ecosystems. Our results show that instructions and branches within catch blocks and throw instructions are less covered, with statistical significance, than the overall instructions and branches. Nevertheless, most of the studied libraries presented test suites capable of detecting more than 70% of the injected faults. From a total of 12,331 mutants created in this study, the test suites were able to detect 68% of them.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://wiki.c2.com/?ExceptionPatterns

  2. https://apache.org/index.html#projects-list

  3. https://commons.apache.org/

  4. http://cobertura.github.io/cobertura/

  5. https://www.jacoco.org/jacoco/

  6. http://openclover.org/

  7. https://www.jacoco.org/jacoco/trunk/coverage/report.dtd

  8. https://www.jacoco.org/jacoco/trunk/coverage/jacoco.xml

  9. https://www.eclipse.org/jgit/

  10. https://maven.apache.org/shared/maven-invoker/

  11. https://docs.gradle.org/current/userguide/embedding.html

  12. https://spoon.gforge.inria.fr/

  13. https://commons.apache.org/proper/commons-io/project-reports.html

  14. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.friedmanchisquare.html

References

  • Ahmed I, Gopinath R, Brindescu C, Groce A, Jensen C (2016) Can testedness be effectively measured?. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, ACM, New York, NY, USA, FSE 2016, pp 547–558. https://doi.org/10.1145/2950290.2950324

  • Ammann P, Offutt J (2016) Introduction to software testing, 2nd edn. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Antinyan V, Derehag J, Sandberg A, Staron M (2018) Mythical unit test coverage. IEEE Softw 35(3):73–79. https://doi.org/10.1109/MS.2017.3281318

    Article  Google Scholar 

  • Barbosa EA, Garcia A, Barbos SDJ (2014) Categorizing faults in exception handling: A study of open source projects. In: 2014 Brazilian Symposium on Software Engineering (SBES), pp 11–20

  • Bavota G, De Lucia A, Di Penta M, Oliveto R, Palomba F (2015) An experimental investigation on the innate relationship between quality and refactoring. J Syst Softw 107:1–14

    Article  Google Scholar 

  • Benjamini Y, Yekutieli D (2001) The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics 29(4):1165–1188. http://www.jstor.org/stable/2674075

    Article  MathSciNet  Google Scholar 

  • Bloch J (2008) Effective java (The Java Series), 2nd edn. Prentice Hall PTR, Upper Saddle River

    Google Scholar 

  • Cacho N, Barbosa EA, Araujo J, Pranto F, Garcia A, Cesar T, Soares E, Cassio A, Filipe T, Garcia I (2014a) How does exception handling behavior evolve? an exploratory study in java and c# applications. In: 2014 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 31–40

  • Cacho N, César T, Filipe T, Soares E, Cassio A, Souza R, Garcia I, Barbosa EA, Garcia A (2014b) Trading robustness for maintainability: An empirical study of evolving c# programs. In: Proceedings of the 36th international conference on software engineering, ACM, New York, NY, USA, ICSE 2014, pp 584–595. https://doi.org/10.1145/2568225.2568308

  • Chang BM, Choi K (2016) A review on exception analysis. Inf Softw Technol 77(C):1–16

    Article  Google Scholar 

  • Chekam TT, Papadakis M, Le Traon Y, Harman M (2017) An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp 597–608. https://doi.org/10.1109/ICSE.2017.61

  • Chen H, Dou W, Jiang Y, Qin F (2019) Understanding exception-related bugs in large-scale cloud systems. In: Proceedings of the 34rd ACM/IEEE International Conference on Automated Software Engineering, ACM, New York, NY, USA, ASE

  • Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychol Bull 114(4):494–509

    Article  Google Scholar 

  • Coelho R, Almeida L, Gousios G, Deursen A V, Treude C (2017) Exception handling bug hazards in android. Empirical Softw Engg 22(3):1264–1304

    Article  Google Scholar 

  • Dalton F, Ribeiro M, Pinto G, Fernandes L, Gheyi R, Fonseca B (2020) Is exceptional behavior testing an exception? an empirical assessment using java automated tests. In: Proceedings of the Evaluation and Assessment in Software Engineering, Association for Computing Machinery, New York, NY, USA, EASE ’20, pp 170–179. https://doi.org/10.1145/3383219.3383237

  • Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30. http://dl.acm.org/citation.cfm?id=1248547.1248548

    MathSciNet  MATH  Google Scholar 

  • Digkas G, Lungu M, Avgeriou P, Chatzigeorgiou A, Ampatzoglou A (2018) How do developers fix issues and pay back technical debt in the apache ecosystem?. In: 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp 153–163. https://doi.org/10.1109/SANER.2018.8330205

  • Ebert F, Castor F, Serebrenik A (2015) An exploratory study on exception handling bugs in java programs. J Syst Softw 106(C):82–101

    Article  Google Scholar 

  • Eck M, Palomba F, Castelluccio M, Bacchelli A (2019) Understanding flaky tests: The developer’s perspective. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, New York, NY, USA, ESEC/FSE, pp 830–840. https://doi.org/10.1145/3338906.3338945

  • Filho JLM, Rocha L, Andrade R, Britto R (2017) Preventing erosion in exception handling design using static-architecture conformance checking. In: Proceedings of the 11th European Conference on Software Architecture, Springer International Publishing, Cham, ECSA ’17, pp 67–83. https://doi.org/10.1007/978-3-319-65831-5_5

  • Friedman M (1940) A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics 11(1):86–92

    Article  MathSciNet  Google Scholar 

  • Gallardo R, Hommel S, Kannan S, Gordon J, Zakhour S B (2014) The Java tutorial: a short course on the basics. Java Series, 6th edn. Addison-Wesley Professional, Boston

    Google Scholar 

  • Garcia A F, Rubira C M, Romanovsky A, Xu J (2001) A comparative study of exception handling mechanisms for building dependable object-oriented software. J Syst Softw 59(2):197–222

    Article  Google Scholar 

  • Gligoric M, Groce A, Zhang C, Sharma R, Alipour MA, Marinov D (2013) Comparing non-adequate test suites using coverage criteria. In: Proceedings of the 2013 International symposium on software testing and analysis, association for computing machinery, New York, NY, USA, ISSTA 2013, pp 302–313. https://doi.org/10.1145/2483760.2483769

  • Goffi A, Gorla A, Ernst MD, Pezzè M (2016) Automatic generation of oracles for exceptional behaviors. In: Proceedings of the 25th International symposium on software testing and analysis, association for computing machinery, New York, NY, USA, ISSTA 2016, pp 213–224. https://doi.org/10.1145/2931037.2931061

  • Goodenough J B, Gerhart S L (1975) Toward a theory of test data selection. IEEE Trans Softw Eng (2)156–173

  • Gopinath R, Jensen C, Groce A (2014) Code coverage for suite evaluation by developers. In: Proceedings of the 36th International conference on software engineering, association for computing machinery, New York, NY, USA, ICSE 2014, pp 72–82. https://doi.org/10.1145/2568225.2568278

  • Gulati S, Sharma R (2017) Java unit testing with JUnit 5: Test Driven Development with JUnit 5, 1st edn. Apress, USA

    Book  Google Scholar 

  • Hilton M, Bell J, Marinov D (2018) A large-scale study of test coverage evolution. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, New York, NY, USA, ASE 2018, pp 53–63. https://doi.org/10.1145/3238147.3238183

  • Hunt A, Thomas D (2003) Pragmatic Unit Testing in Java with JUnit. The pragmatic programmers

  • Inozemtseva L, Holmes R (2014a) Coverage is not strongly correlated with test suite effectiveness. In: Proceedings of the 36th International conference on software engineering, association for computing machinery, New York, NY, USA, ICSE 2014, pp 435–445. https://doi.org/10.1145/2568225.2568271

  • Inozemtseva L, Holmes R (2014b) Coverage is not strongly correlated with test suite effectiveness. In: Proceedings of the 36th International conference on software engineering, ACM, New York, NY, USA, ICSE 2014, pp 435–445. https://doi.org/10.1145/2568225.2568271

  • Ivanković M, Petrović G, Just R, Fraser G (2019) Code coverage at google. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ACM, New York, NY, USA, ESEC/FSE 2019, pp 955–963. https://doi.org/10.1145/3338906.3340459

  • Jenkov J (2013) Java Exception Handling. Jenkov Aps

  • Ji C, Chen Z, Xu B, Wang Z (2009) A new mutation analysis method for testing java exception handling. In: Computer software and applications conference, 2009. COMPSAC’09. 33rd Annual IEEE International, vol 2. IEEE, pp 556–561

  • Just R, Jalali D, Inozemtseva L, Ernst M D, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing?. In: Proceedings of the 22Nd ACM SIGSOFT international symposium on foundations of software engineering, ACM, New York, NY, USA, FSE 2014, pp 654–665. https://doi.org/10.1145/2635868.2635929

  • Kechagia M, Spinellis D (2014) Undocumented and unchecked: Exceptions that spell trouble. In: Proceedings of the 11th Working Conference on Mining Software Repositories, ACM, New York, NY, USA, MSR 2014, pp 312–315

  • Koch S (2007) Software evolution in open source projects—a large-scale investigation. J Softw Maint Evol 19(6):361–382

    Article  Google Scholar 

  • Kochhar PS, Thung F, Lo D (2015) Code coverage and test suite effectiveness: Empirical study with real bugs in large systems. In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp 560–564. https://doi.org/10.1109/SANER.2015.7081877

  • Kochhar P S, Lo D, Lawall J, Nagappan N (2017) Code coverage and postrelease defects: A large-scale study on open source projects. IEEE Trans Reliab 66(4):1213–1228. https://doi.org/10.1109/TR.2017.2727062

    Article  Google Scholar 

  • Kumar K, Gupta P, Parjapat R (2011) New mutants generation for testing java programs. In: International conference on advances in communication, network, and computing. Springer, pp 290–294

  • Lima L, Rocha L, Bezerra C, Paixao M (2021) Replication package for the paper: “assessing exception handling testing practices in open-source software systems. https://zenodo.org/record/4732368, Accessed: May 01, 2021

  • Luo Q, Hariri F, Eloussi L, Marinov D (2014) An empirical analysis of flaky tests. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp 643–653

  • Marinescu C (2011) Are the classes that use exceptions defect prone?. In: Proceedings of the 12th International workshop on principles of software evolution and the 7th annual ERCIM workshop on software evolution. ACM, pp 56–60

  • Marinescu C (2013) Should we beware the exceptions? an empirical study on the eclipse project. In: 2013 15th International symposium on symbolic and numeric algorithms for scientific computing (SYNASC). IEEE, pp 250–257

  • Martins AL, Hanazumi S, d Melo ACV (2014) Testing java exceptions: An instrumentation technique. In: 2014 IEEE 38th international computer software and applications conference workshops, pp 626–631. https://doi.org/10.1109/COMPSACW.2014.105

  • McMinn P (2004) Search-based software test data generation: a survey. Software Testing, Verification and reliability 14(2):105–156

    Article  Google Scholar 

  • Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering - ESEC/FSE 2013. ACM Press, New York, p 466

  • Oliveira J, Borges D, Silva T, Cacho N, Castor F (2018) Do android developers neglect error handling? a maintenance-centric study on the relationship between android abstractions and uncaught exceptions. J Syst Softw 136(C):1–18. https://doi.org/10.1016/j.jss.2017.10.032

    Article  Google Scholar 

  • Osman H, Chiş A, Corrodi C, Ghafari M, Nierstrasz O (2017) Exception evolution in long-lived java systems. In: Proceedings of the 14th International Conference on Mining Software Repositories, IEEE Press, Piscataway, NJ, USA, MSR ’17, pp 302–311

  • de Pádua GB, Shang W (2017a) Revisiting exception handling practices with exception flow analysis. In: 2017 IEEE 17th International working conference on source code analysis and manipulation (SCAM), pp 11–20. https://doi.org/10.1109/SCAM.2017.16

  • de Pádua GB, Shang W (2017b) Studying the prevalence of exception handling anti-patterns. In: Proceedings of the 25th International conference on program comprehension. IEEE Press, Piscataway, NJ, USA, ICPC’17, pp 328–331

  • de Pádua GB, Shang W (2018) Studying the relationship between exception handling practices and post-release defects. In: Proceedings of the 15th International conference on mining software repositories. ACM, New York, NY, USA, MSR ’18, pp 564–575. https://doi.org/10.1145/3196398.3196435

  • Paixao M, Krinke J, Han D, Ragkhitwetsagul C, Harman M (2017) Are developers aware of the architectural impact of their changes?. In: ASE 2017 - Proceedings of the 32nd IEEE/ACM International conference on automated software engineering

  • Papadakis M, Shin D, Yoo S, Bae DH (2018) Are mutation scores correlated with real fault detection?: A large scale empirical study on the relationship between mutants and real faults. In: Proceedings of the 40th International conference on software engineering. ACM, New York, NY, USA, ICSE ’18, pp 537–548. https://doi.org/10.1145/3180155.3180183

  • Papadakis M, Kintis M, Zhang J, Jia Y, Traon Y L, Harman M (2019) Chapter six - mutation testing advances: An analysis and survey. Advances in Computers. Elsevier 112:275–378

    Google Scholar 

  • Pawlak R, Monperrus M, Petitprez N, Noguera C, Seinturier L (2016) Spoon: A library for implementing analyses and transformations of java source code. Softw Pract Exper 46(9):1155–1179 . https://doi.org/10.1002/spe.2346

    Article  Google Scholar 

  • Rashkovits R, Lavy I (2012) Students’ misconceptions of java exceptions. In: Rahman H, Mesquita A, Ramos I, Pernici B (eds) Proceedings of the 7th Mediterranean conference on information systems. Springer, Berlin, MCIS ’12, pp 1–21. https://doi.org/10.1007/978-3-642-33244-9_1

  • Reales P, Polo M, Fernández-Alemán JL, Toval A, Piattini M (2014) Mutation testing. IEEE Software 31(3):30–35. https://doi.org/10.1109/MS.2014.68

    Article  Google Scholar 

  • Romano D, Di Penta M, Antoniol G (2011) An approach for search based testing of null pointer exceptions. In: 2011 Fourth IEEE international conference on software testing, verification and validation. IEEE, pp 160–169

  • Saha RK, Lyu Y, Lam W, Yoshida H, Prasad MR (2018) Bugs.jar: A large-scale, diverse dataset of real-world java bugs. In: Proceedings of the 15th international conference on mining software repositories, association for computing machinery, New York, NY, USA, MSR ’18, pp 10–13. https://doi.org/10.1145/3196398.3196473

  • Sawadpong P, Allen EB (2016) Software defect prediction using exception handling call graphs: A case study. In: 2016 IEEE 17th International symposium on high assurance systems engineering (HASE). IEEE, pp 55–62

  • Schwartz A, Puckett D, Meng Y, Gay G (2018) Investigating faults missed by test suites achieving high code coverage. J Syst Softw 144:106–120. https://doi.org/10.1016/j.jss.2018.06.024

    Article  Google Scholar 

  • Shah H, Harrold MJ (2009) Exception handling negligence due to intra-individual goal conflicts. In: Proceedings of the 2009 ICSE workshop on cooperative and human aspects on software engineering. IEEE Computer Society, Washington, DC, USA, CHASE ’09, pp 80–83. https://doi.org/10.1109/CHASE.2009.5071417

  • Shah H, Görg C, Harrold MJ (2008) Why do developers neglect exception handling?. In: Proceedings of the 4th International Workshop on Exception Handling, ACM, New York, NY, USA, WEH ’08, pp 62–68. https://doi.org/10.1145/1454268.1454277

  • Shah H, Gorg C, Harrold M J (2010) Understanding exception handling: Viewpoints of novices and experts. IEEE Trans Softw Eng 36(2):150–161

    Article  Google Scholar 

  • Shahrokni A, Feldt R (2013) A systematic review of software robustness. Inf Softw Technol 55(1):1–17

    Article  Google Scholar 

  • Shi L, Zhong H, Xie T, Li M (2011) An empirical study on evolution of api documentation. In: Proceedings of the 14th International conference on fundamental approaches to software engineering: part of the joint european conferences on theory and practice of software. Springer, Berlin, Heidelberg, FASE’11/ETAPS’11, pp 416–431

  • Sinha S, Harrold MJ (2000) Analysis and testing of programs with exception handling constructs. IEEE Trans Softw Eng 26(9):849–871. https://doi.org/10.1109/32.877846

    Article  Google Scholar 

  • Turner AJ, White DR, Drake JH (2016) Multi-objective regression test suite minimisation for mockito. In: Sarro F, Deb K (eds) Search based software engineering. Springer International Publishing, Cham, pp 244–249

  • Vieira R, da Silva A, Rocha L, Gomes JaP (2019) From reports to bug-fix commits: A 10 years dataset of bug-fixing activity from 55 apache’s open source projects. In: Proceedings of the fifteenth international conference on predictive models and data analytics in software engineering, Association for Computing Machinery, New York, NY, USA, PROMISE’19, pp 80–89. https://doi.org/10.1145/3345629.3345639

  • Wirfs-Brock R J (2006) Toward exception-handling best practices and patterns. IEEE Softw 23(5):11–13. https://doi.org/10.1109/MS.2006.144

    Article  Google Scholar 

  • Wohlin C, Runeson P, Hst M, Ohlsson M C, Regnell B, Wessln A (2012) Experimentation in software engineering. Springer Publishing Company Incorporated

  • Yang Y, Zhou Y, Sun H, Su Z, Zuo Z, Xu L, Xu B (2019) Hunting for bugs in code coverage tools via randomized differential testing. In: Proceedings of the 41st international conference on software engineering. IEEE Press, Piscataway, NJ, USA, ICSE ’19, pp 488–499. https://doi.org/10.1109/ICSE.2019.00061

  • Zhai H, Casalnuovo C, Devanbu P (2019) Test coverage in python programs. In: Proceedings of the 16th international conference on mining software repositories. IEEE Press, Piscataway, NJ, USA, MSR ’19, pp 116–120. https://doi.org/10.1109/MSR.2019.00027

  • Zhang P, Elbaum S (2014) Amplifying tests to validate exception handling code: An extended study in the mobile application domain. ACM Trans Softw Eng Methodol 23(4):32:1–32:28. https://doi.org/10.1145/2652483

    Article  Google Scholar 

  • Zhong H, Mei H (2019) An empirical study on api usages. IEEE Trans Softw Eng 45(4):319–334. https://doi.org/10.1109/TSE.2017.2782280

    Article  Google Scholar 

  • Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4):366–427. https://doi.org/10.1145/267580.267590

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Federal University of Ceará, Fortaleza, CE, Brazil

    Luan P. Lima & Lincoln S. Rocha

  2. Federal University of Ceará, Quixadá, CE, Brazil

    Carla I. M. Bezerra

  3. State University of Ceará, Fortaleza, CE, Brazil

    Matheus Paixao

Authors
  1. Luan P. Lima
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lincoln S. Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carla I. M. Bezerra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matheus Paixao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lincoln S. Rocha.

Additional information

Communicated by: Alexander Serebrenik

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lima, L.P., Rocha, L.S., Bezerra, C.I.M. et al. Assessing exception handling testing practices in open-source libraries. Empir Software Eng 26, 85 (2021). https://doi.org/10.1007/s10664-021-09983-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-09983-3

Keywords

Search

Navigation