Skip to main content
SpringerLink
Log in

An empirical study on the use of SZZ for identifying inducing changes of non-functional bugs

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Non-functional bugs, e.g., performance bugs and security bugs, bear a heavy cost on both software developers and end-users. For example, IBM estimates the cost of a single data breach to be millions of dollars. Tools to reduce the occurrence, impact, and repair time of non-functional bugs can therefore provide key assistance for software developers racing to fix these issues. Identifying bug-inducing changes is a critical step in software quality assurance. In particular, the SZZ approach is commonly used to identify bug-inducing commits. However, the fixes to non-functional bugs may be scattered and separate from their bug-inducing locations in the source code. The nature of non-functional bugs may therefore make the SZZ approach a sub-optimal approach for identifying bug-inducing changes. Yet, prior studies that leverage or evaluate the SZZ approach do not consider non-functional bugs, leading to potential bias on the results. In this paper, we conduct an empirical study on the results of the SZZ approach when used to identify the inducing changes of the non-functional bugs in the NFBugs dataset. We eliminate a majority of the bug-inducing commits as they are not in the same method or class level. We manually examine whether each identified bug-inducing change is indeed the correct bug-inducing change. Our manual study shows that a large portion of non-functional bugs cannot be properly identified by the SZZ approach. By manually identifying the root causes of the falsely detected bug-inducing changes, we uncover root causes for false detection that have not been found by previous studies. We evaluate the identified bug-inducing changes based on three criteria from prior research, i.e., the earliest bug appearance, the future impact of changes, and the realism of bug introduction. We find that prior criteria may be irrelevant for non-functional bugs. Our results may be used to assist in future research on non-functional bugs, and highlight the need to complement SZZ to accommodate the unique characteristics of non-functional bugs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Listing 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Our extension of the NFBugs dataset is publicly available and can be found at: https://github.com/senseconcordia/NFBugsExtended

References

  • Bryant RE, O’Hallaron DR (2015) Computer systems: a programmer’s perspective, 3rd edn. Pearson

  • Borg M, Svensson O, Berg K, Hansson D (2019) Szz unleashed: an open implementation of the szz algorithm—featuring example usage in a study of just-in-time bug prediction for the jenkins project. In: Proceedings of the 3rd ACM SIGSOFT international workshop on machine learning techniques for software quality evaluation, ser. MaLTeSQuE 2019. Association for Computing Machinery, New York, pp 7–12. [Online]. Available: https://doi.org/10.1145/3340482.3342742

  • da Costa DA, McIntosh S, Shang W, Kulesza U, Coelho R, Hassan AE (2017) A framework for evaluating the results of the szz approach for identifying bug-introducing changes. IEEE Trans Softw Eng 43(7):641–657

    Article  Google Scholar 

  • Davies S, Roper M, Wood M (2014) mparing text-based and dependence-based approaches for determining the origins of bugs. J Softw: Evol Process 26:01

    Article  Google Scholar 

  • Fan Y, Xia X, Alencar da Costa D, Lo D, Hassan AE, Li S (2019) The impact of changes mislabeled by szz on just-in-time defect prediction. IEEE Trans Softw Eng 1–1

  • Glinz M (2007) On non-functional requirements. In: 15th IEEE international requirements engineering conference (RE 2007), pp 21–26

  • Grubb P, Takang A (2003) Software maintenance—concepts and practice (2nd edn). 01

  • Guindon C Swt: the standard widget toolkit. [Online]. Available: https://www.eclipse.org/swt/

  • Gyimothy T, Ferenc R, Siket I (2005) Empirical validation of object-oriented metrics on open source software for fault prediction. IEEE Trans Softw Eng 31(10):897–910

    Article  Google Scholar 

  • Hamill M, Goseva-Popstojanova K (2014) Exploring the missing link: an empirical study of software fixes. Softw Test Verif Reliab 24(8):684–705. [Online]. Available: https://doi.org/10.1002/stvr.1518

    Article  Google Scholar 

  • Hassan AE (2009) Predicting faults using the complexity of code changes. In: 2009 IEEE 31st international conference on software engineering, pp 78–88

  • Jin G, Song L, Shi X, Scherpelz J, Lu S (2012) Understanding and detecting real-world performance bugs. SIGPLAN Not 47(6):77–88

    Article  Google Scholar 

  • Jpace (2018) jpace/diffj. [Online]. Available: https://github.com/jpace/dij

  • Kamei Y, Shihab E, Adams B, Hassan AE, Mockus A, Sinha A, Ubayashi N (2013) A large-scale empirical study of just-in-time quality assurance. IEEE Trans Softw Eng 39(6):757–773

    Article  Google Scholar 

  • Kim M, Lee E (2018) Are information retrieval-based bug localization techniques trustworthy?. In: Proceedings of the 40th international conference on software engineering: companion proceedings, ser. ICSE ’18. Association for Computing Machinery, New York, pp 248–249. [Online]. Available: https://doi.org/10.1145/3183440.3194954

  • Kim S, Whitehead EJ Jr (2006) How long did it take to fix bugs?. In: Proceedings of the 2006 international workshop on mining software repositories, ser. MSR ’06. ACM, New York, pp 173–174

  • Kim S, Zimmermann T, Pan K, Whitehead EJ Jr (2006) Automatic identification of bug-introducing changes. In: 21st IEEE/ACM international conference on automated software engineering (ASE’06), pp 81–90

  • Kotonya G, Sommerville I (1998) Requirements engineering: processes and techniques, 1st edn, Wiley Publishing

  • LaToza TD, Venolia G, DeLine R (2006) Maintaining mental models: a study of developer work habits. In: Proceedings of the 28th international conference on software engineering, ser. ICSE ’06. ACM, New York, pp 492–501

  • Mahrous H, Malhotra B (2018) Managing publicly known security vulnerabilities in software systems. In: 2018 16th Annual conference on privacy, security and trust (PST), pp 1–10

  • McHugh M (2012) Interrater reliability: the kappa statistic. Biochemia medica: časopis Hrvatskoga društva medicinskih biokemičara / HDMB 22:276–82, 10

    Article  Google Scholar 

  • McIntosh S, Kamei Y (2018) Are fix-inducing changes a moving target? A longitudinal case study of just-in-time defect prediction. IEEE Trans Softw Eng 44(5):412–428

    Article  Google Scholar 

  • Molyneaux I (2009) The art of application performance testing: help for programmers and quality assurance, 1st edn. O’Reilly Media, Inc.

  • Neto C, Barbalho E (2018) Enhancing the szz algorithm to deal with refactoring changes

  • Neto E, Costa D, Kulesza U (2018) The impact of refactoring changes on the szz algorithm: an empirical study. 03

  • Nistor A, Jiang T, Tan L (2013) Discovering, reporting, and fixing performance bugs. In: 2013 10th Working conference on mining software repositories (MSR), pp 237–246

  • Nugroho YS, Hata H, Matsumoto K (2019) How different are different diff algorithms in git? Empir Softw Eng 25(1):790–823. [Online]. Available: https://doi.org/10.1007/s10664-019-09772-z

    Article  Google Scholar 

  • Ohira M, Kashiwa Y, Yamatani Y, Yoshiyuki H, Maeda Y, Limsettho N, Fujino K, Hata H, Ihara A, Matsumoto K (2015) A dataset of high impact bugs: manually-classified issue reports. In: Proceedings of the 12th working conference on mining software repositories, ser. MSR ’15. IEEE Press, Piscataway, pp 518–521

  • Pan K, Kim S, Whitehead EJ Jr (2009) Toward an understanding of bug fix patterns. Empir Softw Eng 14(3):286–315

    Article  Google Scholar 

  • Ping L, Jin S, Xinfeng Y (2011) Research on software security vulnerability detection technology. In: Proceedings of 2011 international conference on computer science and network technology, vol 3, pp 1873–1876

  • Radu A, Nadi S (2019) A dataset of non-functional bugs. In: Proceedings of the 16th international conference on mining software repositories, ser. MSR ’19. IEEE Press, Piscataway, pp 399–403

  • Rosen C, Grawi B, Shihab E (2015) Commit guru: analytics and risk prediction of software commits. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering, ser. ESEC/FSE 2015. Association for Computing Machinery, New York, pp 966–969. [Online]. Available: https://doi.org/10.1145/2786805.2803183

  • Śliwerski J, Zimmermann T, Zeller A (2005) When do changes induce fixes? SIGSOFT Softw Eng Notes 30(4):1–5

    Article  Google Scholar 

  • Steidl D, Hummel B, Juergens E (2014) Incremental origin analysis of source code files. In: Proceedings of the 11th working conference on mining software repositories, ser. MSR 2014. Association for Computing Machinery, New York, pp 42–51. [Online]. Available: https://doi.org/10.1145/2597073.2597111

  • Team J. Eclipse java development tools (jdt). [Online]. Available: https://www.eclipse.org/jdt/

  • Williams C, Spacco J (2008) Szz revisited: verifying when changes induce fixes. In: Proceedings of the 2008 workshop on defects in large software systems, ser. DEFECTS ’08. ACM, New York, pp 32–36

  • Williams L, McGraw G, Migues S (2018) Engineering security vulnerability prevention, detection, and response. IEEE Softw 35(5):76–80

    Article  Google Scholar 

  • Zaman S, Adams B, Hassan AE (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories, ser. MSR ’11. ACM, New York, pp 93–102

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Software Engineering, Concordia University, Montreal, QC, Canada

    Sophia Quach, Maxime Lamothe & Weiyi Shang

  2. Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan

    Yasutaka Kamei

Authors
  1. Sophia Quach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maxime Lamothe
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yasutaka Kamei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weiyi Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sophia Quach.

Additional information

Communicated by: Ali Ouni, David Lo, Xin Xia, Alexander Serebrenik and Christoph Treude

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Recommendation Systems for Software Engineering

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Quach, S., Lamothe, M., Kamei, Y. et al. An empirical study on the use of SZZ for identifying inducing changes of non-functional bugs. Empir Software Eng 26, 71 (2021). https://doi.org/10.1007/s10664-021-09970-8

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-09970-8

Keywords

Search

Navigation