Skip to main content

Advertisement

SpringerLink
Log in

An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

To complete tasks faster, developers often have to sacrifice the quality of the software. Such compromised practice results in the increasing burden to developers in future development. The metaphor, technical debt, describes such practice. Prior research has illustrated the negative impact of technical debt, and many researchers investigated how developers deal with a certain type of technical debt. However, few studies focused on the removal of different types of technical debt in practice. To fill this gap, we use the introduction and removal of different types of self-admitted technical debt (i.e., SATD) in 7 deep learning frameworks as an example. This is because deep learning frameworks are some of the most important software systems today due to their prevalent use in life-impacting deep learning applications. Moreover, the field of the development of different deep learning frameworks is the same, which enables us to find common behaviors on the removal of different types of technical debt across projects. By mining the file history of these frameworks, we find that design debt is introduced the most along the development process. As for the removal of technical debt, we find that requirement debt is removed the most, and design debt is removed the fastest. Most of test debt, design debt, and requirement debt are removed by the developers who introduced them. Based on the introduction and removal of different types of technical debt, we discuss the evolution of the frequencies of different types of technical debt to depict the unresolved sub-optimal trade-offs or decisions that are confronted by developers along the development process. We also discuss the removal patterns of different types of technical debt, highlight future research directions, and provide recommendations for practitioners.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://www.tensorflow.org/api_docs/python/tf/contrib/gan

  2. https://www.tensorflow.org/api_docs/python/tf/contrib/tpu

  3. https://mxnet.incubator.apache.org/api/python/symbol/symbo-l.html#mxnet.symbol.BatchNorm

  4. https://blog.github.com/2017-01-31-introducing-topics/

  5. https://github.com/tensorflow/tensorflow

  6. https://github.com/keras-team/keras

  7. https://github.com/deeplearning4j/deeplearning4j

  8. https://github.com/BVLC/caffe

  9. https://github.com/pytorch/pytorch

  10. https://github.com/apache/incubator-mxnet

  11. https://github.com/Microsoft/CNTK

  12. https://dwheeler.com/sloccount/

  13. https://www.srcml.org/

  14. https://docs.python.org/3/library/tokenize.html

  15. tensorflow/tensorflow/core/kernels/cwise_ops.h

  16. tensorflow/tensorflow/python/feature_column/feature_column_test.py

  17. tensorflow/tensorflow/core/kernels/hinge-loss.h

  18. deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/params/Batch-NormalizationParamInitializer.java

  19. tensorflow/tensorflow/python/keras/backend_test.py

  20. caffe/include/caffe/common.hpp

  21. caffe/src/caffe/layers/pooling_layer.cpp

  22. https://www.qasymphony.com/software-testing-tools/qtest-manager/test-case-management/

  23. https://github.com/facebookincubator/gloo

  24. https://github.com/facebookincubator/gloo/commits/1d9e62aff9d7143129a69c8eb23e8351-e686ff3a/gloo/scatter.cc

References

  • Al-Qizwini M, Barjasteh I, Al-Qassab H, Radha H (2017) Deep learning algorithm for autonomous driving using googlenet. In: 2017 IEEE intelligent vehicles symposium (IV). IEEE, pp 89–96

  • Alves NS, Ribeiro LF, Caires V, Mendes TS, Spínola RO (2014) Towards an ontology of terms on technical debt. In: Sixth international workshop on managing technical debt (MTD), 2014. IEEE, pp 1–7

  • Bavota G, Russo B (2016) A large-scale empirical study on self-admitted technical debt. In: IEEE/ACM 13th working conference on mining software repositories (MSR), 2016. IEEE, pp 315–326

  • Brown N, Cai Y, Guo Y, Kazman R, Kim M, Kruchten P, Lim E, MacCormack A, Nord R, Ozkaya I, et al. (2010) Managing technical debt in software-reliant systems. In: Proceedings of the FSE/SDP workshop on future of software engineering research. ACM, pp 47–52

  • Carpenter M (1997) Survival analysis: a self-learning text

  • Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494

    Article  Google Scholar 

  • Cox DR, Oakes D (1984) Analysis of survival data, vol 21. CRC Press

  • Cunningham W (1993) The wycash portfolio management system. ACM SIGPLAN OOPS Messenger 4(2):29–30

    Article  Google Scholar 

  • de Almeida RR, Kulesza U, Treude C, Lima AHG, et al. (2018) Aligning technical debt prioritization with business objectives: a multiple-case study. In: 2018 IEEE international conference on software maintenance and evolution (ICSME), IEEE, pp 655–664

  • Dunn OJ (1961) Multiple comparisons among means. J Am Stat Assoc 56(293):52–64

    Article  MathSciNet  Google Scholar 

  • Ernst NA (2012) On the role of requirements in understanding and managing technical debt. In: Proceedings of the third international workshop on managing technical debt, IEEE Press, pp 61–64

  • Ernst NA, Bellomo S, Ozkaya I, Nord RL, Gorton I (2015) Measure it? Manage it? Ignore it? Software practitioners and technical debt. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering, ACM, pp 50–60

  • Fontana FA, Ferme V, Spinelli S (2012) Investigating the impact of code smells debt on quality code evaluation. In: Proceedings of the third international workshop on managing technical debt, IEEE Press, pp 15–22

  • Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. arXiv:1406.2661 [cs, stat]

  • Huang Q, Shihab E, Xia X, Lo D, Li S (2018) Identifying self-admitted technical debt in open source projects using text mining. Empir Softw Eng 23(1):418–451

    Article  Google Scholar 

  • Huval B, Wang T, Tandon S, Kiske J, Song W, Pazhayampallil J, Andriluka M, Rajpurkar P, Migimatsu T, Cheng-Yue R, et al. (2015) An empirical evaluation of deep learning on highway driving. arXiv:150401716

  • Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 [cs]

  • Islam MJ, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: ESEC/FSE’19: the ACM joint european software engineering conference and symposium on the foundations of software engineering (ESEC/FSE), ESEC/FSE 2019

  • Jouppi NP, Young C, Patil N, Patterson D, Agrawal G, Bajwa R, Bates S, Bhatia S, Boden N, Borchers A, et al. (2017) In-datacenter performance analysis of a tensor processing unit. In: 2017 ACM/IEEE 44th annual international symposium on computer architecture (ISCA), IEEE, pp 1–12

  • Kaplan EL, Meier P (1958) Nonparametric estimation from incomplete observations. J Am Stat Assoc 53(282):457–481

    Article  MathSciNet  Google Scholar 

  • Klinger T, Tarr P, Wagstrom P, Williams C (2011) An enterprise perspective on technical debt. In: Proceedings of the 2nd Workshop on managing technical debt, ACM, pp 35–38

  • Kruchten P, Nord RL, Ozkaya I, Falessi D (2013) Technical debt: towards a crisper definition report on the 4th international workshop on managing technical debt. ACM SIGSOFT Software Engineering Notes 38(5):51–54

    Article  Google Scholar 

  • Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621

    Article  Google Scholar 

  • Li Z, Avgeriou P, Liang P (2015) A systematic mapping study on technical debt and its management. J Syst Softw 101:193–220

    Article  Google Scholar 

  • Lim E, Taksande N, Seaman C (2012) A balancing act: what software practitioners have to say about technical debt. IEEE Softw 29(6):22–27

    Article  Google Scholar 

  • Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  • Liu J, Huang Q, Xia X, Shihab E, Lo D, Li S (2020) Is using deep learning frameworks free? Characterizing technical debt in deep learning frameworks. In: Proceedings of the 42nd ACM/IEEE international conference on software engineering - SE in society (ICSE’20 SEIS), ACM/IEEE

  • Liu Z, Huang Q, Xia X, Shihab E, Lo D, Li S (2018) Satd detector: a text-mining-based self-admitted technical debt detection tool. In: Proceedings of the 40th international conference on software engineering: companion proceedings, ACM, pp 9–12

  • Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, Chen C, Su T, Li L, Liu Y, et al. (2018) Deepgauge: multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 120–131

  • Maldonado E, Shihab E (2015) Detecting and quantifying different types of self-admitted technical debt. In: Proceedings of the 7th IEEE international workshop on managing technical debt (MTD’15), pp 9–15

  • Maldonado E, Abdalkareem R, Shihab E, Serebrenik A (2017) An empirical study on the removal of self-admitted technical debt. In: Proceedings of the 33rd international conference on software maintenance and evolution (ICSME’17), IEEE

  • Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat: 18(1):50–60

    Article  MathSciNet  Google Scholar 

  • Mantel N (1966) Evaluation of survival data and two new rank order statistics arising in its consideration. Cancer Chemother Rep 50:163–170

    Google Scholar 

  • McElreath R (2020) Statistical rethinking: A Bayesian course with examples in R and Stan. CRC Press

  • McHugh ML (2012) Interrater reliability: the kappa statistic. Biochemia Medica: Biochemia Medica 22(3):276–282

    Article  MathSciNet  Google Scholar 

  • Miller RG Jr (2011) Survival analysis, vol 66. Wiley

  • Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th symposium on operating systems principles, ACM, pp 1–18

  • Potdar A, Shihab E (2014) An exploratory study on self-admitted technical debt. In: Proceedings of the 30th IEEE international conference on software maintenance and evolution (ICSME’14), pp 91–100

  • Quesenberry CP Jr, Fireman B, Hiatt RA, Selby JV (1989) A survival analysis of hospitalization among patients with acquired immunodeficiency syndrome. Am J Public Health 79(12):1643–1647

    Article  Google Scholar 

  • Sallab AE, Abdou M, Perot E, Yogamani S (2017) Deep reinforcement learning framework for autonomous driving. Electronic Imaging 2017(19):70–76

    Article  Google Scholar 

  • Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Chaudhary V, Young M, Crespo JF, Dennison D (2015) Hidden technical debt in machine learning systems. In: Advances in neural information processing systems, pp 2503–2511

  • Seaman C, Guo Y (2011) Measuring and monitoring technical debt. In: Advances in Computers, Elsevier, vol 82, pp 25–46

  • Shalev-Shwartz S, Shammah S, Shashua A (2016) Safe, multi-agent, reinforcement learning for autonomous driving. arXiv:161003295

  • Shull F (2011) Perfectionists in a world of finite resources. IEEE Softw 28(2):4–6

    Article  Google Scholar 

  • Spencer D (2009) Card sorting: designing usable categories. Rosenfeld Media

  • Spínola RO, Vetrò A, Zazworka N, Seaman C, Shull F (2013) Investigating technical debt folklore: shedding some light on technical debt opinion. In: 2013 4th international workshop on managing technical debt (MTD), IEEE, pp 1–7

  • Sun Y, Huang X, Kroening D (2018a) Testing deep neural networks. arXiv:180304792

  • Sun Y, Wu M, Ruan W, Huang X, Kwiatkowska M, Kroening D (2018b) Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 109–119

  • Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th international conference on software engineering, ACM, pp 303–314

  • Wehaibi S, Shihab E, Guerrouj L (2016) Examining the impact of self-admitted technical debt on software quality. In: IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), 2016, IEEE, vol 1, pp 179–188

  • Wei L, Lachin J (1984) Two-sample asymptotically distribution-free tests for incomplete multivariate observations. J Am Stat Assoc 79(387):653–661

    Article  MathSciNet  Google Scholar 

  • Zampetti F, Serebrenik A, Di Penta M (2018) Was self-admitted technical debt removal a real removal? An in-depth perspective. In: 2018 IEEE/ACM 15th international conference on mining software repositories (MSR), IEEE, pp 526–536

  • Zampetti F, Serebrenik A, Di Penta M (2020) Automatically learning patterns for self-admitted technical debt removal. In: 2020 IEEE 27th international conference on software analysis, evolution and reengineering (SANER), IEEE, pp 355–366

  • Zazworka N, Shaw MA, Shull F, Seaman C (2011) Investigating the impact of design debt on software quality. In: Proceedings of the 2nd workshop on managing technical debt, ACM, New York, NY, USA, MTD ’11, pp 17–23. https://doi.org/10.1145/1985362.1985366

  • Zazworka N, Spínola RO, Vetro A, Shull F, Seaman C (2013) A case study on effectively identifying technical debt. In: Proceedings of the 17th international conference on evaluation and assessment in software engineering, ACM, New York, NY, USA, EASE ’13. https://doi.org/10.1145/2460999.2461005, pp 42–47

  • Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018a) Deeproad: gan-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 132–142

  • Zhang Y, Chen Y, Cheung SC, Xiong Y, Zhang L (2018b) An empirical study on tensorflow program bugs. International Symposium on Software Testing and Analysis

Download references

Acknowledgements

This research was partially supported by the National Key R&D Program of China (No. 2018YFB1003904) and the Australian Research Council’s Discovery Early Career Researcher Award (DECRA) (DE200100021).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Zhejiang University, Hangzhou, China

    Jiakun Liu, Qiao Huang & Shanping Li

  2. Faculty of Information Technology, Monash University, Melbourne, Australia

    Xin Xia

  3. Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada

    Emad Shihab

  4. School of Information Systems, Singapore Management University, Singapore, Singapore

    David Lo

Authors
  1. Jiakun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiao Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emad Shihab
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Lo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shanping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Xia.

Additional information

Communicated by: Gabriele Bavota

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Huang, Q., Xia, X. et al. An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks. Empir Software Eng 26, 16 (2021). https://doi.org/10.1007/s10664-020-09917-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-020-09917-5

Keywords

Search

Navigation