Software Testing Effort Estimation and Related Problems: A Systematic Literature Review

Authors:
Ilona Bluemke

Warsaw University of Technology, Institute of Computer Science, Warsaw, Poland

Warsaw University of Technology, Institute of Computer Science, Warsaw, Poland
View Profile

,
Agnieszka Malanowska

Warsaw University of Technology, Institute of Computer Science, Warsaw, Poland

Warsaw University of Technology, Institute of Computer Science, Warsaw, Poland
View Profile

Authors Info & Claims

ACM Computing Surveys Volume 54 Issue 3Article No.: 53pp 1–38https://doi.org/10.1145/3442694

Published:17 April 2021Publication History

ACM Computing Surveys

Abstract

Although testing effort estimation is a very important task in software project management, it is rarely described in the literature. There are many difficulties in finding any useful methods or tools for this purpose. Solutions to many other problems related to testing effort calculation are published much more often. There is also no research focusing on both testing effort estimation and all related areas of software engineering. To fill this gap, we performed a systematic literature review on both questions. Although our primary objective was to find some tools or implementable metods for test effort estimation, we have quickly discovered many other interesting topics related to the main one. The main contribution of this work is the presentation of the testing effort estimation task in a very wide context, indicating the relations with other research fields. This systematic literature review presents a detailed overview of testing effort estimation task, including challenges and approaches to automating it and the solutions proposed in the literature. It also exhaustively investigates related research topics, classifying publications that can be found in connection to the testing effort according to seven criteria formulated on the basis of our research questions. We present here both synthesis of our finding and the deep analysis of the stated research problems.

References

Sharma A. Abhilasha. 2013. Test effort estimation in regression testing. In Proceedings of the IEEE International Conference in MOOC Innovation and Technology in Education. 343--348. DOI: https://doi.org/10.1109/MITE.2013.6756364Google ScholarCross Ref
C. Abhishek, V. P. Kumar, H. Vitta, and P. R. Srivastava. 2010. Test effort estimation using neural network. J. Softw. Eng. App. 3, 4 (2010). DOI: https://doi.org/10.4236/jsea.2010.34038Google Scholar
O. E. Adali, N. A. Karagöz, Z. Gürel, T. Tahir, and C. Gencel. 2017. Software test effort estimation: state of the art in Turkish software industry. In Proceedings of the 43rd Euromicro Conference on Software Engineering and Advanced Applications. 412--420, DOI: https://doi.org/10.1109/SEAA.2017.72Google Scholar
W. Afzal, A. N. Ghazi, J. Itkonen, R. Torkar, A. Andrews, and K. Bhatti. 2015. An experiment on the effectiveness and efficiency of exploratory testing. Empir. Softw. Eng. 20, 3 (2015), 844--878. Springer Science + Business Media. DOI: https://doi.org/10.1007/s10664-014-9301-4Google ScholarDigital Library
S. F. Ahmad and P. A. Samat. 2018. Extraction cost of quality and testing in software project. In Proceedings of the IEEE Conference on E-Learning, E-Management and E-Services (IC3e’18). 109--115. DOI: https://doi.org/10.1109/IC3e.2018.8632624Google Scholar
M. Ahmed and R. Ibrahim. 2014. Improving effectiveness of testing using reusability factor. In Proceedings of the International Conference on Computer and Information Science. 1--5. DOI: https://doi.org/10.1109/ICCOINS.2014.6868451Google Scholar
É. R. C. de Almeida, B. T. de Abreu, and R. Moraes. 2009. An alternative approach to test effort estimation based on use cases. In Proceedings of the International Conference on Software Testing, Verification and Validation Workshops. IEEE. 279--288. DOI: https://doi.org/10.1109/ICST.2009.31Google Scholar
S. Aloka, P. Singh, G. Rakshit, and P. R. Srivastava. 2011. Test effort estimation-particle swarm optimization based approach. In Contemporary Computing, S. Aluru et al. (Eds.). Springer-Verlag Berlin. 463--474. DOI: https://doi.org/10.1007/978-3-642-22606-9_46Google Scholar
H. Aman, T. Nakano, H. Ogasawara, and M. Kawahara. 2017. A test case recommendation method based on morphological analysis, clustering and the Mahalanobis-Taguchi method. In Proceedings of the 10th IEEE International Conference on Software Testing, Verification and Validation Workshops. 29--35. DOI: https://doi.org/10.1109/ICSTW.2017.9Google Scholar
A. Ansari, M. B. Shagufta, A. Sadaf Fatima, and S. Tehreem. 2017. Constructing test cases using natural language processing. In Proceedings of the 3rd IEEE International Conference on Advanced in Electrical, Electronics, Information, Communication and Bio-Informatics, P. L. N. Ramesh et al. (Eds.). 95--99. DOI: https://doi.org/10.1109/AEEICB.2017.7972390Google Scholar
E. Aranha and P. Borba. 2007. An estimation model for test execution effort. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement. 107--116. DOI: https://doi.org/10.1109/ESEM.2007.73Google Scholar
E. Aranha and P. Borba. 2007. Empirical studies of test execution effort estimation based on test characteristics and risk factors. Proceedings of the 2nd International Doctoral Symposium on Empirical Software Engineering (IDoESE’07). Retrieved from https://pdfs.semanticscholar.org/2fa8/bc98114c0e32d2e59c1829e1af2b6f20fc7c.pdf.Google Scholar
E. Aranha and P. Borba. 2009. Estimating manual test execution effort and capacity based on execution points. Int. J. Comp. App. 31, 3 (2009), 167--172. ACTA Press. DOI:10.1080/1206212X.2009.11441938Google ScholarCross Ref
E. Aranha and P. Borba. 2007. Test effort estimation models based on test specifications. In Proceedings of the Conference on Testing Academic and Industrial Practice and Research Techniques. 67--71. DOI:10.1109/TAIC.PART.2007.29Google Scholar
E. Aranha and P. Borba. 2019. Test execution effort and capacity estimation. Retrieved from https://pdfs.semanticscholar.org/be9f/b7e4ffd6e9ea536b500f8697558e708a7212.pdf.Google Scholar
E. Aranha, F. de Almeida, T. Diniz, V. Fontes, and P. Borba. 2008. Automated test execution effort estimation based on functional test specifications. Retrieved from https://www.semanticscholar.org/paper/Automated-Test-Execution-Effort-Estimation-Based-on-Aranha-Almeida/a7db957392c3e07a498a1556383abd7822a02c6f.Google Scholar
F. O. de Araujo and L. V. Matieli. 2017. Software testing estimation: Bibliographic survey in Brazilian and international environments. Braz. J.Op. Prod. Manag. 14, 1 (2017), 10--18. DOI:10.14488/BJOPM.2017.v14.n1.a2Google ScholarCross Ref
M. D. Ashraf and N. U. Janjua. 2011. Test execution effort estimation (TEEE) model in extreme programming. Int. J. Rev. Comp. 8 (2011), 35--45. Little Lion Scientific.Google Scholar
M. Badri, L. Badri, and W. Flageol. 2013. On the relationship between use cases and test suites size: An exploratory study. ACM SIGSOFT Softw. Eng. Notes 38, 4 (2013), 1--5. DOI:10.1145/2492248.2492261Google ScholarDigital Library
M. Badri, L. Badri, W. Flageol, and F. Toure. 2017. Investigating the accuracy of test code size prediction using use case metrics and machine learning algorithms: An empirical study. In Proceedings of the International Conference on Machine Learning and Software Computing. 25--33. DOI:10.1145/3036290.‌3036323Google ScholarDigital Library
M. Badri, F. Toure, and L. Lamontagne. 2015. Predicting unit testing effort levels of classes: An exploratory study based on multinomial logistic regression modeling. Procedia Comput. Sci. 62 (2015), 529--538. Elsevier. DOI:10.1016/j.procs.2015.08.528Google ScholarCross Ref
K. Bareja and A. Singhal. 2015. A review of estimation techniques to reduce testing efforts in software development. In Proceedings of the 5th International Conference on Advanced Computing & Communication Technologies. 541--546. DOI:10.1109/ACCT.2015.110Google ScholarDigital Library
T. R. Benala and R. Mall. 2018. DABE: Differential evolution in analogy-based software development effort estimation. Swarm Evol. Comput. 38 (2018), 158--172. Elsevier. DOI:10.1016/j.swevo.2017.07.009Google ScholarCross Ref
A. Bertolino, R. Mirandola, and E. Peciola. 1996. A case study in branch testing automation. In Achieving Quality in Software, S. Bologna et al. (Eds.). Springer Science+Business Media, 369--380. DOI:10.1007/978-0-387-34869-8_30Google Scholar
P. K. Bhatia and G. Kumar. 2011. Role of technical complexity factors in test effort estimation using use case points. Int. J. Softw. Eng. Res. Pract. 1, 3 (2011), 5--12.Google Scholar
P. Bhattacharya, P. R. Srivastava, and B. Prasad. 2012. Software test effort estimation using particle swarm optimization. In Proceedings of the International Conference on Information Systems Design and Intelligent Applications. 827--835. DOI:10.1007/978-3-642-27443-5_95Google Scholar
A. Bhattacharyya and T. Malgazhdarov. 2016. PredSym: Estimating software testing budget for a bug-free release. In Proceedings of the 7th International Workshop on Automating Test Case Design, Selection, and Evaluation. 16--22. DOI:10.1145/2994291.2994294Google ScholarDigital Library
I. Bluemke and A. Malanowska. 2020. Tool for assessment of testing effort. In Proceedings of the 14th International Conference on Dependability of Computer Systems. 69--79. DOI 10.1007/978-3-030-19501-4_7Google Scholar
I. Bluemke and A. Malanowska. 2020. Usage of UML combined fragments in automatic function point analysis. In Proceedings of the 15th International Conference on Evaluation of Novel Approaches to Software Engineering -- Vol. 1: ENASE. 305--312. DOI:10.5220/0009348303050312Google Scholar
F. Bock, S. Siegl, P. Bazan, P. Buchholz, and R. German. 2018. Reliability and test effort analysis of multi-sensor driver assistance systems. J. Syst. Archit. 85--86, 1-13. Elsevier. DOI:10.1016/j.sysarc.2018.01.006Google Scholar
F. Bock, S. Siegl, and R. German. 2017. Analytical test effort estimation for multisensor driver assistance systems. In Proceedings of the 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA’17). 239--246. DOI:10.1109/SEAA.2017.49Google Scholar
F. Bock, S. Siegl, and R. German. 2016. Mathematical test effort estimation for dependability assessment of sensor-based driver assistance systems. In Proceedings of the 42nd Euromicro Conference on Software Engineering and Advanced Applications (SEAA’16). 222--226. DOI:10.1109/SEAA.2016.49Google Scholar
E. Borandag, F. Yucalar, and S. Z. Erdogan. 2016. A case study for the software size estimation through MK II FPA and FP methods. Int. J. Comp. App. Technol. 53. 4 (2016), 309--314.Google Scholar
M. Böhme. 2018. STADS: Software testing as species discovery. ACM Trans. Softw. Eng. Methodol. 27. 2 (2018). DOI:10.1145/3210309Google ScholarDigital Library
F. Calzolari, P. Tonella, and G. Antoniol. 2001. Maintenance and testing effort modeled by linear and nonlinear dynamic systems. Inf. Softw. Technol. 43, 8 (2001), 477--486. DOI:10.1016/S0950-5849(01)00156-2Google ScholarCross Ref
P. Chaudhary and C. S. Yadav. 2012. An approach for calculating the effort needed on testing projects. Int. J. Adv. Res. Comput. Eng. Technol. 1. 1 (2012), 35--40.Google Scholar
P. Chaudhary and C. S. Yadav. 2012. Optimizing test effort estimation-a comparative analysis. Int. J. Sci. Res. Eng. Technol. 1. 2 (2012), 18--20.Google Scholar
R. K. Clemmons. 2006. Project estimation with use case points. CrossTalk: J. Defense Softw. Eng. 19, 2 (2006), 18--22. Retrieved from http://www.royclemmons.com/articles/docs/0602Clemmons.pdf.Google Scholar
International Journal of Engineering Research and Applications (IJERA). 2015. In National Conference on Developments, Advances and Trends in Engineering Sciences (NCDATES'15). CMR Engineering College, 12--19.Google Scholar
G. de Cleva Farto and A. T. Endo. 2017. Reuse of model-based tests in mobile apps. In Proceedings of the 31st Brazilian Symposium on Software Engineering. 184--193. DOI:10.1145/3131151.3131160Google ScholarDigital Library
D. Cotroneo, D. D. Leo, R. Natella, and R. Pietrantuono. 2016. Prediction of the testing effort for the safety certification of open-source software: A case study on a real-time operating system. In Proceedings of the 12th European Dependable Computer Conference. 141--152. DOI:10.1109/EDCC.2016.22Google Scholar
G. Czibula, I. G. Czibula, and Z. Marian. 2018. An effective approach for determining the class integration test order using reinforcement learning. Appl. Soft Comput. 65 (2018), 517--530. Elsevier. DOI:10.1016/j.asoc.2018.01.042Google ScholarDigital Library
C. W. Dawson. 1998. An artificial neural network approach to software testing effort estimation. Trans. Inf. Commun. Technol. 20. WIT Press. DOI:10.2495/AI980361Google Scholar
V. Debroy and W. E. Wong. 2011. On the estimation of adequate test set size using fault failure rates. J. Syst. Softw. 84, 4 (2011), 587--602. Elsevier. DOI:10.1016/j.jss.2010.07.025Google ScholarDigital Library
R. C. G. Dhanajayan and S. A. Pillai. 2017. SLMBC: Spiral life cycle model-based Bayesian classification technique for efficient software fault prediction and classification. Soft Comput. 21, 2 (2017), 403--415. DOI:10.1007/s00500-016-2316-6Google ScholarDigital Library
M. Elallaoui, K. Nafil, R. Touahni, and R. Messoussi. 2016. Automated model driven testing using AndroMDA and UML2 testing profile in scrum process. Procedia Comput. Sci. 83 (2016), 221--228. Elsevier. DOI:10.1016/j.procs.2016.04.119Google ScholarCross Ref
F. Elberzhager, A. Rosbach, J. Münch, and R. Eschbach. 2012. Reducing test effort: A systematic mapping study on existing approaches. Inf. Softw. Technol. 54, 10 (2012), 1092--1106. Elsevier. DOI:10.1016/j.infsof.2012.04.007Google ScholarDigital Library
R. Elghondakly, S. Moussa, and N. Badr. 2016. A comprehensive study for software testing and test cases generation paradigms. In Proceedings of the International Conference on Internet of Things and Cloud Computing, ACM. DOI:10.1145/2896387.2896435Google ScholarDigital Library
A. T. Endo and A. Simao. 2013. Evaluating test suite characteristics, cost, and effectiveness of FSM-based testing methods. Inf. Softw. Technol. 55, 6 (2013), 1045--1062. Elsevier. DOI:10.1016/j.infsof.2013.01.001Google ScholarDigital Library
N. F. Felipe, R. P. Cavalcanti, E. H. B. Maia, W. P. Amaral, A. C. Farnese, L. D. Tavares, E. S. J. de Faria, C. I. P. da Silva e Pádua, and W. de Pádua Paula Filho. 2014. A comparative study of three test effort estimation methods. Rev. Cubana Cienc. Informát. 8, Especial Uciencia (2014), 1--13. Universidad de las Ciencias Informáticas.Google Scholar
J. Ferrer, F. Chicano, and E. Alba. 2013. Estimating software testing complexity. Inf. Softw. Technol. 55, 12 (2013), 2125--2139. Elsevier. DOI:10.1016/j.infsof.2013.07.007Google ScholarDigital Library
L. Fiondella and S. S. Gokhale. 2012. Optimal allocation of testing effort considering software architecture. IEEE Trans. Reliab. 61, 2 (2012), 580--589. DOI:10.1109/TR.2012.2192016Google ScholarCross Ref
D. Flemström, P. Potena, D. Sundmark, W. Afzal, and M. Bohlin. 2018. Similarity-based prioritization of test case automation. Softw. Qual. J. Springer US. DOI:10.1007/s11219-017-9401-7Google ScholarDigital Library
P. Garg. 2011. Simulator to calculate test efforts by using reusability of code. Int. J. Adv. Res. Comput. Sci. 2, 4 (2011) 273--276. Genxcellence Publications. DOI:10.26483/ijarcs.v2i4.622Google Scholar
V. Garousi and M. V. Mäntylä. 2016. A systematic literature review of literature reviews in software testing. Inf. Softw. Technol. 80 (2016), 195--216. Elsevier. DOI:10.1016/j.infsof.2016.09.002Google ScholarDigital Library
G. Gay, A. Rajan, M. Staats, M. Whalen, and M. P. E. Heimdahl. 2016. The effect of program and model structure on the effectiveness of MC/DC test adequacy coverage. ACM Trans. Softw. Eng. Methodol. 25, 3 (2016), 25. DOI:10.1145/2934672Google ScholarDigital Library
Y. Ghanim. 2017. Risk-based test estimation. In Proceedings of the 3rd Africa and Middle East Conference on Software Engineering. 37--42. DOI:10.1145/3178298.3178302Google ScholarDigital Library
A. S. Ghiduk, M. R. Girgis, and M. H. Shehata. 2017. Higher order mutation testing: A systematic literature review. Comput. Sci. Rev. 25 (2017), 29--48. Elsevier. DOI:10.1016/j.cosrev.2017.06.001Google ScholarCross Ref
R. Göldner. 2001. A cost-benefit model for software testing. In Software Quality: State of the Art in Management, Testing, and Tools. M. Wieczorek et al. (Eds.). 126--134. DOI:10.1007/978-3-642-56529-8_9Google Scholar
G. Grano, F. Palomba, and H. C. Gall. 2019. Lightweight assessment of test-case effectiveness using source-code-quality indicators. IEEE Trans. Softw. Eng. DOI:10.1109/TSE.2019.2903057.Google Scholar
A. Groce, J. Holmes, and K. Kellar. 2017. One test to rule them all. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 1--11. DOI:10.1145/3092703.3092704Google ScholarDigital Library
M. Grover and P. K. Bhatia. 2016. Analytical review of software testing effort estimation using case point method. In Proceedings of the 6th International Conference on Advanced Computing & Communication Technologies (ACCT’16). 451--455. DOI:10.3850/978-981-11-0783-2 451Google Scholar
M. Grover, P. K. Bhatia, and H. Mittal. 2017. Estimating software test effort based on revised UCP model using fuzzy technique. In Proceedings of the Conference on Information and Communication Technologies for Intelligent Systems (ICTIS’17), S. C. Satapathy et al. (Eds.). 490--498. DOI:10.1007/978-3-319-63673-3_59Google Scholar
A. Gupta and P. Jalote. 2008. An approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing. Int. J. Softw. Tools Technol. Transf. 10, 2 (2008), 145--160. DOI:10.1007/s10009-007-0059-5Google ScholarCross Ref
A. Gupta and D. S. Kushwaha. 2016. Use case-based software change analysis and reducing regression test effort. In Proceedings of the International Congress on Information and Communication Technology (ICICT’15). 459--466. DOI:10.1007/978-981-10-0767-5_48Google Scholar
R. Harrison and L. G. Samaraweera. 1996. Using test case metrics to predict code quality and effort. ACM SIGSOFT Softw. Eng. Notes 21, 5 (1996), 78--88. DOI:10.1145/235969.235993Google ScholarDigital Library
M. M. Hassan, W. Afzal, B. Lindström, S. M. A. Shah, S. F. Andler, and M. Blom. 2016. Testability and software performance: A systematic mapping study. In Proceedings of the 31st Annual ACM Symposium on Applied Computing. 1566--1569. DOI:10.1145/2851613.2851978Google ScholarDigital Library
N. E. Holt, L. C. Briand, and R. Torkar. 2014. Empirical evaluations on the cost-effectiveness of state-based testing: An industrial case study. Inf. Softw. Technol. 56, 8 (2014), 890--910. Elsevier. DOI:10.1016/j.infsof.2014.02.011Google ScholarCross Ref
C. Y. Huang, S. Y. Kuo, and M. R. Lyu. 2007. An assessment of testing-effort dependent software reliability growth models. IEEE Trans. Reliab. 56, 2 (2007), 198--211. DOI:10.1109/TR.2007.895301Google ScholarCross Ref
A. Idri, M. Hosni, and A. Abran. 2016. Improved estimation of software development effort using classical and fuzzy analogy ensembles. Appl. Soft Comput. 49 (2016), 990--1019. Elsevier. DOI:10.1016/j.asoc.2016.08.012Google ScholarDigital Library
S. Islam, B. B. Pathik, M. H. Khan, and M. M. Habib. 2014. A novel tool for reducing time and cost at software test estimation: A use cases and functions based approach. In Proceedings of the IEEE International Conference on Industrial Engineering and Engineering Management. 312--316. DOI:10.1109/IEEM.2014.7058650Google Scholar
S. Islam, B. B. Pathik, M. H. Khan, and M. Habib. 2016. Software test estimation tool: Comparable with COCOMOII model. In Proceedings of the International Conference on Industrial Engineering and Engineering Management. 204--208. DOI:10.1109/IEEM.2016.7797865Google Scholar
S. Islam, B. B. Pathik, M. H. Khan, and M. Habib. 2013. Software test estimation tools using use cases and functions. In Proceedings of the IEEE International Conference on Industrial Engineering and Engineering Management. 390--394. DOI:10.1109/IEEM.2013.6962440Google Scholar
K. R. Jayakumar and A. Abran. 2013. A survey of software test estimation techniques. J. Softw. Eng. Appl. 6, 10A (2013), 47--52. DOI:10.4236/jsea.2013.610A006Google ScholarCross Ref
K. R. Jayakumar and A. Abran. 2017. Estimation models for software functional test effort. J. Softw. Eng. Appl. 10, 4 (2017), 338--353. DOI:10.4236/jsea.2017.104020Google ScholarCross Ref
L. X. Jiang, W. J. Han, C. C. Yan, and B. Y. Shi. 2012. Research on size estimation model for software system test based on testing steps and its application. In Proceedings of the International Conference on Computer Science and Information Processing. 1245--1248. DOI:10.1109/CSIP.2012.6309085Google Scholar
P. Jodpimai, P. Sophatsathit, and C. Lursinsap. 2018. Re-estimating software effort using prior phase efforts and data mining techniques. Innov. Syst. Softw. Eng. 14, 3 (2018), 209--228. DOI:10.1007/s11334-018-0311-zGoogle ScholarDigital Library
P. Johri, M. Nasar, S. Das, and M. Kumar. 2016. Open source software reliability growth models for distributed environment based on component-specific testing-efforts. In Proceedings of the 2nd International Conference on Information and Communication Technology for Competitive Strategies. 75. DOI:10.1145/2905055.2905283Google ScholarDigital Library
L. P. Kafle. 2014. An empirical study on software test effort estimation. Int. J. Softw. Comput. Artif. Intell. 2, 2 (2014), 96--106. Institute of Research and Journals.Google Scholar
S. Kappler. 2016. Finding and breaking test dependencies to speed up test execution. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 1136--1138. DOI:10.1145/2950290.2983974Google ScholarDigital Library
P. K. Kapur, P. S. Grover, and S. Younes. 1994. Modelling an imperfect debugging phenomenon with testing effort. In Proceedings of the 5th International Symposium on Software Reliability Engineering. 178--183. DOI:10.1109/ISSRE.1994.341371Google Scholar
P. K. Kapur, G. Mishra, and A. K. Shrivastava. 2016. A generalized framework for modelling multi up-gradations of a software with testing effort and change point. In Proceedings of the International Conference on Innovation and Challenges in Cyber Security. 129--134. DOI:10.1109/ICICCS.2016.7542348Google Scholar
G. Karner. 1993. Resource estimation for objectory projects. Objective Systems SF AB. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.604.7842&rep==rep1&type==pdf.Google Scholar
N. Kashyap, P. Vats, and M. Mandot. 2017. AVINASH—A three tier architectural metric suit for the effort estimation in testing of OOS. In Proceedings of the International Conference on Intelligent Communication and Computer Techniques (ICCT’17). 36--41. DOI:10.1109/INTELCCT.2017.8324017Google Scholar
A. Kaur and K. Kaur. 2019. A COSMIC function points based test effort estimation model for mobile applications. J. King Saud Univ. Comput. Inf. Sci. Elsevier. DOI:10.1016/j.jksuci.2019.03.001Google ScholarDigital Library
A. Kaur and K. Kaur. 2019. Investigation on test effort estimation of mobile applications: Systematic literature review and survey. Inf. Softw. Technol. 110 (2019), 56--77. Elsevier. DOI:10.1016/j.infsof.2019.02.003Google ScholarDigital Library
A. Kaur and K. Kaur. 2018. Systematic literature review of mobile application development and testing effort estimation. J. King Saud Univ. Comput. Inf. Sci. Elsevier. DOI:10.1016/j.jksuci.2018.11.002Google ScholarDigital Library
A. M. Kazerouni, C. A. Shaffer, S. H. Edwards, and F. Servant. 2019. Assessing incremental testing practices and their impact on project outcomes. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE ’19). ACM, New York, 407--413. DOI:10.1145/3287324.3287366Google ScholarDigital Library
S. K. Khatri, D. Kumar, A. Dwivedi, and N. Mrinal. 2012. Software reliability growth model with testing effort using learning function. In Proceedings of the 6th International Conference on Software Engineering. 1--5. DOI:10.1109/CONSEG.2012.6349470Google Scholar
P. Khurana, A. Tripathi, and D. S. Kushwaha. 2013. Change impact analysis and its regression test effort estimation. In Proceedings of the 3rd IEEE International Advance Computer Conference (IACC’13). 1420--1424. DOI:10.1109/IAdCC.2013.6514435Google Scholar
B. Kitchenham. 2004. Procedures for Performing Systematic Reviews. Joint Technical Report, Software Engineering Group, Dept. of Computer Science, Keele University. Retrieved from http://www.inf.ufsc.br/&sim;aldo.vw/kitchenham.pdf.Google Scholar
Y. Koroglu, A. Sen, D. Kutluay, A. Bayraktar, Y. Tosun, M. Cinar, and H. Kaya. 2016. Defect prediction on a legacy industrial software: A case study on software with few defects. In Proceedings of the 4th International Workshop on Conducting Empirical Studies in Industry. ACM, 14--20. DOI:10.1145/2896839.2896843Google ScholarDigital Library
F. Křikava and J. Vitek. 2018. Tests from traces: Automated unit test extraction for R. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 232--241. DOI:10.1145/3213846.3213863Google ScholarDigital Library
A. Kumar and V. Beniwal. 2012. Test effort estimation with & without stub and driver using test point analysis (TPA). Int. J. Eng. Res. Technol. 1, 7 (2012). ISSN: 2278--0181.Google Scholar
G. Kumar and P. K. Bhatia. 2013. Software testing optimization through test suite reduction using fuzzy clustering. CSI Trans. ICT 1, 3 (2013), 253--260. DOI:10.1007/s40012-013-0023-3Google ScholarCross Ref
L. Kumar and A. Sureka. 2018. Feature selection techniques to counter class imbalance problem for aging related bug prediction: Aging related bug prediction. In Proceedings of the 11th Innovations in Software Engineering Conference. ACM. 2. DOI:10.1145/3172871.3172872Google ScholarDigital Library
P. Kumari, N. Bakshi, and Y. Pathania. 2015. Test effort estimation and its techniques. Int. J. Technol. Res. Eng. 2, 8 (2015), 1514--1516.Google Scholar
P. Kumari and G. Kaur. 2015. A hybrid firefly-water wave algorithm for effort estimation of software testing. Int. J. Comput. Sci. Netw. 4. 4 (2015), 625--631. ISSN: 2277-5420.Google Scholar
S. Y. Kuo, C. Y. Huang, and M. R. Lyu. 2001. Framework for modeling software reliability, using various testing-efforts and fault-detection rates. IEEE Trans. Reliab. 50, 3 (2001), 310--320. DOI:10.1109/24.974129Google ScholarCross Ref
D. S. Kushwaha and A. K. Misra. 2008. Software test effort estimation. ACM SIGSOFT Softw. Eng. Notes 33, 3 (2008), 6. DOI:10.1145/1360602.1361211Google ScholarDigital Library
R. Lachmann, M. Felderer, M. Nieke, S. Schulze, C. Seidl, and I. Schaefer. 2017. Multi-objective black-box test case selection for system testing. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 1311--1318. DOI:10.1145/3071178.3071189Google ScholarDigital Library
L. Lazić and N. Mastorakis. 2009. The COTECOMO: COnstractive test effort COst model. In Proceedings of the European Computer Conference. 89--110. DOI:10.1007/978-0-387-85437-3_9Google Scholar
X. Li, M. Xie, and S. H. Ng. 2010. Sensitivity analysis of release time of software reliability models incorporating testing effort with multiple change-points. Appl. Math. Model. 34, 11 (2010), 3560--3570. Elsevier. DOI:10.1016/j.apm.2010.03.006Google ScholarCross Ref
J. H. Lo. 2005. An algorithm to allocate the testing-effort expenditures based on sensitive analysis method for software module systems. In Proceedings of the IEEE Region 10 Conference (TENCON’05). 1--6. DOI:10.1109/TENCON.2005.301151Google ScholarCross Ref
A. Malanowska. 2017. Testing Effort Assessment. BSc thesis. Warsaw University of Technology, Institute of Computer Science (in Polish).Google Scholar
A. Malanowska. 2019. Improving Testing Effort Estimation Method with UML Combined Fragments and ISO/IEC 25010:2011 Software Quality Model Support. MSc thesis. Warsaw University of Technology, Institute of Computer Science (in Polish).Google Scholar
A. Malanowska and I. Bluemke. 2020. ISO 25010 support in test point analysis for testing effort estimation. In Integrating Research and Practice in Software Engineering, S. Jarzabek et al. (Eds.). 209--222. DOI:10.1007/978-3-030-26574-8_15Google Scholar
M. Marré and A. Bertolino. 1996. Reducing and estimating the cost of test coverage criteria. In Proceedings of the 18th International Conference on Software Engineering. 486--494. ISBN: 0-8186-7246-3.Google Scholar
A. L. Martins and A. C. V. de Melo. 2016. Can you certify your software to MC/DC?: A static analysis approach to account for the number test cases. In Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing. ACM. DOI:10.1145/2993288.2993290Google ScholarDigital Library
S. Mensah, J. Keung, K. E. Bennin, and M. F. Bosu. 2016. Multi-objective optimization for software testing effort estimation. In Proceedings of the 28th International Conference on Software Engineering & Knowledge Engineering. 527--530. DOI:10.18293/SEKE2016-017Google Scholar
S. Mensah, J. Keung, M. F. Bosu, and K. E. Bennin. 2018. Duplex output software effort estimation model with self-guided interpretation. Inf. Softw. Technol. 94 (2018), 1--13. Elsevier. DOI:10.1016/j.infsof.2017.09.010Google ScholarDigital Library
D. P. P. Mesquita, L. S. Rocha, J. P. P. Gomes, and A. R. R. Neto. 2016. Classification with reject option for software defect prediction. Appl. Soft Comput. Elsevier. 49 1085--1093. DOI:10.1016/j.asoc.2016.06.023Google ScholarDigital Library
B. Miranda and A. Bertolino. 2017. Scope-aided test prioritization, selection and minimization for software reuse. J. Syst. Softw. 131 (2017), 528--549. Elsevier. DOI:10.1016/j.jss.2016.06.058Google ScholarDigital Library
O. Mizuno, E. Shigematsu, Y. Takagi, and T. Kikuno. 2002. On estimating testing effort needed to assure field quality in software development. In Proceedings of the 13th International Symposium on Software Reliability Engineering. IEEE. 139--146. DOI:10.1109/ISSRE.2002.1173234Google Scholar
N. A. Moketar, M. Kamalrudin, S. Sidek, M. Robinson, and J. Grundy. 2016. TestMEReq: Generating abstract tests for requirements validation. In Proceedings of the 3rd International Workshop on Software Engineering Research and Industrial Practice. ACM. 39--45. DOI:10.1145/2897022.2897031Google ScholarDigital Library
R. Morales, A. Sabane, P. Musavi, F. Khomh, F. Chicano, and G. Antoniol. 2016. Finding the best compromise between design quality and testing effort during refactoring. In Proceedings of the 23rd International Conference on Software Analysis, Evolution, and Reengineering. IEEE. 24--35. DOI:10.1109/SANER.2016.23Google Scholar
S. Nageswaran. 2001. Test effort estimation using use case points. In Proceedings of the Quality Week Conference. Retrieved from http://www.bfpug.com.br/Artigos/UCP/Nageswaran-Test_Effort_Estimation_Using_UCP.pdf.Google Scholar
M. Nasar and P. Johri. 2016. Testing resource allocation for fault detection process. In Smart Trends in Information Technology and Computer Communications. A. Unal et al. (Eds.). 683--690. DOI:10.1007/978-981-10-3433-6_82Google Scholar
G. C. Ndem, A. Tahir, A. Ulrich, and H. Goetz. 2011. Test data to reduce the complexity of unit test automation. In Proceedings of the 6th International Workshop on Automation of Software Testing. 105--106. DOI:10.1145/1982595.1982618Google ScholarDigital Library
V. Nguyen, V. Pham, and V. Lam. 2013. qEstimation: A process for estimating size and effort of software testing. In Proceedings of the International Conference on Software and System Processing. ACM. 20--28. DOI:10.1145/2486046.2486052Google ScholarDigital Library
H. Okamura, Y. Etani, and T. Dohi. 2011. Quantifying the effectiveness of testing efforts on software fault detection with a logit software reliability growth model. In Proceedings of the Joint Conference of the 21st International Workshop on Software Measurement and the 6th International Conference on Software Process and Product Measurement. 62--68. DOI:10.1109/IWSM-MENSURA.2011.26Google ScholarDigital Library
F. Palomo-Lozano, A. Estero-Botaro, I. Medina-Bulo, and M. Núñez. 2018. Test suite minimization for mutation testing of WS-BPEL compositions. In Proceedings of the Genetic and Evolutionary Computation Conference, H. Aguirre (ed.). 1427--1434. DOI:10.1145/3205455.3205533Google ScholarDigital Library
A. K. Pandey and N. K. Goyal. 2010. Test effort optimization by prediction and ranking of fault-prone software modules. In Proceedings of the 2nd International Conference on Reliability, Safety and Hazard, P. V. Varde et al. (Eds.). 136--142. DOI:10.1109/ICRESH.2010.5779531Google Scholar
R. M. Parizi, A. A. A. Ghani, S. P. Lee, and S. U. R. Khan. 2017. RAMBUTANS: Automatic AOP-specific test generation tool. Int. J. Softw. Tools Technol. Transf. 19, 6 (2017), 743--761. DOI:10.1007/s10009-016-0432-3Google ScholarDigital Library
A. W. M. M. Parvez. 2013. Efficiency factor and risk factor based user case point test effort estimation model compatible with agile software development. In Proceedings of the International Conference on Information Technology and Electrical Engineering (ICITEE’13). IEEE. 113--118. DOI:10.1109/ICITEED.2013.6676222Google ScholarCross Ref
M. J. Pasha, S. Ranjitha, and H. N. Suresh. 2015. Testing-effort function for debugging in software systems and soft computing model. In Proceedings of the International Conference on Green Computing and Internet of Things. 913--919. DOI:10.1109/ICGCIoT.2015.7380593Google ScholarDigital Library
N. Patel, M. Govindrajan, S. Maharana, and S. Ramdas. 2001. Test Case Point Analysis: White Paper. Cognizant Technology Solutions. Retrieved from https://www.cmcrossroads.com/sites/default/files/article/file/2013/XUS373692file1_0.pdf.Google Scholar
R. Peng, Q. P. Hu, S. H. Ng, and M. Xie. 2010. Testing effort dependent software FDP and FCP models with consideration of imperfect debugging. In Proceedings of the 4th International Conference on Secure Software Integration and Reliability Improvement 141--146. DOI:10.1109/SSIRI.2010.13Google ScholarDigital Library
R. Peng, Y. F. Li, W. J. Zhang, and Q. P. Hu. 2014. Testing effort dependent software reliability model for imperfect debugging process considering both detection and correction. Reliab. Eng. Syst. Safe. 126 (2014), 37--43. DOI:10.1016/j.ress.2014.01.004Google ScholarCross Ref
K. Periyasamy and X. Liu. 1999. A new metrics set for evaluating testing efforts for object-oriented programs. In Proceedings of of the Conference on Technology of Object-Oriented Languages and Systems (TOOLS’99), D. Firesmith (Eds.). 84--93. DOI:10.1109/TOOLS.1999.787538Google Scholar
I. Pinkster, B. van de Burgt, D. Janssen, and E. van Veenendaal. 2004. Successful test management: An integral approach. Estimation 85--112. Springer Science+Business Media. DOI:10.1007/978-3-540-44735-1_6Google Scholar
S. Ramacharan and K. V. G. Rao. 2016. Software effort estimation of GSD projects using calibrated parametric estimation models. In Proceedings of the 2nd International Conference on Information and Communication Technology for Competitive Strategies. DOI:10.1145/2905055.2905177Google ScholarDigital Library
R. Ramler, C. Salomon, G. Buchgeher, and M. Lusser. 2017. Tool support for change-based regression testing: An industry experience report. In Software Quality: Complexity and Challenges of Software Engineering in Emerging Technologies, D. Winkler (Eds.). 133--152. DOI:10.1007/978-3-319-49421-0_10Google Scholar
S. S. Rathore and S. Kumar. 2017. Towards an ensemble based system for predicting the number of software faults. Exp. Syst. Appl. 82 (2017), 357--382. DOI:10.1016/j.eswa.2017.04.014Google ScholarDigital Library
M. Reider, S. Magnus, and J. Krause. 2018. Feature-based testing by using model synthesis, test generation and parameterizable test prioritization. In Proceedings of the IEEE 11th International Conference on Software Testing, Verification and Validation Workshops 130--137. DOI:10.1109/ICSTW.2018.00041Google Scholar
A. Roman. 2015. Testowanie i jakość oprogramowania: Modele, techniki, narzędzia, Warszawa, Wydawnictwo Naukowe PWN, 2015, ISBN: 978-83-01-18160-4 (in Polish).Google Scholar
D. S. Rosenblum and E. J. Weyuker. 1996. Predicting the cost-effectiveness of regression testing strategies. In Proceedings of the 4th ACM SIGSOFT Symposiumon Foundations of Software Engineering, D. Garlan (ed.). 118--126. DOI:10.1145/239098.239118Google ScholarDigital Library
A. Rosenfeld, O. Kardashov, and O. Zang. 2018. Automation of Android applications functional testing using machine learning activities classification. In Proceedings of the 5th International Conference on Mobile Software Engineering and Systems. 122--132. DOI:10.1145/3197231.3197241Google ScholarDigital Library
A. Sadeghi, R. Jabbarvand, and S. Malek. 2017. PATDroid: Permission-aware GUI testing of Android. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. 220--232. DOI:10.1145/3106237.3106250Google ScholarDigital Library
A. Saeed, W. H. Butt, F. Kazmi, and M. Arif. 2018. Survey of software development effort estimation techniques. In Proceedings of the 7th International Conference on Software and Computer Applications. 82--86. DOI:10.1145/3185089.3185140Google ScholarDigital Library
P. Sahoo, J. R. Mohanty, and D. Sahoo. 2018. Early system test effort estimation automation for object-oriented systems. In Proceedings of the 6th International Conference on Frontiers of Intelligent Computing (FICTA’18). 325--333. DOI:10.1007/978-981-10-7563-6_34Google Scholar
P. Sahoo and J. R. Mohanty. 2017. Early test effort prediction using UML diagrams. Indon. J. Electric. Eng. Comput. Sci. 5, 1 (2017), 220--228. Institute of Advanced Engineering and Science. DOI:10.11591/ijeecs.v5.i1.pp220-228Google Scholar
M. Salmanoglu, T. Hacaloglu, and O. Demirors. 2017. Effort estimation for agile software development: Comparative case studies using COSMIC functional size measurement and story points. In Proceedings of the 27th International Workshop on Software Measurement and 12th International Conference on Software Process and Product Measurement. 41--49. DOI:10.1145/3143434.3143450Google ScholarDigital Library
S. Shamshiri, J. M. Rojas, J. P. Galeotti, N. Walkinshaw, and G. Fraser. 2018. How do automatically generated unit tests influence software maintenance? In Proceedingss of the 11th International Conference on Software Testing, Verification and Validation. 250--261. DOI:10.1109/ICST.2018.00033Google Scholar
A. Sharma and D. S. Kushwaha. 2011. A metric suite for early estimation of software testing effort using requirement engineering document and its validation. In Proceedings of the 2nd International Conference on Computer and Communications Technology. 373--378. DOI:10.1109/ICCCT.2011.6075150Google Scholar
A. Sharma and D. S. Kushwaha. 2013. An empirical approach for early estimation of software testing effort using SRS document. CSI Trans. ICT 1, 1 (2013), 51--66. DOI:10.1007/s40012-012-0003-zGoogle ScholarCross Ref
A. Sharma and D. S. Kushwaha. 2012. Applying requirement based complexity for the estimation of software development and testing effort. ACM SIGSOFT Softw. Eng. Notes 37, 1 (2012), 1--11. DOI:10.1145/2088883.2088898Google ScholarDigital Library
E. Shihab, Y. Kamei, B. Adams, and A. E. Hassan. 2013. Is lines of code a good measure of effort in effort-aware models? Inf. Softw. Technol. 55, 11 (2013), 1981--1993. DOI:10.1016/j.infsof.2013.06.002Google ScholarDigital Library
T. Shippey, D. Bowes, and T. Hall. 2018. Automatically identifying code features for software defect prediction: Using AST N-grams. Inf. Softw. Technol. DOI:10.1016/j.infsof.2018.10.001Google Scholar
D. G. de Silva, B. T. de Abreu, and M. Jino. 2009. A simple approach for estimation of execution effort of functional test cases. In Proceedings of the 2nd International Conference on Software Testing, Verification and Validation (ICST’09). 289--298. DOI:10.1109/ICST.2009.47Google ScholarDigital Library
T. Silva-de-Souza and G. H. Travassos. 2017. Observing effort factors in the test design & implementation process of web services projects. In Proceedings of the 2nd Brazilian Symposium on Systematic and Automated Software Testing. DOI:10.1145/3128473.3128480Google ScholarDigital Library
Y. Singh, A. Kaur, and B. Suri. 2008. An empirical study of product metrics in software testing. In Innovative Techniques in Instruction Technology, E-learning, E-assessment, and Education. M. Iskander (ed.). 64--72. DOI:10.1007/978-1-4020-8739-4_12Google Scholar
H. M. Sneed. 2018. Requirement-based testing—Extracting logical test cases from requirement documents. In Software Quality: Methods and Tools for Better Software and Systems, D. Winkler et al. (Eds.). 60--79. DOI:10.1007/978-3-319-71440-0_4Google Scholar
H. Srikanth, C. Hettiarachchi, and H. Do. 2016. Requirements based test prioritization using risk factors: An industrial study. Inf. Softw. Technol. 69 (2016), 71--83. DOI:10.1016/j.infsof.2015.09.002Google ScholarDigital Library
P. R. Srivastava. 2009. Estimation of software testing effort: An intelligent approach. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE’09). https://www.researchgate.net/publication/235799428_Estimation_of_Software_Testing_Effort_An_intelligent_Approach.Google Scholar
P. R. Srivastava. 2015. Estimation of software testing effort using fuzzy multiple linear regression. Int. J. Softw. Eng. Technol. Appl. 1, 2/3/4 (2015), 145--154. DOI:10.1504/IJSETA.2015.075602Google Scholar
P. R. Srivastava, A. Bidwai, A. Khan, K. Rathore, R. Sharma, and X. S. Yang. 2014. An empirical study of test effort estimation based on bat algorithm. Int. J. Bio-Insp. Comput. 6, 1 (2014), 57--70. DOI:10.1504/IJBIC.2014.059966Google ScholarDigital Library
P. R. Srivastava, S. Kumar, A. P. Singh, and G. Raghurama. 2011. Software testing effort: An assessment through fuzzy criteria approach. J. Uncert. Syst. 5, 3 (2011), 183--201. ISSN: 1752-8909.Google Scholar
P. R. Srivastava, A. Varshney, P. Nama, X. S. Yang. 2012. Software test effort estimation: A model based on cuckoo search. Int. J. Bio-Insp. Comput. 4, 5 (2012), 278--285. DOI:10.1504/IJBIC.‌2012.049888Google ScholarCross Ref
Nanda S. Suharjito and B. Soewito. 2016. Modeling software effort estimation using hybrid PSO-ANFIS. In Proceedings of the International Seminar on Intelligent Technology and Its Applications. 219--224. DOI:10.1109/ISITIA.2016.7828661Google Scholar
R. T. Sundari. 2008. TCPA—Tool to test effort estimation. Retrieved from https://pdfs.semanticscholar.org/7ebf/62d5d9cc52f0acec5530212f557d50c1bca2.pdf.Google Scholar
S. Tahvili, W. Afzal, M. Saadatmand, M. Bohlin, and S. H. Ameerjan. 2018. ESPRET: A tool for execution time estimation of manual test cases. J. Syst. Softw. 146 (2018), 26--41. DOI:10.1016/j.jss.2018.09.003Google ScholarCross Ref
S. Tahvili, M. Saadatmand, M. Bohlin, W. Afzal, and S. H. Ameerjan. 2017. Towards execution time prediction for manual test cases from test specification. In Proceedings of the 43rd Euromicro Conference on Software Engineering and Advanced Applications. 421--425. DOI:10.1109/SEAA.2017.10Google Scholar
S. Tahvili, M. Saadatmand, S. Larsson, W. Afzal, M. Bohlin, and D. Sundmark. 2016. Dynamic integration test selection based on test case dependencies. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation Workshops 277--286. DOI:10.1109/ICSTW.2016.14Google Scholar
M. Thirasakthana and S. Kiattisin. 2018. Identifying standard testing time for estimation improvement in IT project management. In Proceedings of the 3rd Technology Innovation Management and Engineering Science International Conference (TIMES-iCON’18). 1--5. DOI:10.1109/TIMES-iCON.2018.8621787Google Scholar
F. Toure, M. Badri, and L. Lamontagne. 2018. Predicting different levels of the unit testing effort of classes using source code metrics: A multiple case study on open-source software. Innov. Syst. Softw. Eng. 14, 1 (2018), 15--46. DOI:10.1007/s11334-017-0306-1Google ScholarDigital Library
S. N. Umar. 2013. Software testing effort estimation with Cobb-Douglas function: A practical application. Int. J. Res. Eng. Technol. 2, 5 (2013), 750--754. DOI:10.15623/ijret.2013.0205003Google ScholarCross Ref
E. P. W. M. van Veenendaal and T. Dekkers. 1999. Testpointanalysis: A method for test estimation. In Project Control for Software Quality. R. Kusters et al. (Eds.). Retrieved from http://www.erikvanveenendaal.nl/NL/files/Testpointanalysis%20a%20method%20for%20test%20estimation.pdf.Google Scholar
K. Wang, C. Zhu, A. Celik, J. Kim, D. Batory, and M. Gligoric. 2018. Towards refactoring-aware regression test selection. In Proceedings of the 40th International Conference on Software Engineering. 233--244. DOI:10.1145/3180155.3180254Google ScholarDigital Library
Y. Wang, L. Wang, Y. Li, and X. Zhu. 2017. Time distribution of software stage effort. In Proceedings of the 24th Asia-Pacific Software Engineering Conference Workshops. 41--49. DOI:10.1109/APSECW.2017.9Google Scholar
Y. Wang, Z. Zhu, B. Yang, F. Guo, and H. Yu. 2018. Using reliability risk analysis to prioritize test cases. J. Syst. Softw. 139 (2018), 14--31. DOI:10.1016/j.jss.2018.01.033Google ScholarDigital Library
A. C. Y. Wong, S. T. Chanson, S. C. Cheung, and H. Fuchs. 1997. A framework for distributed object-oriented testing. In Formal Description Techniques and Protocol Specification, Testing and Verification, T. Mizuno et al. (Eds.). 39--56. DOI:10.1007/978-0-387-35271-8_3Google Scholar
H. Wu, C. Nie, and F. C. Kuo. 2016. The optimal testing order in the presence of switching cost. Inf. Softw. Technol. 80 (2016), 57--72. DOI:10.1016/j.infsof.2016.08.006Google ScholarDigital Library
D. K. Yadav and S. Dutta. 2016. Test case prioritization technique based on early fault detection using fuzzy logic. In Proceedings of the 10th 3rd International Conference on Computing for Sustainable Global Development (INDIACom’16). 1033--1036.Google Scholar
S. Yamada, J. Hishitani, and S. Osaki. 1993. Software-reliability growth with a Weibull test-effort: A model & application. IEEE Trans. Reliab. 42, 1 (1993), 100--106. DOI:10.1109/24.210278Google ScholarCross Ref
S. Yamada, H. Ohtera, and H. Narihisa. 1986. Software reliability growth models with testing-effort. IEEE Trans. Reliab. 35, 1 (1986), 19--23. DOI:10.1109/TR.1986.4335332Google ScholarCross Ref
M. Yan, Y. Fang, D. Lo, X. Xia, and X. Zhang. 2017. File-level defect prediction: Unsupervised vs. supervised models. In Proceedings of the 11th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 344--353. DOI:10.1109/ESEM.2017.48Google ScholarDigital Library
J. Yang, J. Chen, W. Hu, and Z. Deng. 2017. Web-based software reliability growth modelling for mobile applications. In Proceedings of the 14th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP’17). 342--346. DOI:10.1109/ICCWAMTIP.2017.8301510Google Scholar
J. Yang. 2017. Estimating effort of test automation projects. In uTest, Applause App Quality. Retrieved from https://www.utest.com/articles/estimating-effort-of-test-automation-projects.Google Scholar
J. Yang, R. Wang, Z. Deng, and W. Hu. 2011. Web software reliability analysis with yamada exponential testing-effort. In Proceedings of the 9th International Conference on Reliability, Maintainability and Safety. 760--765. DOI:10.1109/ICRMS.2011.5979367Google Scholar
H. Younessi, P. Zeephongsekul, and W. Bodhisuwan. 2002. A general model of unit testing efficacy. Softw. Qual. J. 10, 1 (2002), 69--92. DOI:10.1023/A:1015724900702Google ScholarDigital Library
C. M. Zapata-Jaramillo and D. M. Torres-Ricaurte. 2014. Test effort: A pre-conceptual-schema-based representation. DYNA 81, 186 (2014), 132--137. National University of Colombia, Medellín, Mines Faculty. DOI:10.15446/dyna.v81n186.39753Google ScholarCross Ref
X. Zhu, B. Zhou, L. Hou, J. Chen, and L. Chen. 2008. An experience-based approach for test execution effort estimation. In Proceedings of the 9th International Conference for Young Computer Scientists. 1193--1198. DOI:10.1109/ICYCS.2008.53Google ScholarDigital Library
X. Zhu, B. Zhou, F. Wang, Y. Qu, and L. Chen. 2008. Estimate test execution effort at an early stage: An empirical study. In Proceedings of the International Conference on Cyberworlds. 195--200. DOI:10.1109/CW.2008.34Google ScholarDigital Library
F. Zou and R. Xu. 2004. Empirical measurement of the software testing and reliability. Wuhan Univ. J. Nat. Sci. 9, 1 (2004), 23--26. DOI:10.1007/BF02912711Google ScholarCross Ref

Index Terms

Software Testing Effort Estimation and Related Problems: A Systematic Literature Review
1. General and reference
  1. Document types
    1. Surveys and overviews
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Extra-functional properties
      1. Software reliability

Recommendations

Survey of Software Development Effort Estimation Techniques
ICSCA '18: Proceedings of the 2018 7th International Conference on Software and Computer Applications

Software development effort estimation is one of the most crucial activities in software engineering. Effort estimation permits managers and software engineers to anticipate, forecast, and precisely quote the schedule, budget and manpower requirements. ...
Read More
Effort estimation in agile software development: a systematic literature review
PROMISE '14: Proceedings of the 10th International Conference on Predictive Models in Software Engineering

Context: Ever since the emergence of agile methodologies in 2001, many software companies have shifted to Agile Software Development (ASD), and since then many studies have been conducted to investigate effort estimation within such context; however to ...
Read More
Software test effort estimation

Software Testing is an important process of software development that is performed to support and enhance reliability and quality of the software. It consist of estimating testing effort, selecting suitable test team, designing test cases, executing the ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Computing Surveys Volume 54, Issue 3
April 2022
836 pages
ISSN:0360-0300
EISSN:1557-7341
DOI:10.1145/3461619
Editor:
Albert Zomaya
University of Sydney, Australia
Issue’s Table of Contents
Copyright © 2021 ACM
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 17 April 2021
- Accepted: 1 December 2020
- Revised: 1 September 2020
- Received: 1 December 2019
Published in csur Volume 54, Issue 3

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Testing effort
systematic literature review
testing effort estimation
testing effort estimation-related problems
Qualifiers
- research-article
- Research
- Refereed
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 5
  Total Citations
  View Citations
- 3,070
  Total Downloads
- Downloads (Last 12 months)793
- Downloads (Last 6 weeks)131
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format .

View HTML Format

Software Testing Effort Estimation and Related Problems: A Systematic Literature Review

ACM Computing Surveys

Abstract

References

Cited By

Index Terms

Recommendations

Survey of Software Development Effort Estimation Techniques

Effort estimation in agile software development: a systematic literature review

Software test effort estimation