Skip to main content
SpringerLink
Log in

Software requirements testing approaches: a systematic literature review

  • Original Article
  • Published:
Requirements Engineering Aims and scope Submit manuscript

Abstract

Testing a software system is an important step approach to ensuring quality, safety, and reliability in safety-critical systems (SCS). Several authors have published new approaches to improve the processes of testing safety requirements taking into consideration existing processes that seek to improve techniques and contribute positively with software developers. This article aims to investigate the main approaches to requirements testing, particularly focusing on safety requirements in the context of SCS. We investigated how these approaches have been developed and what contributions they provide to academia and industry. We evaluated the pros and cons of the approaches and how they related to the joint work of requirements engineers and testers. We performed a systematic literature review (SLR), selecting 53 papers published between 1990 and 2018. Our research was conducted according to the guidelines proposed by Kitchenham and Biolchini. The results of this SLR point out to the new research related to the software and safety-critical systems testing. The results show issues in the integration of requirements engineers with the application test team and gaps in the approaches found, particularly in the applications of the techniques in the industry setting. Moreover, several approaches are presented to solve problems and help to prevent future problems. The results of this research point to the main approaches to requirements testing and their use in academia and industry, as well as the advantages and disadvantages. The shortcomings allow us to suggest new research in safety-critical systems in the scope of validation, verification, specification, and testing of safety requirements, as well as to integrate test teams with requirements engineers in order to get better results. Based on the results, we suggest future studies for improvements in the requirements testing techniques to improve the integration of safety requirements and test cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://ieeexplore.ieee.org.

  2. https://dl.acm.org.

  3. https://link.springer.com.

  4. https://www.sciencedirect.com.

  5. https://mendeley.com.

  6. More information at http://cadp.inria.fr/man/lotos.html.

  7. http://www.mit.edu.

  8. The term avionics comes from aviation electronics. Avionics are designated aircraft navigation and communication systems, autopilot, and flight control systems, also include non-pilot onboard electrotechnical systems such as passenger video systems.

Abbreviations

SCR:

Score according to quality assessment

CIT:

Amount of citations of paper

References

  1. Delamaro M, Jino M, Maldonado J (2017) Introdução ao teste de software, vol 1. Elsevier, Brasil

    Google Scholar 

  2. Lau MF, Yu YT (2005) An extended fault class hierarchy for specification-based testing. ACM Trans Softw Eng Methodol 14(3):247–276

    Article  Google Scholar 

  3. Leveson NG (1995) Safeware: system safety and computers. ACM, New York

    Google Scholar 

  4. Ali MM, Moawad R (2010) An approach for requirements based software product line testing. In: 2010 the 7th international conference on informatics and systems (INFOS), 1 March 2010

  5. Ali R, Dalpiaz F, Giorgini P (2013) Reasoning with contextual requirements: detecting inconsistency and conflicts. Inf Softw Technol 55(1):35–57

    Article  Google Scholar 

  6. Alves MCB, Drusinsky D, Shing M-T (2011) A practical formal approach for requirements validation and verification of dependable systems. In: 2011 Fifth Latin- American Symposium on Dependable Computing Workshops. IEEE, pp 47–51

  7. Cimatti A, Roveri M, Susi A, Tonetta S (2012) Validation of requirements for hybrid systems. ACM Trans Softw Eng Methodol 21(4):1–34

    Article  Google Scholar 

  8. Ibrahim R, Saringat MZ, Ibrahim N, Ismail N (2007) An automatic tool for generating test cases from the system’s requirements. In: 7th IEEE international conference on computer and information technology (CIT 2007). IEEE, pp 861–866

  9. Tang W, Ning B, Xu T, Zhao L (2010) Scenario-based modeling and verification for CTCS-3 system requirement specification. In: 2010 2nd International conference on computer engineering and technology, vol 1. IEEE, pp V1-400–V1-403

  10. Andrade E, Maciel P, Gustavo C, Bruno N (2009) A methodology for mapping SysML activity diagram to time petri net for requirement validation of embedded real-time systems with energy constraints. In: 2009 3rd international conference on digital society. IEEE, pp 266–271

  11. Hammani FZ (2014) Survey of non- functional requirements modeling and verification of Software Product Lines. In: 2014 IEEE 8th international conference on research challenges in information science (RCIS), pp 1–6

  12. Sharma A, Kushwaha SD (2011) A metric suite for early estimation of software testing effort using requirement engineering document and its validation. In: 2011 2nd International conference on computer and communication technology, ICCCT-2011, pp 373–378

  13. Kelley K (2009) Automated test case generation from correct and complete system requirements models. In: 2009 IEEE aerospace conference. IEEE, pp 1–10

  14. Ahmend T, Tripathi AR (2007) Specification and verification of safety requirements in a programming model for decentralized CSCW systems. ACM Trans Inf Syst Saf 10(2):7

    Article  Google Scholar 

  15. Sommerville I (2011) Engenharia de software. In: Tradução Ivan Bosnic e Kalinka G. de O. Goncçalves; revisão técnica Kechi Hirama, 9 edition. Pearson Prentice Hall, São Paulo

  16. Driskell SB, Murphy J, Michael BJ, Shing MT (2010) Independent validation of software safety requirements for systems of systems. In: 2010 5th international conference on system of systems engineering, SoSE 2010

  17. Martins LEG, Gorschek T (2016) Requirements engineering for safety-critical systems: a systematic literature review. Inf Softw Technol 75:71–89

    Article  Google Scholar 

  18. Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Engineering 2:1051

    Google Scholar 

  19. Vilela J, Castro J, Martins LEG, Gorschek T (2017) Integration between requirements engineering and safety analysis: a systematic literature review. J Syst Softw 125:68–92

    Article  Google Scholar 

  20. Gurbuz HG, Tekinerdogan B (2018) Model-based testing for software safety: a systematic mapping study. Softw Quality J 26(4):1327–1372

    Article  Google Scholar 

  21. Häser F, Felderer M, Ruth B (2014) Soft-ware paradigms, assessment types and non-functional requirements in model-based integration testing: a systematic literature review. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, EASE’14, New York, NY, USA. ACM, pp 29:1–29:10

  22. Unterkalmsteiner M, Feldt R, Gorschek T (2014) A taxonomy for requirements engineering and software test alignment. ACM Trans Softw Eng Methodol (TOSEM) 23(2):16:1–16:38

    Article  Google Scholar 

  23. Crow J, Di Vito B (1998) Formalizing space shuttle software requirements: four case studies. ACM Trans Softw Eng Methodol 7(3):296–332

    Article  Google Scholar 

  24. Dokhanchi A, Hoxha B, Georgios F (2017) Formal requirement debugging for testing and verification of cyber-physical systems. ACM Trans Embed Comput Syst 17(2):1–26

    Article  Google Scholar 

  25. Felder M, Morzenti A (1992) Validating real-time systems by history- checking TRIO specifications. In: Proceedings of the 14th international conference on software engineering—ICSE’92, vol 3, New York, USA. ACM Press, pp 199–211

  26. Heitmeyer CL, Jeffords RD, Labaw BG (1996) Automated consistency checking of requirements specifications. ACM Trans Softw Eng Methodol 5(3):231–261

    Article  Google Scholar 

  27. Schneider F, Easterbrook SM, Callahan JR, Holzmann GJ (1998) Validating requirements for fault tolerant systems using model checking. In: Proceedings of IEEE international symposium on requirements engineering: RE’98. IEEE Comput. Soc, pp 4–13

  28. Aiello F, Garro A, Lemmens YA, Stefan D (2017) Simulation-based verification of system requirements: an integrated solution. In: 2017 IEEE 14th international conference on networking, sensing and control (ICNSC), vol 1. IEEE, pp 726–731

  29. Amyot D, Logrippo L, Buhr RJA, Gray T (1999) Use case maps for the capture and validation of distributed systems requirements. In: Proceedings IEEE international symposium on requirements engineering (Cat.No.PR00188), pp 44–53

  30. Bouskela D, Nguyen T, Audrey J (2015) Towards a rigorous approach for verifying cyber-physical systems against requirements. In: 2015 IEEE electrical power and energy conference (EPEC). IEEE, pp 250–255

  31. Dalal SR, Jain A, Karunathi N, Leaton JM, Patton CM, Lott GC, Horowitz BM (2005) Model based testing in practice at microsoft. Electr Notes Theor Comput Sci 111:5–12

    Article  Google Scholar 

  32. Dudila R, Letia IA (2013) Towards combining functional requirements tests and unit tests as a preventive practice against software defects. In: 2013 IEEE 9th international conference on intelligent computer communication and processing (ICCP). IEEE, pp 279–282

  33. Farhat S, Simco G, Mitropoulos FJ (2010) Using aspects for testing nonfunctional requirements in object-oriented systems. In: Proceedings of the IEEE SoutheastCon 2010 (SoutheastCon). IEEE, pp 356–359

  34. Foster GJ, Helm AL (2010) A unified approach to requirements validation and system verification. In: 2010 IEEE international systems conference, IEEE, pp 404–408

  35. Fraser G, Ammann P (2008) Reachability and propagation for LTL requirements testing. In: 2008 the 8th international conference on quality software. IEEE, pp 189–198

  36. Ghazel M, Masmoudi M, Toguyeni A (2009) Verification of temporal requirements of complex systems using UML patterns, application to a railway control example. In: IEEE international conference on system of systems engineering. SoSE2009, pp 1–6

  37. Han L, Liu J, Zhou T, Sun J, Chen X (2016) Safety requirements specification and verification for railway interlocking systems. In: 2016 IEEE 40th annual computer software and applications conference (COMPSAC), vol 1. IEEE, pp 335–340

  38. Hasling B, Goetz H, Beetz K (2008) Model based testing of system requirements using UML use case models. In: 2008 international conference on software testing, verification, and validation. IEEE, pp 367–376

  39. Hemmati H, Arcuri A, Briand L (2013) Achieving scalable model-based testing through test case diversity. ACM Trans Softw Eng Methodol 22(1):1–42

    Article  Google Scholar 

  40. Jwo J-S, Cheng YC (2007) Pseudo software: a new concept for iterative requirement development and validation. In: 14th Asia-Pacific software engineering conference (APSEC’07). IEEE, pp 105–111

  41. Seo KI, Choi EM (2006) Comparison of five black-box testing methods for object- oriented software. In: 4th International conference on software engineering research, management and applications (SERA’06). IEEE, pp 213–220

  42. Lee K-H, Min P-G, Cho J-H, Lim D-J (2012) Model-driven requirements validation for automotive embedded software using UML. In: 8th International conference on computing technology and information management (ICCM), vol 1, pp 46–50

  43. Lei H, Wang Y (2016) A model-driven testing framework based on requirement for embedded software. In: 2016 11th International conference on reliability, maintainability and safety (ICRMS). IEEE, pp 1–6

  44. Yin L, Liu J, Li X (2009) Validating requirements model of a B2B system. In: 2009 8th IEEE/ACIS international conference on computer and information science. IEEE, pp 1020–1025

  45. Liu G, Huang S, Piao X (2008) Study on requirement testing method based on alpha-beta cut-off procedure. In: 2008 international conference on internet computing in science and engineering. IEEE, pp 396–402

  46. Gao M, Zhong D, Lu M, Yin Y (2007) Research on test requirement modeling for software-intensive avionics and the tool implementation. In: 2007IEEE/AIAA 26th digital avionics systems conference. IEEE, pp 6.D.2 1–6.D.2 10

  47. Mirarab S, Ganjali A, Ladan T, Li S, Liu W, Morrissey M (2008) A requirement-based software testing framework: an industrial practice. In: 2008 IEEE international conference on software maintenance, pp 452–455

  48. Raja UA (2009) Empirical studies of requirements validation techniques. In: 2009 2nd International conference on computer, control and communication. IEEE, pp 1–9

  49. Ravn AP, Rischel H, Hansen KM (1993) Specifying and verifying requirements of real- time systems. IEEE Trans Softw Eng 19(1):41–55

    Article  Google Scholar 

  50. Sarwar T, Habib W, Arif F (2013) Requirements based testing of software. In: 2013 2nd International conference on informatics and applications (ICIA). IEEE, pp 347–352

  51. Siegl S, Hielscher K-S, German R (2010) model based requirements analysis and testing of automotive systems with timed usage models. In: 2010 18th IEEE international requirements engineering conference. IEEE, pp 345–350

  52. Siegl S, Hielscher K-S, German R, Berger C (2011) Automated testing of embedded automotive systems from requirement specification models. In: LATW 2011—12th IEEE Latin- American Test Workshop

  53. Straszak T, Smialek M (2014) Automating acceptance testing with tool support. In: Automating acceptance testing with tool support, vol 2, pp 1569–1574

  54. Sun Y, Brain M, Kroening D, Hawthorn A, Wilson T, Schanda F, Jimenez FJG, Daniel S, Bryan C, Broster I (2017) Functional requirements-based automated testing for avionics. In: 2017 22nd international conference on engineering of complex computer systems (ICECCS), vol 1. IEEE, pp 170–173

  55. Sutcliffe A, Gregoriades A (2002) Validating functional system requirements with scenarios. In: Proceedings IEEE joint international conference on requirements engineering, vol 2002—Janua. IEEE Comput. Soc, pp 181–188

  56. Wendland M, Schieferdecker I, Vouffo-Feudjio A (2011) Requirements-driven testing with behavior trees. In: 2011 IEEE fourth international conference on software testing, verification and validation workshops. IEEE, pp 501–510

  57. Yang C, Hu X, Zhu Y, Huang M (2014) The domain knowledge-based software test model researches. In: 2014 10th International conference on reliability, maintainability and safety (ICRMS). IEEE, pp 323–328

  58. Yau SS (1994) An approach to object-oriented requirements verification in software development for distributed computing systems. In: Proceedings eighteenth annual international computer software and applications conference (COMPSAC 94), pp 96–102

  59. Yu G, Xu ZW, Du JW (2009) An approach for automated safety testing of safety-critical software system based on safety requirements. In: 2009 International forum on information technology and applications, vol 3. IEEE, pp 166–169

  60. Yu WD (1994) Verifying software requirements: a requirement tracing methodology and its software tool-RADIX. IEEE J Sel Areas Commun 12(2):234–240

    Article  MathSciNet  Google Scholar 

  61. Gutiérrez JJ, Escalona MJ, Mejías M (2015) A model-driven approach for functional test case generation. J Syst Softw 109:214–228

    Article  Google Scholar 

  62. Ober I, Graf S, Ober I (2006) Validating timed UML models by simulation and verification. Int J Softw Tools Technol Transf 8(2):128–145

    Article  Google Scholar 

  63. Stachtiari E, Mavridou A, Panagiotis P, Bliudze S, Sifakis J (2018) Early validation of system requirements and design through correctness-by-construction. J Syst Softw 145:52–78

    Article  Google Scholar 

  64. El-Attar M, Hezam AA-G (2016) Using safety robustness analysis for early-stage validation of functional safety requirements. Requir Eng 21(1):1–27

    Article  Google Scholar 

  65. Kesserwan N, Dssouli R, Bentahar J, Stepien B, Labrèche P (2019) From use case maps to executable test procedures: a scenario-based approach. Softw Syst Model 18(2):1543–1570

    Article  Google Scholar 

  66. Saito S, Takeuchi M, Yamada S, Aoyama M (2014) RISDM: a requirements inspection systems design methodology: Perspective-based design of the pragmatic quality model and question set to SRS. In: 2014 IEEE 22nd international requirements engineering conference (RE). IEEE, pp 223–232

  67. Khosrowjerdi H, Meinke K, Andreas R (2018) Virtualized-fault injection testing: a machine learning approach. In: 2018 IEEE 11th international conference on software testing, verification and validation (ICST). IEEE, pp 297–308

  68. Siegl S, Hielscher K-S, Reinhard G, Christian B (2011) Automated testing of embedded automotive systems from requirement specification models. In: 2011 12th Latin American Test Workshop (LATW). IEEE, pp 1–6

Download references

Acknowledgements

I would thank CAPES (Coordination of Improvement of Higher Level Personnel) for having subsidized 11 months of the pro-grad program in computer science at UNIFESP.

Author information

Authors and Affiliations

  1. Department of Science and Technology, Federal University of São Paulo (UNIFESP), São José dos Campos, SP, Brazil

    Jemison dos Santos & Luiz Eduardo G. Martins

  2. Associated Laboratory for Computing and Applied Mathematics (LABAC), National Institute for Space Research (INPE), São José dos Campos, SP, Brazil

    Valdivino A. de Santiago Júnior

  3. Computer Science Department - Federal Institute for Education, Science, and Technology of São Paulo (IFSP), Caraguatatuba, SP, Brazil

    Lucas Venezian Povoa & Luciana Brasil R. dos Santos

Authors
  1. Jemison dos Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luiz Eduardo G. Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Valdivino A. de Santiago Júnior
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lucas Venezian Povoa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Luciana Brasil R. dos Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jemison dos Santos.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

See Table 10.

Table 10 List of selected primary studies
Full size table

Appendix 2

See Table 11.

Table 11 Pros and cons
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

dos Santos, J., Martins, L.E.G., de Santiago Júnior, V.A. et al. Software requirements testing approaches: a systematic literature review. Requirements Eng 25, 317–337 (2020). https://doi.org/10.1007/s00766-019-00325-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00766-019-00325-w

Keywords

Search

Navigation