The prevalence and severity of persistent ambiguity in software requirements specifications: Is a special effort needed to find them?

https://doi.org/10.1016/j.scico.2020.102472Get rights and content

Highlights

  • There are methods and tools to avoid ambiguities in requirements specifications.

  • But, no one has proved that finding these ambiguities is necessary and pays off.

  • Two empirical studies showed that these ambiguities did not cause real problems.

  • Our research tried and failed to falsify the results of these studies.

  • Maybe searching for these ambiguities is not very cost effective and useful.

Abstract

Context and motivation

All the research in methods and tools for avoiding, detecting, and removing ambiguities in requirements specifications assumes that doing so is necessary and that the methods and tools for doing so are worth the effort to use them. Each of two attempts by de Bruijn et al. and Philippo et al. to test these assumptions empirically with a case study examined a random sampling of the ambiguities in the requirements specification for already constructed software. Each study concluded that ambiguities in the examined requirements specification did not result in any serious defects in the downstream development and seem to have been resolved through the normal multiple inspections and discussions that characterize a serious requirements engineering process.

Question/problem

However, because each study examined only a small random sampling of the many ambiguities in its specification, it may have missed the rare ambiguity that causes a serious defect in the constructed software. Moreover, as a case study, its results cannot be generalized. So the unanswered questions are: (1) “How prevalent are ambiguities that cause defects?” and (2) “What kinds of defects do these ambiguities cause?”

Principal idea/Goal

The research reported in this paper tried hard to falsify de Bruijn's and Philippo's result in three different case studies, each with a requirements specification and already developed software. Each study used a purposive sampling of the ambiguities in its requirements specification to find those ambiguities that are least likely to have been discussed and resolved during the inspections and discussions about the specifications in an attempt to find undetected ambiguities that caused or can cause major defects in the implemented software. The purposive sampling was to identify types of ambiguity, called persistent ambiguities of which many people are not aware; which, therefore, will not come up in any of the discussions about the requirements; and which will persist into the implementation to cause defects. After obtaining the persistent ambiguities in the project's requirements specification, we asked the project's chief requirements engineer if any of them caused or can cause serious defects in the project's software.

Conclusion/Contribution

For the three projects, none of the sampled ambiguities reviewed by each chief requirements engineer caused expensive damage because all of the project's requirements engineers seem to have subconsciously disambiguated the ambiguities in the same way. The first main conclusion is that persistent ambiguities remain undetected during requirements engineering and the subsequent development. The second main conclusion is that a serious requirements engineering process is sufficient to cause all project stakeholders to disambiguate, consciously or not, all ambiguities, persistent or not, in a requirements specification the same way; thus, ambiguities, while present in the specification, do not cause defects in the downstream software. The third main conclusion is that the identification of persistent ambiguities in a requirements specification is potentially an effective and efficient strategy for minimizing damage caused by ambiguity precisely because of its focus on ambiguities that remain undetected due to lack of awareness. Further study is necessary to determine what factors are involved in persistent ambiguity and its prevalence, as well as its potential impacts.

Introduction

The primary goal of requirements engineering (RE)1 is to capture in a requirements specification (RS) all of the requirements that the client, users, and other stakeholders believe to be imperative in the computer-based system (CBS) being developed [2]. Ambiguity in an RS, by which the RS or a piece of it could have multiple meanings, could cause programmers to implement the CBS incorrectly from the client's viewpoint, resulting in major code rewrites, leading to delays in delivery, and introducing even more defects [3]. Therefore, it is worth finding ways to avoid, to even prevent, and failing those, to detect ambiguity in RSs [4], [5], [6], [7], [8], [9].

Project failure has often been attributed to ambiguities in RSs. For example, Gause lists too much unrecognized disambiguation in RSs as one of the five most important sources of requirements failure [10]. This attribution claim has not been conclusively empirically proved.

The underlying claim of the research on ambiguity and on ways to avoid, prevent, and detect ambiguities in RSs is that

  • only by detecting each ambiguous statement in an RS for a CBS, finding each possible interpretation of the statement and rewriting the statement so that its only interpretation is what the CBS's stakeholders intend,

  • can it be assured that what is implemented is what is intended.

Thus, finding ways to avoid, prevent, and detect ambiguities in RSs should be essential for achieving high quality CBSs. Consequently, there has been a lot of research on methods and tools, described in Section 2, to avoid, prevent, and detect ambiguities in RSs. A problem with all of these methods and tools is that we do not actually know if any of them are worth the effort. The same is true for the research to develop these methods and tools. A cost–benefit analysis of these tools and methods would be valuable. Two such analyses have been done in the form of case studies, one by Fabian de Bruijn et al. [11] and one by Erik Jan Philippo et al. [12]. Each study's conclusion is that ambiguities in RSs do not cause serious problems in the downstream implementation of the CBS, and therefore, a special effort to find ambiguities, beyond the normal stakeholders' thoroughly discussing the RS is not necessary.

This paper describes case study research that tried its very best to come to conclusions different from those of de Bruijn and Philippo. The attempt failed, and the study ended up confirming their conclusions. In the rest of this paper, Section 2 discusses the role of ambiguity in RE and briefly describes methods and tools for avoiding, preventing, and detecting ambiguity in RSs. Section 3 lays out the general context and the vocabulary assumed in the rest of the paper. Section 4 gives a detailed description of past attempts to validate the importance of ambiguity detection during RE. Section 5 describes the objectives and the approach taken by our research that attempted to falsify de Bruijn's and Philippo's conclusions. Section 6 defines and gives examples of what this paper calls persistent ambiguities, those that stakeholders do not know about and therefore are likely not to be discussed during RE. Section 7 lays out the research plan for the case studies. Section 8 briefly describes the conduct of the case studies, carried out according to the plan. Section 9 reports the data and the results of the case studies, and Section 10 comes to the verdict about the attempt to falsify de Bruijn's and Philippo's results. Section 11 discusses the results and their implications on the practice of RE. Section 12 concludes the paper with a discussion of future work.

Section snippets

Background on ambiguity in RE

Almost all RSs are written in natural language (NL) [13]. Even when an RS is written in a formal language or with UML diagrams, the RE process still begins with ideas expressed in NL [9]. NLs are inherently ambiguous. Therefore, since NL is unavoidable in the RE process, there is no escaping ambiguous RSs [6].

Gervasi and Zowghi point out that ambiguity in NL RSs “has long been pictured as one of the worst enemies of the specifier” [14]. Consequently, as mentioned, there has been lots of

Context and vocabulary

This section establishes some context about CBS developments in which the phenomenon of ambiguity in RSs plays a role.

An ambiguous statement in the RS for a CBS is a statement that can be interpreted in multiple ways, thus giving rise to the possibility that different people will implement the statement in different ways. The phenomenon of ambiguity in RSs has been described thoroughly elsewhere [7], [8], [40]. In the interest of saving space in this paper, the reader desiring a more detailed

Past attempts to validate importance of ambiguity detection in RE

Fabian de Bruijn, under supervision by H. Dekker, tried to determine the effect of ambiguity in a project's RS on the project's downstream success or failure [11]. He analyzed the 279-requirement RS for one failed CBS development project. With his analysis, he attempted to answer the following research questions [11]:

  • 1.

    How many requirement statements are ambiguous?

  • 2.

    How many problems were caused by ambiguous requirements?

De Bruijn's analysis determined that only one not very severe defect in the

Research objectives and approach

We decided to try our damndest to falsify de Bruijn's and Philippo's result, using the RSs from three different projects3 that were well into deployment, using a totally different sampling method designed to both (1) reduce the number of ambiguities to investigate to a manageable number and (2) increase the chances for falsification. We used a purposive sampling

Persistent ambiguities

Language usage is constantly evolving. Occasionally people's usage does not follow the rules of language for brevity or emphasis. Some sentences, although grammatically problematic, have become common usage. The description below of ambiguities in RSs focuses on how people use language, grammatically correct or not, and how that usage might cause ambiguity. Although the description and examples are restricted to the English language, each ambiguity exists in all languages that have the

Research method

The research method is to conduct several case studies, each involving examination of

  • a real-world project, P, with a serious RE process, to develop a CBS, C,

  • the project's RS, R, specifying C, and

  • the project's implementation, I, of C according to R,

to determine if ambiguities in R that were not detected by P's members, ended up causing damage to I.

That the case study be of a real-world project means that it would be necessary to obtain R from a real-world company, C, that is developing CBSs. In

Conduct of studies

We were able to find a company that was willing to provide RSs for already-completed CBS developments. To obtain the company's documents and cooperation in conducting the case studies, we had to sign a non-disclosure agreement, i.e., not to reveal the company's name, business, and RSs. Consequently, none of the examples in this paper of ambiguities are from the RSs. To allow the reader to understand the nature of the RSs, we can say that the company's business involves serving customers that

Results

For each of most ambiguity types, the results of each of the RSs are discussed separately because different requirements engineers were involved for each project. Also, for each ambiguity type, an analysis of patterns and differences between the three projects is provided. The projects are referred to as “P1”, “P2”, and “P3”, and their RSs are R1, R2, and R3, respectively.

Final verdict

The concerted attempt of the case studies to falsify de Bruijn's and Philippo's conclusions failed, and the results of the case studies end up supporting their conclusion, namely that ambiguities in an RS are not a serious problem for a well-run CBS development because in-project discussions about the CBS's requirements ends up resolving all ambiguities either consciously or unconsciously. In fact, in the case studies, none of the ambiguities that the CREs did not know about exposed any defects

Discussion

This section attempts to use the results to answer the RQs:

  • 1.

    What is the prevalence of PA?

  • 2.

    What is the impact of PA?

Then it speculates about which approach has the lower cost: (1) identifying PAs in an RS during RE and (2) repairing the damage caused by undetected ambiguities in later phases of the SDLC.

Conclusions

Current research is questioning the value of inspecting RSs for ambiguity because it has been shown by de Bruijn et al. [11] and Philippo et al. [12] in two separate case studies that during normal serious RE that leads to an RS, ambiguities in the RS are resolved through the continual discussion that takes place normally during the RE. The purpose of the research described in this paper was to try the hardest to falsify de Bruijn's and Philippo's results, each of which is based on a random

Acronyms

  • CRE

    chief requirements engineer

  • CBS

    computer-based system

  • NL

    natural language

  • PA

    persistent ambiguity

  • RE

    requirements engineering

  • RS

    requirements specification

  • RQ

    research question

  • SDLC

    software development lifecycle

  • SA

    subconscious ambiguation

  • SD

    subconscious disambiguation

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

Cristina Ribeiro's and Daniel Berry's work were supported in parts by a Canadian NSERC Discovery Grant NSERC-RGPIN227055.

References (58)

  • C. Ribeiro

    The Prevalence and Impact of Persistent Ambiguity in Software Requirements Specification Documents

    (2016)
  • S. Robertson et al.

    Mastering the Requirements Process

    (2006)
  • D. Gause et al.

    Exploring Requirements: Quality Before Design

    (1989)
  • B.L. Kovitz

    Practical Software Requirements: A Manual of Content and Style

    (1998)
  • N. Fuchs et al.

    Attempto controlled english (ACE) language manual version 3.0

    (1999)
  • L. Mich et al.

    Ambiguity measures in requirement engineering

  • D.M. Berry et al.

    From contract drafting to software specification: Linguistic sources of ambiguity

    (2003)
  • D.M. Berry et al.

    Ambiguity in requirements specification

  • D.M. Berry

    Ambiguity in natural language requirements documents

  • D.C. Gause

    User DRIVEN design—the luxury that has become a necessity

  • F. De Bruijn et al.

    Ambiguity in natural language software requirements: a case study

  • E.J. Philippo et al.

    Requirement ambiguity not as important as expected — results of an empirical evaluation

  • L. Mich et al.

    Market research for requirements analysis using linguistic tools

    Requir. Eng. J.

    (2004)
  • V. Gervasi et al.

    On the role of ambiguity in re

  • L. Dupré

    Bugs in Writing: A Guide to Debugging Your Prose

    (1998)
  • D.M. Berry et al.

    The syntactically dangerous all and plural in specifications

    IEEE Softw.

    (2005)
  • H. Enomoto et al.

    Natural language based software development system TELL

  • J. Comer

    An experimental natural-language processor for generating data type specifications

    SIGPLAN Not.

    (1983)
  • M. Poesio

    Semantic ambiguity and perceived ambiguity, semantic ambiguity and underspecification

  • E. Kamsties et al.

    Detecting ambiguities in requirements documents using inspections

  • S.F. Tjong et al.

    Extended disambiguation rules for requirements specifications

  • A. Ferrari et al.

    Pragmatic ambiguity detection in natural language requirements

  • W.M. Wilson et al.

    Automated analysis of requirement specifications

  • M. Osborne et al.

    Processing natural language software requirement specifications

  • L.H. Rosenberg et al.

    Requirements, testing, and metrics

  • S. Armstrong et al.

    Natural Language Processing Using Very Large Corpora

    (1999)
  • D.M. Berry et al.

    A new quality model for natural language requirements specifications

  • S.F. Tjong et al.

    An Adaptive Parsing Technique for ProRules Grammar

    (2006)
  • N. Kiyavitskaya et al.

    Requirements for tools for ambiguity identification and measurement in natural language requirements specifications

    Requir. Eng. J.

    (2008)
  • Cited by (0)

    This paper is derived from and includes verbatim text from the first author's PhD thesis [1].

    View full text