State identification sequences from the splitting tree

Abstract

Context: Software testing based on finite-state machines.

Objective:Improving the performance of existing testing methods by construction of more efficient separating sequences, so that states entered by a system under test can be identified in a much shorter span of time.

Method: This paper proposes an efficient way to construct separating sequences for subsets of states for any deterministic finite-state machine. It extends an existing algorithm that builds an adaptive distinguishing sequence (ADS) from a splitting tree to machines that do not possess an ADS. Our extension to this construction algorithm allows one not only to construct a separating sequence for any subset of states but also form sets of separating sequences, such as harmonized state identifiers (HSI) and incomplete adaptive distinguishing sequences, that are used by efficient testing and learning algorithms.

Results: The experiments confirm that the length and number of test sequences produced by testing methods that use HSIs constructed by our extension is significantly improved.

Conclusion:By constructing more efficient separating sequences the performance of existing test methods significantly improves.

Introduction

Software testing takes a significant amount of time, so effective testing methods can improve the quality of software, the cost of development and the cost of execution as well. Testing methods for finite-state machines are known for their capability of both finding subtle defects and for theoretical guarantees of fault-finding. The downside of these methods is the amount of testing that has to be completed before any of the claimed guarantees can be attained. A significant contribution to the amount of testing is the number and length of what is known as separating sequences. These are the sequences of inputs derived from a finite-state model of a specification that are intended to identify states entered by an implementation during testing. For instance, clicking on a link on a web page to submit an order should place such an order and empty a basket, so that attempting to submit the same form twice would not result in a duplicate order. Knowing whether a basket is indeed empty might not be visible on a ‘thank you for your order’ page so for most web sites a tester has to navigate back on the main page to check the contents of the basket and possibly check the list of orders to observe that an order has indeed been recorded as placed. This is important: verification that a correct state has been entered requires interaction with a system under test with effects not immediately visible. Construction of sequences that efficiently verify states is the subject of this paper. The intention is to make it possible to construct efficient sequences to separate any chosen subset of states in a specification, because usually it is known which states may or may not be entered at the end of a sequence of commands.

The algorithm proposed is an extension to an existing algorithm that aims to build a splitting tree (ST) for finite-state machines (FSM) possessing what is known as an adaptive distinguishing sequence (ADS). An ADS is in fact a set of sequences with common prefixes. It is best represented with a tree where nodes are named with inputs and branches carrying outputs. States are identified by walking through this tree starting from the root: inputs are submitted to a system under test and depending on outputs observed, a branch is taken. An input corresponding to the entered node is sent to a system under test next and an output determines the next node. Every leaf of this tree is associated with a state in a model which is uniquely identified by the sequence of inputs/outputs from the root of the tree to the leaf. Such sequences from root to leaf are called state verifying sequences (SVS).

The existing algorithm is limited to a specific range of FSMs where an ADS can be constructed. Where it cannot be built, one has to use more than a single sequence to identify states. Most existing testing methods support multiple sequences to identify states, although they are obviously more efficient where an ADS is available.

The extension to the original algorithm presented in this paper builds a small number of sequences to separate states if an ADS does not exist and produces an ADS where it does. It can be used both for identification of individual states and for separation of a group of states (where a tester knows that only some states may be entered by the implementation but does not know which exactly). The described method also permits harmonized state identifiers (HSI) to be constructed which lead to test sequences shorter than those constructed using traditional testing methods [1], [2]. This dramatically improves the efficiency of testing methods relying on HSI sequences.

Section 2 defines used terms and Section 3 summarizes the related work. Section 4 sketches the benefits of our extension. Section 5 introduces the structure of splitting tree and what a valid input means. Algorithms for the construction of separating sequences, HSIs and IADSs from the splitting tree are proposed in Section 6. Section 7 proposes our extension that is then described on a running example in 8 Running example, 9 Experiments describes experiments on randomly-generated machines and the paper is concluded in Section 10.