A Combining Method for Wireless Protocol Conformance Testing: A Empirical Case

Wei-Wei, Lin; Hong-Wei, Zeng; Kim, Jung Yoon

doi:https://doi.org/10.1155/2021/6378139

Wireless Communications and Mobile Computing

On this page

Abstract Introduction Preliminaries Related Work Conclusion Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Special Issue

Smart Edge in Intelligent Industrial Systems

View this Special Issue

Research Article | Open Access

Volume 2021 | Article ID 6378139 | https://doi.org/10.1155/2021/6378139

A Combining Method for Wireless Protocol Conformance Testing: A Empirical Case

Lin Wei-Wei,^1,2Zeng Hong-Wei,¹and Jung Yoon Kim³

Academic Editor: Xiaoxian Yang

Received29 May 2021

Revised06 Jun 2021

Accepted18 Jun 2021

Published15 Jul 2021

Abstract

Ensuring the consistency of protocol implementation and protocol specification is the basic premise of wireless communication. With the application of wireless communication in more and more fields, the wireless communication environment becomes more and more complex, and the fault coverage of wireless protocol conformance testing is also facing more and more challenges. To solve this problem, this paper uses Finite State Machine (FSM) as a formal description tool for wireless protocols and presents a combining test method based on two test methods with complementary characteristics in the test technologies based on structural coverage and state identification. Then, the paper evaluates the effectiveness of the method based on 14 empirical cases. The experimental results show that the fault coverage of each empirical case can be effectively improved to 100% when the average test cost is only increased by 17.99%.

1. Introduction

Wireless protocol is the rule that must be obeyed by the communication entities and processes in wireless communication [1]. It is also the basic guarantee to realize the stability of wireless communication and the competitive advantage in the low-power wireless communication market [2, 3]. Conformance testing, performance testing, interoperability testing, and robustness testing are all essential to verify the correctness of wireless protocol implementation [4]. Conformance testing is the basis of the other three kinds of testing, which is used to ensure the consistency of protocol implementation and protocol specification [1, 5, 6]. Finite State Machine (FSM) [7–10] is an effective formal modeling tool [11, 12] in wireless protocol conformance testing since the operations implemented on the entities in wireless protocols (such as button click or window input in GUI and message request in Web service) can be mapped to the inputs in FSMs, and the responses to the operations according to the wireless protocols (such as message response in GUI or web service) can be mapped to the outputs in FSMs [13–16]. To facilitate the description, we abstract these events into alphabetic or numeric symbols in the examples and experiments of this paper. ZigBee is an emerging short-range, low-rate, and low-power wireless network technology. Taking the connection and release process of MAC layer in ZigBee protocol as an example [17], the interaction process between communication entities is shown in Figure 1(a) and the corresponding FSM is shown in Figure 1(b). The relationships between specific events in ZigBee protocol and abstract symbol definitions in FSM are described in Tables 1, 2, and 3, respectively. In this way, the problem of judging whether the implementation of ZigBee protocol is consistent with the ZigBee protocol specification can be transformed into the problem of judging whether the implementation of ZigBee protocol is consistent with the specification FSM of ZigBee protocol. Specifically, a test set is generated based on the specification FSM and the test set is applied to the wireless protocol implementation. When the actual output is consistent with the expected output, the implementation is consistent with the protocol. Otherwise, it means that there is a fault in the implementation.

(a)

(b)

Both the transition-pair coverage (TPC) method [18–20] based on structure coverage and the unique input output (UIO) method [7, 13] based on state identification can be used in wireless protocol conformance testing. However, neither TPC method nor UIO method can achieve 100% fault coverage. To meet the high reliability requirements of wireless protocol conformance testing, this paper presents a combined method based on the complementary characteristics of the TPC method and UIO method. And the combined method is studied by 14 established empirical cases including ZigBee protocol and the WM2RP protocol both of which are widely used in wireless communication. The contribution of this paper is mainly as follows:(1)The fault coverage of wireless protocol conformance testing is effectively improved(2)Since the experimental objects are all FSM empirical cases designed and generated by testers according to professional experience and domain knowledge, the experimental data and results have more practical significance, which can also promote the application of FSM conformance testing in industry

The remainder of the paper is organized as follows. Section 2 introduces the concepts and terms about FSM. Section 3 presents a combined method for wireless protocol conformance testing. Section 4 describes the case study and evaluates the experimental results. Section 5 overviews the related work. Finally, Section 6 summarizes the full text and prospects of the future work.

2. Preliminaries

An FSM is formally defined as a 6-tuple [21, 22] where(i) and are finite and nonempty sets of input symbols and output symbols, respectively(ii) is a finite and nonempty set of states(iii) represents the initial state of (iv) denotes the state transition function(v) is the output function

According to this definition, when an input symbol is delivered to the current state , moves to state with an output produced by . A transition is defined by a tuple where is the start state of , is an input of , is the associated output, and is the end state of .

An FSM M is initially connected if for any state there is at least one input sequence from to s, i.e., (where is the set of finite sequences of input symbols).

If , then and are a pair of distinguishable states, and is called a separating sequence of and . If all the states in FSM M are pairwise distinguishable, i.e., , then is said to be minimal or reduced.

An FSM M is deterministic if at any state and for any input , there is at most one transition leading to the next state. Otherwise, is nondeterministic.

An FSM is not initially connected means there are some states that cannot be reached from the initial state. In other words, the states that cannot be reached from the initial state are meaningless to the FSM. Therefore, an FSM that is not initially connected can be converted to an initially connected FSM by deleting the states that cannot be reached from the initial state. An FSM with equivalent states can be transformed into a reduced FSM by joining equivalent states. A nondeterministic FSM can be converted to a deterministic FSM by combining states, i.e., raising the level of abstraction. Since all the above definitions can be satisfied, only initially connected, reduced and deterministic FSMs are considered in this paper. In Figure 2, is an initially connected, reduced, and deterministic FSM where and highlighted in bold defaults to the initial state.

The connection of input sequences in FSM is represented by the juxtaposition of input sequences; that is, if , then represents the connection of and . If , then is the prefix of , expressed as . If ( is an empty string), then is the true prefix of , expressed as . Given a set of input sequences , represents the set of all prefixes of all sequences in ; that is, ; if then, is a set of prefix closures.

In this work, we assume that FSM M has a reset operation which can bring to its initial state from any state without any output. A string is said to be a defined input sequence at if there exists , where , such that and for all . We use to denote the set of all defined input sequences at state and as an abbreviation for , i.e., the input sequences defined at the initial state of M, and hence for itself. A test case of is an input sequence starting with . A test suite of is a finite set of test cases, and there are not two test cases and such that . The number of symbols of a sequence is represented by . For instance, given a test case for , the length of is 4, since it contains four symbols, that is, . This notation is extended to a test suite ; i.e., means the sum of all the length of sequences in . For instance, given a test suite for , the length of is 13, i.e., .

An initialization fault occurs when FSM starts from an incorrect initial state (≠). An output fault occurs when the correct output of a transition (, , ) is changed, and a transition (, , ) in which the output is different from the expected one is generated. An end state fault occurs when the end state of a transition (, , ) is changed, and a transition (, , ) in which the end state is different from the expected one is generated. Figure 3 demonstrates the three types of faults based on . There is an initialization fault starting from an incorrect initial state in Figure 3(a). There is an output fault in Figure 3(b). The transition (, , ) is changed to the transition (, , ) with a wrong output. There is an end state fault in Figure 3(c). The transition (, , ) is changed to the transition (, , ) with a wrong end state.

(a)

(b)

(c)

3. TPC-UIO Test Method

3.1. Motivation Example

Figures 4 and 5 are two examples to present the complementarity between the TPC method and UIO method.

(a)

(b)

(a)

(b)

The FSM in Figure 4(a) is a wireless protocol specification, and the FSM in Figure 4(b) is an implementation, where the dotted line represents a transition with an end state fault. {rcbbdb, rcbbecda, rcbbecbbac} is the test set of TPC method generated based on Figure 4(a), and {rcbba, rcbbdb, rcbbecbbacb, rcbbecda} is the test set of UIO method generated based on Figure 4(a).

The FSM in Figure 5(a) is a wireless protocol specification, and the FSM in Figure 5(b) is an implementation, where the dotted line also represents a transition with an end state fault. {rba, rbb, raaa, raab, rabaa, rabab, rabba, rabbb} is the test set of the TPC method generated based on Figure 5(a), and {raaa, rbba, rabba, rabaaa} is the test set of the UIO method generated based on Figure 5(a).

Test results indicate the following:(1)The test set of the TPC method can test the fault in Figure 5(b) but cannot test the fault in Figure 4(b)(2)The test set of the UIO method can test the fault in Figure 4(b) but cannot test the fault in Figure 5(b)

It can be concluded from the above examples that there are obvious differences in the performance of TPC method and UIO method in different examples. Specifically, the two methods show strong complementarity. When the number of transitions outgoing from the correct end state and the wrong end state is equal, furthermore, the input/output of the transitions outgoing from the correct end state and the wrong end state are all the same, fault coverage ability of UIO method is superior to TPC method. When there are differences about the number of transitions outgoing from the correct end state and the wrong end state or there are differences about the input/output of transitions outgoing from the correct end state and the wrong end state and the UIO sequence of the correct end state is also available in the wrong end state, fault coverage ability of the TPC method is superior to that of the UIO method.

3.2. Test Generation of TPC-UIO Method

Based on the above analysis, we introduce a combined TPC-UIO test method to realize the complementarity of the TPC method and UIO method.

3.2.1. State Covering Set Generation

Definition 1 (State covering set). Given an FSM and an input sequence set , if for any state satisfies , where is the initial state, then is a state covering set of .

There must be a state covering set in initially connected FSM, and an empty sequence is included to reach the initial state . In addition, only state covering set of prefix closure is considered in this paper. As described in Algorithm 1, given an FSM with states {} in which is the initial state and a transition tree with only a root node , state covering set is first initialized to . Then, for each transition starting with , a branch annotated with the input/output of the transition is added in . Next, for each transition starting with a leaf node appears for the first time in , a branch annotated with the input/output of the transition is added in iteratively until there is no leaf node appearing for the first time that can be found. So far, the complete transition tree is constructed. Finally, for each state , an input/output sequence from to that appears for the first time in is obtained and is thrown to .

Input: FSM M with n states {, ,…} and is the initial state, transition tree T and Root(T)=
Output: State covering set SC(M)
1: Initialize SC(M)=ϕ
2: for each transition starting with
3: Add a branch in T and mark the branch with the input/output of the transition
4: end for
5: do
6: for each leaf node appears for the first time in T
7: Add a branch in T for each transition starting with the leaf node and mark the branch with the input/output of the transition
8: end for
9: until leaf node appears for the first time cannot be found in T
10: for (i=1; i≤n; i++)
11: Get an input/output sequence from to that appears for the first time in T
12: SC(M)= SC(M)∪
13: end for

According to Algorithm 1, the transition tree of is shown in Figure 6, and is a state covering set obtained from the transition tree.

3.2.2. TPC Test Set Generation

Definition 2 (adjacent transition). If the end state of a transition =(, s, ) is the start state of a transition =(s, , ), and are a pair of adjacent transition.
The test set of TPC method covers each pair of adjacent transitions in FSM at least once. As shown in Algorithm 2, given an FSM with states {} in which is the initial state and a state covering set }, TPC test set is first initialized to . Then, for each state , each transition sequence of length 2 starting from is obtained and is thrown to .

Input: FSM M with n states {, ,…} and is the initial state, state covering set SC(M)={, ,…}
Output: TPC test set TPC(M)
1: Initialize TPC(M)=ϕ
2: for (i=1; i≤n; i++)
3: for each transition sequence β of length 2 starting from
4: TPC(M)= TPC(M) ∪
5: end for
6: end for

Consider in Figure 2, {rxx, rxy, ryy, ryx, ryxx, ryxy, ryyx, ryyy, ryyxx, ry, ryyxy, ryyyx, ryyyy} is the test set of under the TPC method.

3.2.3. UIO Generation

Definition 3 (UIO sequence). An input/output sequence is a UIO sequence of state if and only if for any state , where is the input part of .

Definition 4 (uncertainty of the start states). Given an FSM and an input sequence , a partition of the state set can be obtained according to the output generated by each state for . For any state and , they will be in the same partition block if and only if they cannot be distinguished by , i.e., , and this partition is called the uncertainty of the start states.

Definition 5 (uncertainty of the current states). The current states of after delivering can be expressed as , and this partition is called the uncertainty of the current states.

In this way, the scope of the state before and after the input can be learned according to the output corresponding to . The idea of state partition can be applied to the construction of successor tree.

Definition 6 (successor tree). The successor tree of an FSM is a tree showing the behavior of the FSM at any state and upon any input sequence.

For each possible input sequence, there is a path starting from the root node in the successor tree and each node in the successor tree is annotated with the uncertainty of start states as well as the uncertainty of current states.

The successor tree of is shown in Figure 7. When is delivered to the set of states {, , }, according to the different output, the partition blocks of start state uncertainty are } and 1{}, respectively, where 0 and 1 represent the output generated by the state in the partition block. The corresponding partition blocks of current state uncertainty are {} and {}, respectively. When is again delivered to the current state uncertainty {} and {} according to the different output, the partition blocks of start state uncertainty are 00{} and 11{}, respectively. The corresponding partition blocks of current state uncertainty are {} and {}. Since the set {} can never be further divided by , this branch can terminate the decomposition about input . Next, is delivered to the partition blocks of the current state uncertainty {} and{}; according to the different output, the partition blocks of start state uncertainty are 01{}, 00{}, and 11{}, respectively. The corresponding current state uncertainty partition blocks are {}, {}, and {}, respectively. Since there is only one element in each partition block, the division of this branch ends here. In the same way, we can construct other branches in the successor tree until a successor tree meeting the requirement is generated.

In this paper, UIO sequence is obtained by successor tree. Given an FSM and its successor tree, if there is only one state in a partition block of the start state uncertainty, we will get a UIO sequence of . Here, we only consider the shortest UIO sequence; that is, the UIO sequence we first obtained in the top-down search of the successor tree. As shown in Figure 7, the UIO sequence of is , , or , the UIO sequence of is , and the UIO sequence of is .

3.2.4. Test Set Generation of TPC-UIO Method

Definition 7 (TPC-UIO method). The test set of the TPC-UIO method should satisfy two conditions. (1) Cover each pair of adjacent transition in an FSM at least once. (2) Identify the end state of each pair of adjacent transition using the shortest UIO sequence.

As shown in Algorithm 3, given TPC test set of an FSM and UIO sequence of each state, for each test case in , identify the end state of the test case by the corresponding UIO sequence. Then, the test cases which are true prefixes of others are deleted and TPC-UIO test set TPC- is constructed.

Input: TPC test set TPC(M), UIO sequence of each state
Output: TPC-UIO test set TPC-UIO(M)
1: for each test case in TPC(M)
2: Identify the end state of the test case by the corresponding UIO sequence
3: end for
4: delete the test cases which are true prefixes of others

Taking for example, {rxxxy, rxyx, ryxxx, ryxyy, ryyyxy, ryyxyxy, ryyyxxy, ryyyyx} is the final test set of the TPC-UIO method.

4. Case Study

4.1. Experimental Subject

In existing literatures on FSM conformance testing, most of researchers conduct experiment using randomly generated FSMs. Generate FSMs randomly can obtain a large amount of FSMs in a short time, however, whether the experimental conclusions obtained from randomly generated FSMs are completely applicable on FSM empirical cases are still uncertain. This paper selects 14 established FSM empirical cases which are designed and generated by testers with professional experience and domain knowledge [17, 22–29]. The information of all FSM empirical cases is shown in Table 4. , , and denote the number of states, the number of inputs, and the number of transitions, respectively.

4.2. Mutation Generation

In this paper, we use mutations of the specification to model faults in the implementation. Mutation operators [30] are introduced to obtain the three most typical faults in FSM conformance testing.

Definition 8 (change initial state (CIS)). This operator represents the initialization fault: change initial state to such that the implementation starts from an incorrect initial state, that is, .

Definition 9 (change output (CO)). This operator represents the output fault such that an unexpected output is produced in an implementation. Formally, .

Definition 10 (change end state (CES)). This operator represents the end state fault such that the implementation moves to an incorrect state. Formally, .

4.3. Experimental Setup

In order to improve the credibility of the experimental results, experimental replication is adopted in this paper, and the number of replications in each round is not the same. For every FSM empirical case, each type of mutation is randomly operated 10 times, 100 times, 1000 times, 10000 times, 20000 times, 30000 times, 40000 times, 50000 times, 60000 times, 70000 times, 80000 times, 90000 times, and 100000 times; that is, a total of three groups including 551110 mutations in each group are obtained.

4.4. Result Analysis

4.4.1. Comparison of Test Cost

The test set information of all FSM empirical cases is shown in Table 5. L represents the length of the test set and N represents the number of the test cases in the test set, that is, the number of reset operations.(i)Only consider the length of the test set. For all FSM empirical cases, when considering the average length of the test sets of each test method, the average length of the UIO method is 59.64, the average length of the TPC method is 186.57, and the average length of the TPC-UIO method is 227.93. Figure 8 indicates that the length of the test set for TPC-UIO method is larger than that for the TPC method and UIO method for each empirical case. Compared with that of the TPC method, the average increase of the length of the TPC-UIO method is 24.32%(ii)Only consider the number of the test cases. For all FSM empirical cases, when considering the average number of the test cases of each test method, the average number of the test cases for the UIO method is 9.93, the average number of the test cases for the TCP method is 32.07, and the average number of the test cases for the TPC-UIO method is 33.14. Figure 9 indicates that the number of the test cases for TPC-UIO method is more than or equal to that for the TPC method and UIO method for each empirical case. Compared with the TPC method, the average increase of the number of the test cases of the TPC-UIO method is 6.43%(iii)Consider both the length of the test set and the number of the test cases. In order to quantitatively analyze the cost considering both the length of the test set and the number of the test cases, is used to denote the length of the test set and the cost generated by the length of the test sets is expressed as a function , and is used to denote the number of the test cases and the cost generated by the number of the test cases is expressed as a function . The total test cost considering both the length of the test set and the number of the test cases can be expressed as +. Assuming that the test cost of a test set containing only one test case of length 1 is 1, then . Assuming , then , , , and where represents the length of the th test case, so that the cost of each test method can be quantified by assigning a value to . Generally, the impact of the length of the test set is greater than the impact of the number of the test cases, so it is usually . Figure 10 shows the relationship among the comprehensive test costs of each empirical case under different test methods when is assigned 0.6, 0.7, 0.8, and 0.9, respectively. From the examples of different values of , it can be seen that when takes different values greater than 0.5, the comprehensive test cost of each empirical case under different test methods changes, and the larger the value of , the greater the test cost. However, the relative relationship among test costs of different empirical cases has not changed. The comprehensive test cost of the TPC-UIO method is always larger than that of the TPC method and UIO method for each empirical case. Compared with the TPC method, the average increase of the comprehensive test cost of the TPC-UIO method is 23.22%.

(a)

(b)

(c)

(d)

4.4.2. Comparison of Fault Coverage

As shown in Table 6, the TPC method, UIO method, and TPC-UIO method can achieve 100% fault coverage for initialization fault and output fault. Neither the TPC method nor the UIO method can reach 100% fault coverage for end state fault. The TPC-UIO method can effectively improve the fault coverage rate and achieve 100% fault coverage.

Although the differences of data in the experiment seem to be subtle, these differences are still worthy of attention. (1) When the fault domain is very large, even a small increase of fault coverage represents a large number of faults. (2) In a safety-critical environment, there is a tiny optimization even if an optimization of only one fault has high practical value. It is noted that the end state faults in the experiment are generated randomly. In fact, the higher the proportion of the analyzed end state faults in Section 4, the better the fault coverage of the combined method.

Conformance testing based on FSM has always been an active research direction in software testing [31–39]. In the existing literature, there are precedents for improving fault coverage in conformance testing by combining different testing methods. Mouchawrab et al. [40] investigated the complementarity of the test method of round trip path based on UML state machine and the test method of edge coverage based on source code and then extended round-trip paths testing with edge coverage testing. Results show that the two test strategies are significantly more effective when combined by augmenting state machine testing with structural testing. Hutchins et al. [41] combined the all-use criterion in a white-box test and the edge coverage criterion in a black-box test to improve fault coverage with higher test cost. Results show that significant improvements in the effectiveness of coverage-based tests usually occurred as coverage increased from 90% to 100% and the complementarity of the two methods in their effectiveness was proved. Briand et al. [42] extended the round trip path criterion in the white-box test with the classification partition in the black-box test and obtained better fault detection performance.

This paper also analyzes the complementary characteristic of different test methods and improves the fault coverage at the expense of an acceptable increase of test cost, so as to meet the high requirements of fault coverage in wireless communications.

6. Conclusion and Future Work

There are some differences about the test requirements of the TPC method and UIO method in wireless protocol conformance testing. The TPC method focuses on covering all consecutive operations of length two between entities or processes in wireless communication. UIO method focuses on identifying each state of entities or processes and confirming the input and output information between any two states in wireless communication. In order to improve the effectiveness of wireless protocol testing, a combined method is presented and the method is evaluated by 14 established empirical cases. A unified method based on state covering set is used to generate test sets for each test method and three types of classic faults are generated by mutating the specification. Fault coverage has been effectively improved by the combining method. The experimental results show that the fault coverage of each empirical case can be improved to 100% when the average test cost is only increased by 17.99%. Since what we considered is the specifications modeled by FSMs and the faults generated based on FSMs, although the fault coverage has been effectively improved, these results cannot be popularized to the specifications modeled by other formal models or faults generated based on code.

In a future work, we will try to enrich the experimental data with more publicly available FSM empirical cases. To better describe the behavior characteristics of wireless protocol testing, the future work will also consider extended semantic FSMs to improve the expression ability of the model.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

None of the authors have any conflicts of interest.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 61572306.

References

W. W. Lin, H. W. Zeng, H. H. Gao, H. K. Miao, and X. L. Wang, “Test sequence reduction of wireless protocol conformance testing to Internet of Things,” Security and Communication Networks, vol. 2018, Article ID 3723691, 13 pages, 2018.
View at: Publisher Site | Google Scholar
H. H. Gao, W. Q. Huang, and X. X. Yang, “Applying probabilistic model checking to path planning in an intelligent transportation system using mobility trajectories and their statistical data,” Intelligent Automation and Soft Computing, vol. 25, no. 3, pp. 547–559, 2019.
View at: Google Scholar
X. L. Wang, H. W. Zeng, H. H. Gao, H. K. Miao, and W. W. Lin, “Location-based test case prioritization for software embedded in mobile devices using the law of gravitation,” Mobile Information Systems, vol. 2019, Article ID 9083956, 14 pages, 2019.
View at: Publisher Site | Google Scholar
H. H. Gao, W. Q. Huang, X. X. Yang, Y. C. Duan, and Y. Y. Yin, “Toward service selection for workflow reconfiguration: an interface-based computing solution,” Future Generation Computer Systems, vol. 87, pp. 298–311, 2018.
View at: Publisher Site | Google Scholar
H. L. Yang, K. Ma, C. Deng, H. S. Liao, J. Yan, and J. Zhang, “Towards conformance testing of choreography based on scenario2013 International Symposium on Theoretical Aspects of Software Engineering,” pp. 59–62, Birmingham, UK, 2013.
View at: Publisher Site | Google Scholar
M. S. Ramada, T. W. De Lima, A. Simao, and A. D. Soares, “Generating reduced tests for FSMs using a search-based testing approach,” in 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pp. 400–407, Portland, OR, USA, November 4-6, 2019.
View at: Publisher Site | Google Scholar
P. Liu, H. K. Miao, H. W. Zeng, and Y. Liu, “FSM-based testing: theory, method and evaluation,” Chinese Journal of Computers, vol. 34, no. 6, pp. 965–984, 2011.
View at: Publisher Site | Google Scholar
A. Jaffari, C. Yoo, and J. Lee, “Automatic test data generation using the activity diagram and search-based technique,” Applied Sciences, vol. 10, no. 10, p. 3397, 2020.
View at: Publisher Site | Google Scholar
G. Z. Lu and H. K. Miao, “Optimized test cases generation for EFSM model combining abstraction refinement and satisfiability,” Chinese Journal of Computers, vol. 39, no. 11, pp. 2236–2252, 2016.
View at: Publisher Site | Google Scholar
A. Saeed, S. H. Ab Hamid, and A. A. Sani, “Cost and effectiveness of search-based techniques for model-based testing: an empirical analysis,” International Journal of Software Engineering and Knowledge Engineering, vol. 27, no. 4, pp. 601–622, 2017.
View at: Publisher Site | Google Scholar
H. H. Gao, S. Y. Mao, W. Q. Huang, and X. X. Yang, “Applying probabilistic model checking to financial production risk evaluation and control: a case study of Alibaba's Yu'e Bao,” IEEE Transactions on Computational Social Systems, vol. 5, no. 3, pp. 785–795, 2018.
View at: Publisher Site | Google Scholar
H. H. Gao, H. K. Miao, L. L. Liu, J. Y. Kai, and K. Zhao, “Automated quantitative verification for service-based system design: a visualization transform tool perspective,” International Journal of Software Engineering and Knowledge Engineering, vol. 28, no. 10, pp. 1369–1397, 2018.
View at: Publisher Site | Google Scholar
D. Lee and M. Yannakakis, “Principles and methods of testing finite state machines-a survey,” Proceedings of the IEEE, vol. 84, no. 8, pp. 1090–1123, 1996.
View at: Publisher Site | Google Scholar
F. R. Porto, A. T. Endo, and A. Simao, “Generation of checking sequences using identification sets,” in In 15th International Conference on Formal Engineering Methods (ICFEM), pp. 115–130, Berlin, Heidelberg, October 29-November 1, 2013.
View at: Publisher Site | Google Scholar
K. Hoda and L. Yvan, “On FSM-based testing: An empirical study: complete round-trip versus transition trees,” in 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), pp. 305–315, Toulouse, France, October 23-26, 2017.
View at: Publisher Site | Google Scholar
M. C. Gaudel, “Formal methods for software testing,” in In International Symposium on Theoretical Aspects of Software Engineering (TASE), pp. 1–3, Sophia Antipolis, France, September 13-15, 2017.
View at: Google Scholar
S. R. Liang, Fuzzy Test Algorithm of ZigBee Protocol Based on FSM [M.S. Thesis], BUPT, Beijing, China, 2014.
A. Simão, A. Petrenko, and J. C. Maldonado, “Experimental evaluation of coverage criteria for FSM-based testing,” in In the 21st Simpósio Brasileiro de Engenharia de Software (SBES), pp. 359–374, Joao Pessoa, Brazil, January 13-15, 2007.
View at: Google Scholar
P. Ammann and J. Offutt, Introduction to Software Testing, China Machine Press, BeiJing, 2018.
J. Lee, S. Kang, and P. Jung, “Test coverage criteria for software product-line testing: systematic literature review,” Information and Software Technology, vol. 122, no. 2020, article 106272, 2020.
View at: Publisher Site | Google Scholar
K. EI-Fakih, N. Yevtushenko, and G. Bochmann, “FSM-based incremental conformance testing methods,” IEEE Transactions on Software Engineering, vol. 30, no. 7, pp. 425–436, 2004.
View at: Publisher Site | Google Scholar
J. F. Cutigi, A. Simao, and S. R. Souza, “Reducing FSM-based test suites with guaranteed fault coverage,” The Computer Journal, vol. 59, no. 8, pp. 1129–1143, 2016.
View at: Publisher Site | Google Scholar
C. C. Liu and H. L. Yang, “Testing of data gathering protocol for wireless sensor networks based on FSM model,” Software Guide, vol. 16, no. 9, pp. 14–18, 2017.
View at: Google Scholar
J. D. Lee, J. I. Jung, J. H. Lee, J. G. Hwang, J. H. Hwang, and S. U. Kim, “Verification and conformance test generation of communication protocol for railway signaling systems,” Computer Standards & Interfaces, vol. 29, no. 2, pp. 143–151, 2007.
View at: Publisher Site | Google Scholar
B. Sarikaya and G. V. Bochmann, “Synchronization and specification issues in protocol testing,” IEEE Transactions on Communications, vol. 32, no. 4, pp. 389–395, 1984.
View at: Publisher Site | Google Scholar
Z. B. Wang, H. Zhou, and B. H. Zhao, “Path overlapped method for protocol conformance test generation,” Computer Systems & Applications, vol. 20, no. 7, pp. 47–52, 2011.
View at: Google Scholar
R. E. Miller and S. Paul, “On the generation of minimal-length conformance tests for communication protocols,” IEEE/ACM Transactions on Networking, vol. 1, no. 1, pp. 116–129, 1993.
View at: Publisher Site | Google Scholar
K. Sabnani and A. Dahbura, “A new technique for generating protocol test,” ACM SIGCOMM Computer Communication Review, vol. 15, no. 4, pp. 36–43, 1985.
View at: Publisher Site | Google Scholar
J. Tang, X. Huang, J. Qian, and C. Viho, “A FSM-based test sequence generation method for RPL conformance testing,” in 2013 IEEE International Conference on Green Computing and Communications and IEEE Internet of Things and IEEE Cyber, Physical and Social Computing, pp. 591–597, Beijing, China, August 20-23, 2013.
View at: Publisher Site | Google Scholar
X. Y. Dang, X. J. Yao, D. W. Gong, and T. Tian, “Efficiently generating test data to kill stubborn mutants by dynamically reducing the search domain,” IEEE Transactions on Reliability, vol. 69, no. 1, pp. 334–348, 2020.
View at: Publisher Site | Google Scholar
V. H. Fragal, A. Simao, A. T. Endo, and M. R. Mousavi, “Reducing the concretization effort in FSM-based testing of software product lines,” in 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 329–336, Tokyo, Japan, March 13-17, 2017.
View at: Publisher Site | Google Scholar
P. Liu, Y. D. Li, and Z. J. Li, “Some thoughts on model-based test optimization,” in 2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 268–274, Sofia, Bulgaria, July 22-26, 2019.
View at: Publisher Site | Google Scholar
S. Pradhan, M. Ray, and S. K. Swain, “Transition coverage based test case generation from state chart diagram,” Journal of King Saud University-Computer and Information Sciences, vol. 5, no. 5, pp. 1–10, 2019.
View at: Publisher Site | Google Scholar
T. L. Zhou, H. Y. Sun, J. Liu, X. H. Chen, and D. H. Du, “Improving testing coverage for safety-critical system by mutated specification,” in 2014 21st Asia-Pacific Software Engineering Conference, pp. 43–46, Jeju, South Korea, 2014.
View at: Publisher Site | Google Scholar
A. Denise, M. C. Gaudel, S. D. Gouraud, R. Lassaigne, J. Oudinet, and S. Peyronnet, “Coverage-biased random exploration of large models and application to testing,” International Journal on Software Tools for Technology Transfer, vol. 14, no. 1, pp. 73–93, 2012.
View at: Publisher Site | Google Scholar
R. M. Hierons and H. Ural, “Optimizing the length of checking sequences,” IEEE Transactions on Computers, vol. 55, no. 5, pp. 618–629, 2006.
View at: Publisher Site | Google Scholar
H. Ural and F. Zhang, “Reducing the lengths of checking sequences by overlapping,” in In International Conference on Testing of Communicating Systems, pp. 274–288, New York, NY, USA, May 16-18, 2006.
View at: Google Scholar
L. H. Duan and J. Chen, “Reducing test sequence length using invertible sequences,” in In 9th International Conference on Formal Engineering Methods (ICFEM), pp. 171–190, Boca Raton, FL, USA, November 14-15, 2007.
View at: Google Scholar
G. V. Jourdan, H. Ural, and H. Yenigün, “Reduced checking sequences using unreliable reset,” Information Processing Letters, vol. 115, no. 5, pp. 532–535, 2015.
View at: Publisher Site | Google Scholar
S. Mouchawrab, L. C. Briand, Y. Labiche, and M. Di Penta, “Assessing, comparing, and combining state machine-based testing and structural testing: a series of experiments,” IEEE Transactions on Software Engineering, vol. 37, no. 2, pp. 161–187, 2011.
View at: Publisher Site | Google Scholar
M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments on the effectiveness of data-flow and control-flow-based test adequacy criteria,” in In Proceedings of 16th International conference on Software engineering (ICSE), pp. 191–200, Sorrento, Italy, 1994.
View at: Publisher Site | Google Scholar
L. C. Briand, M. Di Penta, and Y. Labiche, “Assessing and improving state-based class testing: a series of experiments,” IEEE Transactions on Software Engineering, vol. 30, no. 11, pp. 770–783, 2004.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2021 Lin Wei-Wei et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

276

Downloads

479

Citations

Wireless Communications and Mobile Computing

Smart Edge in Intelligent Industrial Systems

A Combining Method for Wireless Protocol Conformance Testing: A Empirical Case

Abstract

1. Introduction

2. Preliminaries

3. TPC-UIO Test Method

3.1. Motivation Example

3.2. Test Generation of TPC-UIO Method

3.2.1. State Covering Set Generation

3.2.2. TPC Test Set Generation

3.2.3. UIO Generation

3.2.4. Test Set Generation of TPC-UIO Method

4. Case Study

4.1. Experimental Subject

4.2. Mutation Generation

4.3. Experimental Setup

4.4. Result Analysis

4.4.1. Comparison of Test Cost

4.4.2. Comparison of Fault Coverage

5. Related Work

6. Conclusion and Future Work

Data Availability

Conflicts of Interest

Acknowledgments

References

Copyright