MAESTRO: Automated test generation framework for high test coverage and reduced human effort in automotive industry
Introduction
The automotive industry has developed automotive software to control various components in the motor vehicle, such as the body control module (BCM), smart-key system (SMK), and tire pressure monitoring system (TPMS) [1], [2]. As automotive software becomes larger and more complex with the addition of newly introduced automated features (e.g., automatic parking system (APRK) of advanced driver assistance system (ADAS)) and more sophisticated functionality (e.g., driving mode systems) [3], [4], the cost of testing automotive software is rapidly increasing. Also, it is difficult for human engineers to develop test inputs that can ensure high-quality automotive software within tight software development schedules and budgets. To resolve these problems, the automotive industry is trying to apply automated software testing/verification techniques [5], [6], [7], [8] to achieve high code quality with reduced human effort.
Concolic testing [9], [10] has been applied to automatically generate test inputs for software in various industries. Concolic testing combines dynamic concrete execution and static symbolic execution to explore all possible execution paths of a target program, which can achieve high code coverage. Concolic testing has been applied to various industrial projects (e.g., flash memory device driver [11], mobile phone software [12], [13], and large-scale embedded software [14]) and has effectively improved the quality of industrial software by increasing test coverage and detecting corner-case bugs with modest human effort. Also, fuzzing is starting to show its potential as a general automated test input generation technique, like concolic testing, although it had been originally developed to reveal security vulnerabilities of target systems.
While we were working to apply automated test generation techniques to automotive software developed by Mobis, we observed the following technical challenges that need to be resolved to successfully apply automated test generation techniques:
- 1.
We need to generate test drivers and stubs carefully to achieve high unit test coverage while avoiding generating test cases corresponding to the executions that are not feasible at the system-level. Otherwise (e.g., generating naive test drivers and stubs that provide unconstrained symbolic inputs to every function in a target program), we will waste human effort to manually filter out infeasible tests that lead to misleading high coverage and false alarms.
- 2.
Current concolic testing tools do not support symbolic bit-fields in C which are frequently used for automotive software.1For example, automotive software uses bit-fields in message packets in the controller area network (CAN) bus to save memory and bus bandwidth. However, most concolic testing tools do not support symbolic bit-fields since a bit-field does not have a memory address (Section 3.4) and most programs running on PCs rarely use bit-fields.
- 3.
Although automotive software uses function pointers to simplify code to dynamically select a function to execute, current automatic test generation techniques and tools do not support symbolic setting for function pointers due to the limitation of SMT (Satisfiability Modulo Theories) solvers of concolic testing and input mutation technique of fuzzing.
The main contributions of this paper are as follows:
- 1.
We have developed MAESTRO which automatically generates the test driver, stubs, and test inputs achieving high coverage for automotive software. MAESTRO applies concolic testing and fuzzing together in an adaptive way (Section 3.5.4) to achieve high coverage.
- 2.
We have identified the technical challenges in applications of automated test input generation to automotive software and describe how MAESTRO resolves them (i.e., task-oriented driver/stub generation (Section 3.3.1), symbolic bit-field support (Section 3.4), and symbolic setting for function pointers (Section 3.3.4)). Thus, this paper can support field engineers in the automotive industry to adopt automated test generation with less trial-and-error.
- 3.
To the authors’ best knowledge, this is the first industrial study that concretely demonstrates reduced human effort (i.e., human effort reduced by 58.8%) by applying concolic testing and fuzzing together in the automotive industry (Section 5.3). Thus, this study can promote the adoption of concolic testing and fuzzing in the automotive industry.
- 4.
This paper shares lessons learned and valuable information for both field engineers in the automotive industry and researchers who develop automated testing techniques (Section 6). For example, we have found that the generation of precise test drivers and stubs is important to increase test coverage (Section 6.3) and concolic testing and fuzzing have different characteristics to achieve test coverage (Section 6.4).
This journal article is an extended version of our prior automated testing framework MAIST [18] as follows:
- 1.
We have extended MAIST [18] to MAESTRO to achieve higher test coverage as follows:
- (a)
MAESTRO uses a hybrid technique of concolic testing and fuzzing as test input generators (Sections 3.5.3 and 3.5.4). The experiment results show that the hybrid approach achieved higher branch and MC/DC coverage than concolic testing or fuzzing alone (Section 5.7). Also, we have discussed the different characteristics of concolic testing and fuzzing (Section 6.4).
- (b)
MAESTRO extended MAIST by generating symbolic stubs that provide more realistic contexts to a target code unit (Section 3.3.3). The experiment results show that MAESTRO’s new symbolic stub increases branch and MC/DC coverage (Section 5.8).
- (a)
- 2.
We have targeted a new module (advanced driver assistance system’s automatic parking (APRK)) as well as the ones in the prior work [18]. APRK is a crucial component for safety and it is highly complex, handling multiple sensors and actuators (Sections 2.1 and 2.2).
- 3.
With the new adaptive hybrid test input generator and the precise symbolic stub generation, MAESTRO achieved 94.2% branch coverage and 82.3% MC/DC coverage for the four target modules. Compared to MAIST, MAESTRO improved 4.1% branch and 5.8% MC/DC coverage.
- 4.
We have added new sections, Sections 3.4.1–3.4.3, to describe MAESTRO’s bit-field transformation algorithms and related examples.
Section snippets
Overview
Advanced Driver Assistance System (ADAS) is a vehicle monitoring and control system that prevents or reduces damage of car accidents. ADAS developed by Hyundai Mobis consists of automatic parking system, driver monitoring system which warns a driver in drowsiness, and so on. We target the automatic parking system (APRK) because APRK is one of the most complex features in ADAS and, thus, needs intensive testing. APRK takes information from 12 ultra sonic sensors and one camera, and controls the
Overview
Fig. 4 (a) overviews MAESTRO(Mobis Automated tESTing fRamewOrk), which takes C source code files as inputs. MAESTRO consists of the three components: test harness generator, converter, and test input generator. First MAESTRO harness generator analyzes the input C source files and generates test driver and stub functions for every task in the source files (Section 3.3). MAESTRO converter transforms the C code that uses bit-fields into semantically equivalent one that does not use bit-fields (
Industrial case study: Applying MAESTRO to APRK, BCM, SMK, and TPMS
We have developed and applied MAESTRO to APRK (automatic parking system), BCM (body control module), SMK (smart-key system), and TPMS (tire pressure monitoring system) from October 2017 to March 2019 as explained in the following subsections.
RQ1. Effectiveness of the automated test generation
Table 2 shows the number of the generated test inputs, execution time, and branch and MC/DC coverage of APRK, BCM, SMK, and TPMS achieved by MAESTRO. MAESTRO generated 1,945,720 test inputs in 27.2 h on three machines (i.e., on 12 cores), which achieved 94.2% branch coverage and 82.3% MC/DC coverage on the target modules.
MAESTRO achieved 100% branch and 100% MC/DC coverage of 76.5% (=3116/4072) and 60.8% (=2474/4072) of all functions in the target modules, respectively. Also, MAESTRO achieved
Practical benefit of automated test generation in the automotive industry
As Sections 5.1 and 5.3 show, an automated test generation technique like MAESTRO can improve the quality of automotive software by achieving high test coverage (i.e., 94.2% branch coverage) with reduced testing cost (i.e., 58.8% man-month per year on coverage testing) in practice. Although it is not trivial to develop an automated test generation framework that resolves various technical challenges in industrial projects, we believe that the automotive industry can significantly benefit from
Concolic testing
Concolic techniques can be grouped into instrumentation-based one and Virtual machine (VM)-based one.
Instrumentation-based concolic testing techniques insert probes in target source code to obtain dynamic execution information to build symbolic path formulas. This approach is lighter and easier-to-customize than the VM-based one. However, it requires complex source code parsing and instrumentation. CUTE [9], DART [10], CREST [27] (and its distributed version SCORE [33]), CROWN [23] target C
Conclusion and future work
We have presented the industrial study of applying MAESTRO to the automotive software developed by Mobis. After we identified and addressed the technical challenges of applying automated test generation to automotive software, we have developed an automated test generation framework MAESTRO. It generates a task-oriented test driver and stubs to reduce infeasible test executions and supports bit-fields input generation, input setting for function pointers that automotive software uses, and
Acknowledgments
We thank Ahcheong Lee and Hyunwoo Kim for their initial effort of applying MAESTRO to IBU. This research has been supported by Hyundai Mobis, Next-Generation Information Computing Development Program through NRF funded by MSIT (No. NRF-2017M3C4A7068177), Basic Science Research Program through NRF funded by MSIT (NRF-2019R1A2B5B01069865), and Basic Science Research Program through NRF funded by the Ministry of Education (NRF-2017R1D1A1B03035851).
References (69)
- et al.
J3 model: a novel framework for improved modified condition/decision coverage analysis
Comput. Stand. Interfaces
(2017) - et al.
GECOJAP: a novel source-code preprocessing technique to improve code coverage
Comput. Stand. Interfaces
(2018) - et al.
Scaling modified condition / decision coverage using distributed concolic testing for Java programs
Comput. Stand. Interfaces
(2018) Challenges in automotive software engineering
ICSE
(2006)- et al.
Engineering automotive software
Proc. IEEE
(2007) - et al.
Deep learning in automotive software
IEEE Softw.
(2017) - et al.
Future automotive architecture and the impact of IT trends
IEEE Softw.
(2017) - et al.
Requirements and architectures for secure vehicles
IEEE Softw.
(2016) - et al.
Formalization and verification of AUTOSAR OS standard’s memory protection
TASE
(2018) - et al.
A reusable framework for modeling and verifying in-vehicle networking systems in the presence of CAN and flexray
APSEC
(2017)
A configurable v&v framework using formal behavioral patterns for OSEK/VDX operating systems
JSS
CUTE: A concolic unit testing engine for C
ESEC/FSE
DART: Directed automated random testing
PLDI
Concolic testing of the multi-sector read operation for flash storage platform software
FACJ
Industrial application of concolic testing on embedded software: case studies
ICST
Industrial application of concolic testing approach: a case study on libexif by using CREST-BV and KLEE
ICSE
Automated unit testing of large industrial embedded software using concolic testing
ASE
Concolic testing for high test coverage and reduced human effort in automotive industry
ICSE
Do automatically generated unit tests find real faults? An empirical study of effectiveness and challenges
Automated Software Engineering (ASE)
1600 faults in 100 projects: automatically finding faults while achieving high coverage with evosuite
Empir. Softw. Eng.
Search-based system testing: high coverage, no false alarms
ISSTA 2012
Precise concolic unit testing of C programs using extended units and symbolic alarm filtering
ICSE
LLVM: a compilation framework for lifelong program analysis & transformation
CGO
CIL: intermediate language and tools for analysis and transformation of C programs
CC
Heuristics for scalable dynamic test generation
ASE
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs
OSDI
Pathcrawler: automatic generation of path tests by combining static and dynamic analysis
EDCC
A survey of symbolic execution techniques
ACM Comput. Surv.
Cited by (2)
Formal Methods and Validation Techniques for Ensuring Automotive Systems Security
2023, Information (Switzerland)