Next Article in Journal
An Integrated Strategy for Autonomous Exploration of Spatial Processes in Unknown Environments
Next Article in Special Issue
Fault-Tolerant Model Predictive Control Algorithm for Path Tracking of Autonomous Vehicle
Previous Article in Journal
Laboratory Evaluations of Correction Equations with Multiple Choices for Seed Low-Cost Particle Sensing Devices in Sensor Networks
Previous Article in Special Issue
Towards Simultaneous Actuator and Sensor Faults Estimation for a Class of Takagi-Sugeno Fuzzy Systems: A Twin-Rotor System Application
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 20
Issue 13
10.3390/s20133662
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Fault Detection, Isolation, Identification and Recovery (FDIIR) Methods for Automotive Perception Sensors Including a Detailed Literature Survey for Lidar

by
Thomas Goelles
*,
Birgit Schlager
and
Stefan Muckenhuber
VIRTUAL VEHICLE Research GmbH, Inffeldgasse 21a, 8010 Graz, Austria
*
Author to whom correspondence should be addressed.
Sensors 2020, 20(13), 3662; https://doi.org/10.3390/s20133662
Submission received: 19 May 2020 / Revised: 15 June 2020 / Accepted: 24 June 2020 / Published: 30 June 2020
(This article belongs to the Special Issue Sensors Fault Diagnosis Trends and Applications)
Browse Figures
Versions Notes

Abstract

:
Perception sensors such as camera, radar, and lidar have gained considerable popularity in the automotive industry in recent years. In order to reach the next step towards automated driving it is necessary to implement fault diagnosis systems together with suitable mitigation solutions in automotive perception sensors. This is a crucial prerequisite, since the quality of an automated driving function strongly depends on the reliability of the perception data, especially under adverse conditions. This publication presents a systematic review on faults and suitable detection and recovery methods for automotive perception sensors and suggests a corresponding classification schema. A systematic literature analysis has been performed with focus on lidar in order to review the state-of-the-art and identify promising research opportunities. Faults related to adverse weather conditions have been studied the most, but often without providing suitable recovery methods. Issues related to sensor attachment and mechanical damage of the sensor cover were studied very little and provide opportunities for future research. Algorithms, which use the data stream of a single sensor, proofed to be a viable solution for both fault detection and recovery.

1. Introduction

Advancing the level of automation for vehicles is a major challenge in today’s automotive industry. Automated vehicles are expected to provide great benefits for the driver and enable new transportation use cases and applications, e.g., [1]. Improving passenger safety is among the key arguments for the development of automated vehicles. Today, more than 1.35 million people die in road traffic crashes each year, making road traffic crashes the leading cause of death among children and young adults between 5 and 29 years of age [2]. Advanced driver assistance system (ADAS) and automated driving (AD) functions have the potential to reduce this number significantly, since most car accidents are traceable to a human error [3,4,5].
The Society of Automotive Engineers (SAE) classifies six levels of driving automation [6]. Currently, available vehicles provide up to SAE level 2 “partial driving automation”, where an ADAS function can take over lateral and longitudinal vehicle motion control. Examples are Cadillac’s Super Cruise, Mercedes-Benz’s Drive Pilot, Nissan’s ProPILOT Assist, Tesla’s Autopilot, and Volvo’s Pilot Assist. Vehicles that fulfil SAE level 3 “conditional driving automation” must provide an AD function that allows the driver to remove his attention off the road and only intervene when the system requests. In this case, the vehicle is responsible for object and event detection and proper response. Going from SAE level 2 to level 3+ implies that the responsibility for environment perception is transferred from the driver to the vehicle.
Automated vehicles work according to the SENSE-PLAN-ACT cycle [1]. SENSE represents everything related to environment perception. Within PLAN the necessary decisions are made based on the information from SENSE. ACT includes the controlling of steering, acceleration, and deceleration of the vehicle based on commands from PLAN. This chain of command illustrates how important a reliable environment perception system is for every automated vehicle. In particular, SAE level 3+ vehicles, that cannot rely on a human driver monitoring the environment constantly, have very high demands in terms of robustness and reliability. A combination of diverse and redundant sensor types is required to provide a robust environment perception during all possible environmental conditions. A sensor set including camera, radar (radio detection and ranging), and lidar (light detection and ranging) sensors is considered the best option to fulfil the SENSE demands of level 3+ vehicles [7] eventually.
Sensor faults, either caused by internal malfunctions (e.g., broken antennas) or disturbing external factors (e.g., adverse weather conditions), propose a considerable danger for each automated vehicle’s anticipated functionality. This can be health and even life-threatening for passengers in the vehicle and people in the proximity of automated vehicles. A standard method to tackle sensor faults of critical systems in the aviation or military domain is sensor redundancy. However, in the automotive industry, the cost factor plays a significant role and therefore proposes a limiting factor to sensor redundancy.
To address sensor faults in automated vehicles in a cost effective manner, sensor fault detection, isolation, identification, and recovery (FDIIR) systems, e.g., [8] can be included into each individual perception sensor that is contributing to the SENSE-PLAN-ACT cycle (Figure 1). The sensor FDIIR system constantly monitors the activity of the respective perception sensor for correct operation. In case of a detected sensor fault, the system intervenes by preventing erroneous information reaching the PLAN step, and tries to recover the affected sensor (e.g., a wiper removes dirt from the optical aperture of the sensor). Additionally, FDIIR systems can provide sensor performance information to the ADAS/AD function (e.g., adverse weather conditions are currently reducing the sensors field of view (FOV) by 50 %) to increase the information content for the PLAN step.
Two EU research projects are currently addressing faults of automotive perception sensors under unfavourable environmental conditions. The RobustSENSE project [9] aims to develop a system performance assessment module, which shall monitor the reliability of every perception sensor in the vehicle under adverse weather conditions. The DENSE project [10] approaches the problem by developing a fused sensor set that combines radar, gated short-wave infrared camera, and short-wave infrared lidar to provide a reliable perception under challenging conditions.
Sensor FDIIR systems are equally important for all types of automotive perception sensors, i.e., camera, radar, and lidar. This publication introduces a classification schema for faults, detection methods and recovery methods that applies to all perception sensor types in the automotive domain.
The literature survey and state-of-the-art analysis in this publication put the focus on FDIIR systems for lidar sensors, since lidar is a recently emerging sensor for automotive applications, and there is far less experience in the automotive industry with lidar compared to radar and camera. Today’s level 2 vehicles typically rely on radar and camera sensors to perceive the environment [7]. However, lidar sensors are expected to play an essential role in level 3+ vehicles, since lidars provide a three-dimensional depth map with very high angular resolution [11,12]. High prices of mechanically spinning lidars are currently a limiting factor, but costs will decrease significantly with new technologies such as optical phased array, MEMS-based mirrors, etc. [11,13]. For example, Druml et al. [14] introduced a lidar prototype that shall eventually lead to commercially available lidar sensors with a range of more than 200 m for costs less than USD 200.
The main objectives of the presented study are the following:
  • Identify all types of faults, detection methods, and recovery methods for automotive perception sensors and develop a corresponding classification schema.
  • Evaluate the state of the art of FDIIR methods for automotive lidar.
  • Explain, discuss, and compare the most promising existing FDIIR methods for automotive lidar.
  • Identify research opportunities related to FDIIR methods for automotive lidar.
The publication is structured as follows: Section 2 introduces a new fault classification for automotive perception sensors and relates the fault classes to current and upcoming international safety standards for automated vehicles. Section 3 introduces a corresponding classification schema for fault detection methods and Section 4 introduces a corresponding classification schema for fault recovery methods. Section 5 describes the methods applied in the literature survey on lidar FDIIR methods that is presented in Section 6. Section 7 completes the paper with a discussion and conclusion and gives an outlook on future work.
A glossary of the technical terms can be found at the end of this paper.

2. Classification of Faults of Perception Sensors

Faults can occur on different levels in an automotive perception system. Ranging from the whole perception system, including the sensor fusion layer, to the individual sensor and its components. Faults on each level need different detection and recovery strategies. In this review, we focus on the individual sensor and its subcomponents, disregarding the perception algorithms that extract information based on raw data from individual sensors or perception systems consisting of several sensors. Therefore, we consider faults that occur until the raw data of the sensor is generated before any optional object detection and classification algorithm is applied (Figure 2).
Different faults occurring in perception sensors may have a similar cause. Therefore, perception sensor faults can be classified. A classification of faults is useful when it comes to the design of fault detection, fault isolation, fault identification, and fault recovery algorithms since similar faults may benefit from similar or equal algorithms. We suggest a classification into the following fault classes: defect subcomponent, mechanical damage to sensor cover, layer on sensor, mounting issue, security attack, unfavourable environmental condition, and crosstalk (see Figure 2). Table 1 lists the seven fault classes including exemplary faults and international safety standards addressing the faults.
A defect subcomponent is a faulty internal part of the sensor, e.g., a defect transmitter, a defect receiver, or a defect processing unit. A mechanical damage to the sensor cover may be scratches, cracks, or holes in the sensor cover but also a missing sensor cover or a deformation of the sensor cover count to this fault class. A layer on the sensor cover like dirt, water, ice, salt, or snow can cause that objects are not detected (false negatives), that not existing objects are detected (false positives) or that objects are misclassified. Mounting issues may be a change of the sensor’s position or vibrations while driving. A further risk are security attacks as shown in, e.g., [15,16]. This may be a security attack by another active electromagnetic source, i.e., denial of service and false data injection, or an electronics hack over a wired or wireless connection to the sensor. Furthermore, a perception sensor has limitations under different unfavourable environmental conditions, which can cause, e.g., a reduction of the field of view. Other active perception sensors may cause a fault in the sensor data because of crosstalk, as described, e.g., in [17,18]. The difference to security attacks with laser hacks is that crosstalk occurs accidentally, and security attacks occur by malicious hacking.

International Safety Standards Addressing Faults of Perception Sensors

The international standard ISO 26262 [19] and the international specification ISO PAS 21448 [20] represent the state-of-the-art for safety standardization of road vehicles. FDIIR methods play an important role to fulfil the current and upcoming requirements of ISO 26262 and ISO PAS 21448 for automotive perception sensors. This section explains, how the above introduced fault classification schema (Section 2) relates to the ISO 26262 and ISO PAS 21448.
The ISO 26262 “Road vehicles—Functional safety” from 2011 includes safety measures and safety mechanisms to ensure functional safety of electrical and electronic systems of road vehicles. The standard differentiates between systematic faults and random hardware faults and provides a systematic development framework for electrical and electronic systems. The Automotive Safety Integrity Level (ASIL) is defined in this standard. This level defines which requirements and measures have to be considered and implemented to avoid an unreasonable risk [19]. Concerning FDIIR methods for automotive perception sensors, the ISO 26262 especially applies to the sensor fault class defect subcomponent listed in Table 1. This fault class affects the sensor itself and is not caused by sensor limitations due to the environment.
The ISO PAS 21448 “Road vehicles—Safety of the intended functionality” addresses limitations of systems where no fault, according to ISO 26262, occurred. The letter code PAS implies that this is a publicly available specification and not a standard at that point in time. The rework of the PAS version to an international standard has started in the beginning of 2019, and it is planned to be released as an international standard by the end of 2021. The ISO PAS 21448 especially applies to the limitations of systems that are based on sensing the environment. Such limitations can be caused by, e.g., different environmental conditions or other road users. The Annex F of the standard lists such limitations [20]. The sensor fault classes mechanical damage to sensor cover, layer on sensor cover, mounting issue, security attack, unfavourable environmental condition, and crosstalk listed in Table 1 address the issues described in the ISO PAS 21448. These sensor fault classes are caused by triggering conditions, which are combinations of sensor weaknesses in the design or specific technology or negative influencing factors of the environment.

3. Classification of FDII Methods for Perception Sensors

Fault detection, isolation, and identification (FDII) methods determine whether a fault occurred in a system, where the fault occurred, and how severe the fault is. The term “fault diagnosis” is often used as a synonym, e.g., in Chen and Patton [21], Chen et al. [22], Piltan and Kim [23].
The fault detection unit or fault detection algorithm of a perception sensor determines whether a fault occurred in the perception sensor or not. If the fault detection unit decides that a fault occurred, the fault isolation unit identifies where the fault occurred and which part of the sensor is faulty. When the location of the fault is determined, the fault identification unit determines the severity of the fault and the level of sensor degradation.
To categorize FDII methods for perception sensors, we suggest a classification into the following classes (Table 2): comparison to sensor model, monitoring sensor output, comparison to static ground-truth, comparison to dynamic ground-truth, comparison to other sensor of same type, comparison to other sensor of different type, monitoring internal interface, and comparison of multiple interfaces.
The class comparison to sensor model includes the comparison of real sensor measurements with the output of a sensor model (Figure 3a). In general, comparing a process with its model is a very common method in FDII literature for simple systems like bearings. Monitoring a perception sensor with this method requires very detailed knowledge about the environment and a sensor model.
The class monitoring sensor output refers to methods that use only the output data of the sensor under observation (Figure 3b). The monitoring algorithm determines whether the sensor is healthy or faulty, the fault’s location, and the severity of the fault solely based on the sensor’s output.
The class comparison to static ground-truth identifies faults based on comparing the sensor’s output with static ground-truth data (Figure 3c). Static ground-truth may be immobile objects (houses, traffic signs, …) stored in a map or well-defined test targets placed at known locations specifically for sensor assessment.
The class comparison to dynamic ground-truth identifies faults based on comparing the sensor’s output data with dynamic ground-truth data. Dynamic ground-truth may be obtained by connected infrastructure in terms of vehicle to everything (V2X), e.g., vehicles that collect and broadcast data with their own sensors (Figure 3d). Furthermore, the position and the geometry of other vehicles may be sent from vehicle to vehicle (V2V) to check if the sensor on the ego-vehicle perceives the right position and geometry of the other vehicle.
The class comparison to other sensor of same type identifies faults based on comparing the output of two sensors of the same type, e.g., two lidar sensors, two radar sensors, or two camera sensors, with overlapping FOV (Figure 3e).
Contrary, the class comparison to other sensor of different type identifies faults based on comparing the output of two sensors of different types, e.g., lidar compared to radar, with overlapping FOV (Figure 3f). Therefore, an algorithm that transforms the data of one sensor to comparable data of the other sensor has to be applied.
The class monitoring internal interface identifies faults based on monitoring the output of a single internal sensor interface (Figure 3g). The monitoring algorithm determines whether the sensor is healthy or faulty, the location of the fault, and the severity of the fault based on the output of a single sensor interface.
The class comparison of multiple interfaces identifies faults based on comparing the output of two interfaces of one sensor (Figure 3h). The subcomponents between the two interfaces are modelled. A fault can be detected by comparing the model’s output at a specific interface with the real output of the same interface.

4. Classification of Recovery Methods for Perception Sensors

If the FDII system detects a fault, a fault recovery method can be applied to eliminate or mitigate the fault and restore the functionality of the perception sensor and thereby the system that relies on the sensor.
To categorize fault recovery methods for perception sensors, we suggest a classification into the following classes: software adaption, hardware adaption, temperature regulation, and cleaning of sensor cover. Table 3 shows fault recovery classes and respective exemplary methods.
Software adaption as a recovery class includes algorithms that are either applied on the sensor output data to mitigate faults directly or that are applied in the processing chain that uses the sensor data, for example, filtering of the point cloud for unwanted signals from precipitation.
Similarly to software adaption, it is also possible to apply hardware adaption, if required. Hardware adaption can be adjusting the sensor’s alignment after a change of the sensor’s position, or aligning the sensor cover if only a part of the sensor cover is faulty.
A further recovery class is temperature regulation. This can be heating if the sensor is covered by an ice layer, for example, or cooling to prevent overheating of the sensor.
The last recovery class, cleaning of sensor cover, is required in case of a disturbing layer on the sensor cover. For example, wipers may be used to remove dirt or water from the sensor cover.

5. Literature Survey Methodology

This section explains the methods used for the literature survey, presented in the following Section 6. Relevant publications addressing faults, fault detection methods, and recovery methods for automotive lidars were identified, analysed and then classified in a quantitative approach based on the PRISMA scheme, e.g., [24,25]. The presented literature survey methodology can also be applied for literature studies on other perception sensors.
Figure 4 shows the literature search procedure adopted for identifying relevant studies. We selected the interdisciplinary databases ISI Web of Science (also known as Web of Knowledge), the engineering database of IEEE, and the patent search engines of Google and depatisnet. Further, we also included a general Google search to cover grey literature and websites in addition to the research articles, conferences, and patents. The search was conducted between August and December of 2019; therefore we included sources published between 1900 and 31 December 2019.
After the initial search, we removed duplicates and missing full texts. In the next stage, we assessed the documents for eligibility. Here, we excluded records that were outside the focus of this study, i.e., records which did not mention any of the following: a fault, a detection method, or a recovery method. In terms of recovery methods, the focus was put on publications, where the sensor itself can perform the recovery method. This means that the following recovery solutions were excluded: recovery in the form of functional redundancy, an adaption of sensor fusion, direct or hardware redundancy, or fall back to a safe state like “limp home” and shut down.
In the next stage the literature was categorized according to the classification schema introduced in Table 1, Table 2 and Table 3 and the results were stored in a spreadsheet. Followed by filtering oft records which did not explicitly deal with lidar, leaving 95 records which were further analysed.
The records were stored in a BiBteX file and were analysed together with the spreadsheet file with Python. The bibliography processor Pybtex (https://pybtex.org) was used to parse the BibTeX files. The BibTeX key was used to link the BiBteX file with the spreadsheet. For further details, please refer to the supplements, which include the BibTeX file, the spreadsheet, and the saved Web of Science search queries.

6. Literature Survey on FDIIR Methods for Automotive Lidar

This section provides an overview of the state-of-the-art of fault detection, isolation, identification, and recovery methods for automotive lidar and their faults. We identified and collected 95 highly relevant publications. Out of those publications the majority were conference papers, followed by patents and journal articles, as Figure 5a shows. Of those publications, the majority were published in the last five years, as can be seen in panel Figure 5b. Since 2014, there has been a steady increase in publications until 2019. This increase is most likely due to the recent focus of the automotive industry on lidar technology.

6.1. Fault Classes of Automotive Lidar

First, we classified the literature according to the fault classes. Table 4 and Figure 6 give an overview of the mentioned faults and fault classes in the literature. Faults belonging to the class unfavourable environmental condition are mentioned in 46 documents, making it the class with the most entries. Followed by layer on sensor cover and security attack. Mounting issues are covered in two documents and mechanical damage to sensor cover only in one. The total number of records is larger than the number of publications because one publication can present several fault classes.

6.2. FDII Classes and Realizations for Automotive Lidar

Figure 7 shows the number of mentions per FDII class according to the definitions in Section 3. The most common method of FDII is monitoring the output of a single sensor. Detecting faults by comparing different sensors has been described 12 times, which makes it the second most common method. FDII by monitoring internal interface was never mentioned.
In the comparison to sensor models method, residuals are calculated between the output of the sensor and a sensor model. Faults are indicated when the residual is above a threshold value. For example, a general observer scheme with extended Kalman filters is described by Huang and Su [95]. Several publications describe sensor fault detection based on monitoring sensor output with different realizations. For example, Tascione and Bode [101] detects lidar faults by checking if spin rate, laser power, laser frequency, and photodetector alignment are in its tolerance ranges. Jokela et al. [84] evaluate the standard deviation of range measurements under different environmental conditions. Some publications like Leslar et al. [99] and Segata et al. [26] describe outlier detection. Filters or machine learning approaches may be applied to sensor data to suppress outliers. For example, a supervised learning method is mentioned by Breed [98], which used pattern recognition algorithms and neural networks. Furthermore, James et al. [33] used deep neural networks to classify sensor contaminations. An alternative method used for lidar fault detection is the comparison to static ground-truth which is described in Zhu and Ferguson [105] and Bruns and Venator [104]. As described in these publications, static ground-truth may be landmarks, road markings, or terrain maps. Contrary to using static ground-truth as reference data, some publications describe how to implement the comparison to dynamic ground-truth. Dynamic ground-truth is typically received from other vehicles in the vicinity over V2V or by servers over V2X as described by Mehrdad [106] and Ghimire and Prokhorov [107]. The three patents described in Zhu et al. [108], Zhu et al. [110] and Kim et al. [109] utilize the comparison to sensor of same type. A distance function is used to decide whether an error occurred or not. For example, in Kim et al. [109] a fault is detected if the Mahalanobis distance exceeds a specified threshold. Kim et al. [109] detects the fault first and then recalibrates the sensor whereas Zhu et al. [108] and Zhu et al. [110] describe no further reactions after fault detection. Another method is the comparison to a different type of sensor, which is typically more complex, since the data format has to be transformed before comparison. Dannheim et al. [52] use a camera and a lidar to detect and classify unfavourable environmental conditions like snow, rain, and fog. Daniel et al. [62], as another example, uses a radar, a lidar, and a stereo camera under foggy conditions. Other publications like Choi et al. [46] and Guo et al. [43] describe how to suppress security attacks by comparing between sensors of different types.

6.3. Recovery Methods for Automotive Lidar

In terms of recovery methods for automotive lidar, software adaption was used the most and for a variety of faults (see Figure 8). Algorithms were used to mitigate security hacks, crosstalk, and filter out effects of unfavourable environmental conditions. In addition, software adaption methods were presented without going into detail about the fault, indicated as the “n/a” category in Figure 8. Hardware adaption was only used once to stabilize a lidar on a tractor [39]. Temperature adaption was used to recover from faults caused by a layer on the sensor cover and to keep the temperature within specifications. Naturally, cleaning of sensor cover recovery methods were applied solely when the fault was caused by an unwanted layer on the sensor cover.
The recovery methods used for each individual exemplary fault shown in Table 1, are shown in Figure 9. The majority of publications, which discussed one or more faults, did not cover a suitable recovery method. Especially, faults of the class unfavourable environmental condition and defect subcomponent lacked suitable recovery methods.
Defect subcomponents are faults that are directly addressed by the manufacturer, and information is not readily available. Only some documentations of data protocols reveal hints about the internals. For example, in Ibeo [28], where error and warnings indicate internal fault detection.
Mechanical damage to the sensor cover by scratches was described in Rivero et al. [29] without a recovery method. However, cover related issues mostly dealt with a layer on the sensor cover.
Disruptive layers on the cover were mostly removed mechanically by air nozzles, spraying fluid, and wiping. Other publications approached the problem by shaking the unit [34] or by heating of the cover [36].
Only vibration was discussed as a mounting issue in the literature in Periu et al. [39]. They introduced an active stabilization mechanism to mitigate vibration effects on an agricultural vehicle. Changes of the sensor’s position were not covered in the literature.
Security attacks were addressed via prevention or software adaption. Dutta [47] developed an algorithm based on the least square estimator to mitigate attacks such as a denial of service and false data injection. Petit et al. [44] mitigated attacks with, besides redundancy and probing multiple times, algorithms based on increasing the difficulty for the hacker to hit the right time-window for attacks.
From the fault class unfavourable environmental condition rain and fog were mentioned the most (Figure 8). Nevertheless, only two publications [74,75] applied an algorithm to mitigate its negative effects. Both of them used a filter to minimize reflections from the particles in the air. Peynot et al. [75] use a Bayes filter and Zhang et al. [74] use a histogram and particle filter. Besides algorithms, temperature adaption methods are used to recover from environmental hazards. These are used to keep the sensor temperature within the specified range. Sick [86] used heating, while Wilson and Sinha [67] and Tegler [30] used cooling to keep the temperature within specifications.
Crosstalk by another automotive lidar was investigated by Kim et al. [17] and Zhang et al. [18]. They presented an adaptive algorithm to distinguish between target echo and crosstalk. There are also several patents, e.g., [89,91,92,93,94,117], which address methods for preventing crosstalk.

7. Discussion and Conclusions

7.1. Limitations of the Literature Study

Literature that might have used a different terminology could have been overlooked during the search, as well as methods buried deep in the text of patents. Furthermore, the interpretation of the classification of the FDII class is bound to some subjective judging. Nevertheless, the remaining 95 records have been read and interpreted carefully and analysed quantitatively.

7.2. Faults and Fault Classes

The presented lists of fault classes and, in particular, the exemplary faults are certainly not complete, but they are meant to cover all major issues which are of importance to reach the next step in terms of reliable environment perception. Out of our defined fault classes, unfavourable environmental conditions have been studied by far the most for lidar sensors, likely due to its frequent occurrence and its strong impact on the lidar perception principle.
In an alternative classification approach, faults could be classified according to the underlying physics. This would potentially allow common fault identification, isolation, and recovery methods per class. For example, the faults snow, dust, rain, smoke, and fog all can be described by scattering and multiple scattering. In more detail, the relevant description depends on the relationship between the particle size and the laser wavelength. Depending on the laser source, the wavelength is between 0.8 and 1.5 micrometers. Mie scattering can be used if the particle is of similar size compared to the wavelength, e.g., [118,119,120]. If the particle is smaller than the laser wavelength, Rayleigh scattering can be used. Otherwise, if the particle is much larger than the wavelength like it is the case for hail, non-selective scattering and absorption formulation is suitable. Scattering and absorption are not only relevant for precipitation, but also for issues related to the sensor cover.

7.3. Fault Detection, Isolation, and Identification

Fault detection, isolation, and identification methods based on monitoring the data of a single sensor are attractive since they are independent of other input sources. They can be directly implemented on the sensor firmware and are capable of detecting faults throughout all fault classes.
Third-party development of fault detection, isolation, and identification methods based on internal interfaces are hindered by the manufactures which are very protective of their intellectual property. This is especially the case for automotive lidar as the market is rather new, and competition is high.

7.4. Recovery Methods

Software adaption is the most commonly described recovery class and often implemented in the firmware of the sensor by the manufacturers. Algorithms for recovery can be applied directly to the raw data, before object detection.

7.5. Research Opportunities

Based on the presented survey, the following research opportunities can be identified:
  • Mounting issues and mechanical damage to the sensor cover have been studied very little.
  • Comparison to static and dynamic ground truth are very promising FDII methods. Still, these methods require more research to evaluate which and how ground truth information can be accessed by the FDIIR system.
  • Although unfavourable environmental conditions have been discussed in many publications, possible recovery methods for this fault class were typically absent.
  • Literature on defective subcomponents is sparse and no suitable recovery methods were described. This is most likely because commercial lidar units are provided as “black box”, because manufacturers are very protective of their intellectual properties. This calls for an open standard in order to access interfaces between subcomponents.
  • Due to similar reasons, fault detection, isolation, and identification methods that access internal interfaces are mentioned only once in the literature.
  • Detailed investigations of first principles of root causes for faults could lead to fault detection and recovery methods that apply for a wide range of faults.
  • Hardware adaption has been studies very little and might be well suited for other faults related to mounting issues.

7.6. Closing Remarks

The introduced classification schema for faults, detection methods, and recovery methods, as well as the literature survey methodology and many of the presented fault detection and recovery methods, are not restricted to the automotive domain, but may also be applied to other domains such as aviation and maritime. The presented FDIIR classification and methods can be relevant for any automated system that requires a very reliable environment perception system (i.e., automated drones, automated trains, automated ships, automated airplanes, …).
In the last five years, an increasing interest in automotive perception sensors, especially lidar, and associated faults and recovery methods has been observed. However, there is still much to be done to offer reliable perception for ADAS/AD functions. Fault detection, isolation, identification, and recovery methods for automotive lidar are still at their infancy.

Author Contributions

Conceptualization, T.G., B.S., S.M.; methodology, T.G., B.S., S.M.; software, T.G.; formal analysis, T.G., B.S.; investigation, T.G., B.S.; data curation, T.G., B.S.; writing–original draft preparation, T.G., B.S., S.M.; writing–review and editing, T.G., B.S., S.M.; visualization, T.G., S.M.; supervision, T.G., S.M.; project administration, T.G., S.M.; funding acquisition, S.M.; All authors have read and agreed to the published version of the manuscript.

Funding

This research received financial support within the COMET K2 Competence Centers for Excellent Technologies from the Austrian Federal Ministry for Climate Action (BMK), the Austrian Federal Ministry for Digital and Economic Affairs (BMDW), the Province of Styria (Dept. 12) and the Styrian Business Promotion Agency (SFG).

Acknowledgments

The publication was written at Virtual Vehicle Research GmbH in Graz, Austria. The authors would like to acknowledge the financial support within the COMET K2 Competence Centers for Excellent Technologies from the Austrian Federal Ministry for Climate Action (BMK), the Austrian Federal Ministry for Digital and Economic Affairs (BMDW), the Province of Styria (Dept. 12) and the Styrian Business Promotion Agency (SFG). The Austrian Research Promotion Agency (FFG) has been authorised for the programme management. The authors would furthermore like to express their thanks to Helmut Martin for a helpful conversation about ISO standards. The security icon by Vitaly Gorbachev is available at flaticon.com.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Abbreviations

ADAutomated Driving
ADASAdvanced Driver-Assistance Systems
ASILAutomotive Safety Integrity Level
FDIIRFault Detection Isolation Identification Recovery
FOVField Of View
MEMSMicroElectroMechanical Systems
SAESociety of Automotive Engineers
V2VVehicle-to-vehicle communication
V2XVehicle-to-everything communication

References

  1. Watzenig, D.; Horn, M. (Eds.) Automated Driving—Safer and More Efficient Future Driving; Springer International Publishing: Cham, Switzerland, 2017. [Google Scholar] [CrossRef]
  2. World Health Organisation. Global Status Report on Road Safety 2018; Technical Report; World Health Organisation: Geneva, Switzerland, 2018. [Google Scholar]
  3. Anderson, J.; Kalra, N.; Stanley, K.; Sorensen, P.; Samaras, C.; Oluwatola, T. Autonomous Vehicle Technology: A Guide for Policymakers; RAND Corporation: Santa Monica, CA, USA, 2014. [Google Scholar]
  4. Fagnant, D.J.; Kockelman, K. Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations. Transp. Res. Part A Policy Pract. 2015, 77, 167–181. [Google Scholar] [CrossRef]
  5. Thomas, P.; Morris, A.; Talbot, R.; Fagerlind, H. Identifying the causes of road crashes in Europe. Ann. Adv. Automot. Medicine. Assoc. Adv. Automot. Med. Annu. Sci. Conf. 2013, 57, 13–22. [Google Scholar]
  6. SAE International. Taxonomy and Definitions for Terms Related to Driving Automation Systems for on-Road Motor Vehicles; SAE International: Warrendale, PA, USA, 2018. [Google Scholar] [CrossRef]
  7. Marti, E.; de Miguel, M.A.; Garcia, F.; Perez, J. A Review of Sensor Technologies for Perception in Automated Driving. IEEE Intell. Transp. Syst. Mag. 2019, 11, 94–108. [Google Scholar] [CrossRef]
  8. Isermann, R. Fault-Diagnosis Systems: An Introduction from Fault Detection to Fault Tolerance; Springer: Berlin, Germany, 2006. [Google Scholar]
  9. RobustSENSE. Available online: http://www.robustsense.eu/ (accessed on 12 January 2020).
  10. Dense. Available online: https://dense247.eu (accessed on 17 October 2019).
  11. Hecht, J. Lidar for Self-Driving Cars. Opt. Photonics News 2018, 29, 26–33. [Google Scholar] [CrossRef]
  12. Thakur, R. Scanning LIDAR in Advanced Driver Assistance Systems and Beyond: Building a road map for next-generation LIDAR technology. IEEE Consum. Electron. Mag. 2016, 5, 48–54. [Google Scholar] [CrossRef]
  13. Warren, M.E. Automotive LIDAR Technology. In Proceedings of the 2019 Symposium on VLSI Circuits, Kyoto, Japan, 9–14 June 2019; pp. C254–C255. [Google Scholar] [CrossRef]
  14. Druml, N.; Maksymova, I.; Thurner, T.; Lierop, D.; Hennecke, M.; Foroutan, A. 1D MEMS Micro-Scanning LiDAR. In Proceedings of the International Conference on Sensor Device Technologies and Applications (SENSORDEVICES), Venice, Italy, 16–20 September 2018. [Google Scholar]
  15. Petit, J.; Shladover, S.E. Potential cyberattacks on automated vehicles. IEEE Trans. Intell. Transp. Syst. 2014, 16, 546–556. [Google Scholar] [CrossRef]
  16. Shin, H.; Kim, D.; Kwon, Y.; Kim, Y. Illusion and Dazzle: Adversarial Optical Channel Exploits against Lidars for Automotive Applications. In Lecture Notes in Computer Science; Springer International Publishing: Cham, Switzerland, 2018; pp. 445–467. [Google Scholar] [CrossRef]
  17. Kim, G.; Eom, J.; Park, Y. An Experiment of Mutual Interference between Automotive LIDAR Scanners. In Proceedings of the 2015 12th International Conference on Information Technology—New Generations, Las Vegas, NV, USA, 13–15 April 2015; pp. 680–685. [Google Scholar] [CrossRef]
  18. Zhang, F.; Liu, Q.; Gong, M.; Fu, X. Anti-dynamic-crosstalk method for single photon LIDAR detection. In Proceedings of the SPIE 10605, LIDAR Imaging Detection and Target Recognition 2017, Bellingham, WA, USA, 23–25 July 2017; Lv, Y., Su, J., Gong, W., Yang, J., Bao, W., Chen, W., Shi, Z., Fei, J., Han, S., Jin, W., Eds.; International Society for Optics and Photonics: Bellingham, WA, USA, 2017; Volume 10605, p. 1060503. [Google Scholar] [CrossRef]
  19. ISO26262. 26262: Road Vehicles-Functional Safety; International Standard; International Organization for Standardization: Geneva, Switzerland, 2011. [Google Scholar]
  20. ISO/PAS 21448:2019. Road Vehicles—Safety of the Intended Functionality; Technical Report; International Organization for Standardization: Geneva, Switzerland, 2019. [Google Scholar]
  21. Chen, J.; Patton, R. Robust Model-based Fault Diagnosis for Dynamic Systems, 1st ed.; Springer: New York, NY, USA, 1999; Volume 3. [Google Scholar] [CrossRef]
  22. Chen, X.; Wang, Z.; Zhang, Z.; Jia, L.; Qin, Y. A Semi-Supervised Approach to Bearing Fault Diagnosis under Variable Conditions towards Imbalanced Unlabeled Data. Sensors 2018, 18, 2097. [Google Scholar] [CrossRef] [Green Version]
  23. Piltan, F.; Kim, J.M. Bearing Fault Diagnosis by a Robust Higher-Order Super-Twisting Sliding Mode Observer. Sensors 2018, 18, 1128. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. Liberati, A.; Altman, D.G.; Tetzlaff, J.; Mulrow, C.; Gøtzsche, P.C.; Ioannidis, J.P.A.; Clarke, M.; Devereaux, P.J.; Kleijnen, J.; Moher, D. The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. PLoS Med. 2009, 6, 1–28. [Google Scholar] [CrossRef] [PubMed]
  25. Yunusa-Kaltungo, A.; Labib, A. A hybrid of industrial maintenance decision making grids. Prod. Plan. Control 2020, 1–18. [Google Scholar] [CrossRef]
  26. Segata, M.; Cigno, R.L.; Bhadani, R.K.; Bunting, M.; Sprinkle, J. A LiDAR Error Model for Cooperative Driving Simulations. In Proceedings of the 2018 IEEE Vehicular Networking Conference (VNC), Taipei, Taiwan, 5–7 December 2018; pp. 1–8. [Google Scholar] [CrossRef] [Green Version]
  27. Sun, X.H. Method and Apparatus for Detection and Ranging Fault Detection and Recovery. U.S. Patent 010203408B, 12 February 2019. [Google Scholar]
  28. Ibeo. Ethernet Data Protocol Ibeo LUX and Ibeo LUX Systems; Technical Report; Ibeo: Hamburg, Germany, 2009. [Google Scholar]
  29. Rivero, J.R.V.; Tahiraj, I.; Schubert, O.; Glassl, C.; Buschardt, B.; Berk, M.; Chen, J. Characterization and simulation of the effect of road dirt on the performance of a laser scanner. In Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan, 16–19 October 2017; pp. 1–6. [Google Scholar] [CrossRef]
  30. Tegler, E. Available online: https://www.popularmechanics.com/cars/car-technology/a29309664/bugs-self-driving-cars/ (accessed on 17 September 2019).
  31. Trierweiler, M.; Caldelas, P.; Gröninger, G.; Peterseim, T.; Neumann, C. Influence of sensor blockage on automotive LiDAR systems. In Proceedings of the 2019 IEEE SENSORS, Montreal, QC, Canada, 27–30 October 2019; pp. 1–4. [Google Scholar] [CrossRef]
  32. Jalopnik. Available online: https://jalopnik.com/google-has-a-wiper-for-its-lidar-to-handle-bird-shit-an-1795346982 (accessed on 5 October 2019).
  33. James, J.K.; Puhlfürst, G.; Golyanik, V.; Stricker, D. Classification of LIDAR Sensor Contaminations with Deep Neural Networks. In Proceedings of the Computer Science in Cars Symposium (CSCS) 2018, Munich, Germany, 13–14 September 2018; p. 8. [Google Scholar] [CrossRef]
  34. Green, A.; Bidner, D.K. Lidar Windscreen Vibration Control. U.S. Patent 20190359179A1, 28 November 2019. [Google Scholar]
  35. Zhao, C.; Gopalan, S. Automotive Image Sensor Surface Washing and Drying System. U.S. Patent 20190202410A, 4 July 2019. [Google Scholar]
  36. Wideye. Available online: https://www.wideye.vision/use-cases-value/lidar-covers/ (accessed on 17 September 2019).
  37. Doorley, G.; Karplus, P.T.H.; Avram, P. Control for Passive Wiper System. U.S. Patent 2017/0151933 A1, 1 June 2017. [Google Scholar]
  38. McMichael, R.; Kentley-Klay, T.D.; Schabb, D.E.; Thakur, A.; Torrey, J. Sensor Obstruction Detection and Mitigation Using Vibration and/ot Heat. U.S. Patent WO 2019246029A1, 19 December 2019. [Google Scholar]
  39. Periu, C.; Mohsenimanesh, A.; Laguë, C.; McLaughlin, N. Isolation of Vibrations Transmitted to a LIDAR Sensor Mounted on an Agricultural Vehicle to Improve Obstacle Detection. Can. Biosyst. Eng. 2013, 55, 233–242. [Google Scholar] [CrossRef] [Green Version]
  40. Hama, S.; Toda, H. Basic Experiment of LIDAR Sensor Measurement Directional Instability for Moving and Vibrating Object. In Proceedings of the 6th Annual International Conference on Material Science and Environmental Engineering, Chongqing, China, 23–25 November 2018; IOP Conference Series-Materials Science and Engineering. Wang, K., Ed.; IOP Publishing Ltd.: Bristol, UK, 2019; Volume 472. [Google Scholar] [CrossRef] [Green Version]
  41. McManamon, P. Field Guide to Lidar; SPIE Field Guides; SPIE: Bellingham, WA, USA, 2015; Volume FG36. [Google Scholar]
  42. Stottelaar, B.G. Practical Cyber-Attacks on Autonomous Vehicles. Master’s Thesis, University of Twente, Enschede, The Netherlands, 2015. [Google Scholar]
  43. Guo, P.; Kim, H.; Virani, N.; Xu, J.; Zhu, M.; Liu, P. RoboADS: Anomaly Detection Against Sensor and Actuator Misbehaviors in Mobile Robots. In Proceedings of the 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), Luxembourg, 25–28 June 2018; pp. 574–585. [Google Scholar] [CrossRef]
  44. Petit, J.; Stottelaar, B.; Feiri, M.; Kargl, F. Remote attacks on automated vehicles sensors: Experiments on camera and lidar. Black Hat Eur. 2015, 11, 2015. [Google Scholar]
  45. Thing, V.L.L.; Wu, J. Autonomous Vehicle Security: A Taxonomy of Attacks and Defences. In Proceedings of the 2016 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), Chengdu, China, 15–18 December 2016; pp. 164–170. [Google Scholar] [CrossRef]
  46. Choi, H.; Lee, W.C.; Aafer, Y.; Fei, F.; Tu, Z.; Zhang, X.; Xu, D.; Deng, X. Detecting Attacks Against Robotic Vehicles: A Control Invariant Approach. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS ’18, Toronto, ON, Canada, 15–19 October 2018; ACM: New York, NY, USA, 2018; pp. 801–816. [Google Scholar] [CrossRef]
  47. Dutta, R.G. Security of Autonomous Systems Under Physical Attacks: With Application to Self-Driving Cars. Ph.D. Thesis, University of Central Florida, Orlando, FL, USA, 2018. [Google Scholar]
  48. Zang, S.; Ding, M.; Smith, D.; Tyler, P.; Rakotoarivelo, T.; Kaafar, M.A. The Impact of Adverse Weather Conditions on Autonomous Vehicles: How Rain, Snow, Fog, and Hail Affect the Performance of a Self-Driving Car. IEEE Veh. Technol. Mag. 2019, 14, 103–111. [Google Scholar] [CrossRef]
  49. Filgueira, A.; González-Jorge, H.; Lagüela, S.; Díaz-Vilariño, L.; Arias, P. Quantifying the influence of rain in LiDAR performance. Measurement 2017, 95, 143–148. [Google Scholar] [CrossRef]
  50. Hasirlioglu, S.; Doric, I.; Kamann, A.; Riener, A. Reproducible Fog Simulation for Testing Automotive Surround Sensors. In Proceedings of the 2017 IEEE 85th Vehicular Technology Conference (VTC Spring), Sydney, NSW, Australia, 4–7 June 2017; pp. 1–7. [Google Scholar] [CrossRef]
  51. Kim, B.K.; Sumi, Y. Performance evaluation of safety sensors in the indoor fog chamber. In Proceedings of the 2017 IEEE Underwater Technology (UT), Busan, Korea, 21–24 February 2017; pp. 1–3. [Google Scholar] [CrossRef]
  52. Dannheim, C.; Icking, C.; Mader, M.; Sallis, P. Weather Detection in Vehicles by Means of Camera and LIDAR Systems. In Proceedings of the 2014 Sixth International Conference on Computational Intelligence, Communication Systems and Networks, Tetova, Macedonia, 27–29 May 2014; pp. 186–191. [Google Scholar] [CrossRef]
  53. RobustSENSE. Deliverable D5.1 System Performance Assessment Specification; Technical Report; RobustSENSE: Ulm, Germany, 2016. [Google Scholar]
  54. Hasirlioglu, S.; Riener, A. Introduction to Rain and Fog Attenuation on Automotive Surround Sensors. In Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan, 16–19 October 2017. [Google Scholar] [CrossRef]
  55. Mäyrä, A.; Hietala, E.; Kutila, M.; Pyykönen, P. Spectral attenuation in low visibility artificial fog: Experimental study and comparison to literature models. In Proceedings of the 2017 13th IEEE International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania, 7–9 September 2017; pp. 303–308. [Google Scholar] [CrossRef]
  56. Mäyrä, A.; Hietala, E.; Kutila, M.; Pyykönen, P.; Tiihonen, M.; Jokela, T. Experimental study on spectral absorbance in fog as a function of temperature, liquid water content, and particle size. In Optics in Atmospheric Propagation and Adaptive Systems; International Society for Optics and Photonics: Bellingham, WA, USA, 2017; Volume 10425, p. 104250G. [Google Scholar] [CrossRef]
  57. Heinzler, R.; Schindler, P.; Seekircher, J.; Ritter, W.; Stork, W. Weather Influence and Classification with Automotive Lidar Sensors. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 9–12 June 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1527–1534. [Google Scholar] [CrossRef] [Green Version]
  58. Aho, A.T.; Viheriälä, J.; Mäkelä, J.; Virtanen, H.; Ranta, S.; Dumitrescu, M.; Guina, M. High-power 1550 nm tapered DBR laser diodes for LIDAR applications. In Proceedings of the 2017 Conference on Lasers and Electro-Optics Europe European Quantum Electronics Conference (CLEO/Europe-EQEC), Munich, Germany, 25–29 June 2017. [Google Scholar] [CrossRef]
  59. Goodin, C.; Carruth, D.; Doude, M.; Hudson, C. Predicting the Influence of Rain on LIDAR in ADAS. Electronics 2019, 8, 89. [Google Scholar] [CrossRef] [Green Version]
  60. Ruiz-Llata, M.; Rodriguez-Cortina, M.; Martin-Mateos, P.; Bonilla-Manrique, O.E.; Ramon Lopez-Fernandez, J. LiDAR design for Road Condition Measurement ahead of a moving vehicle. In Proceedings of the 2017 16th IEEE Sensors, IEEE Sensors, Glasgow, UK, 29 October–1 November 2017; pp. 1062–1064. [Google Scholar] [CrossRef] [Green Version]
  61. Rosique, F.; Navarro, P.J.; Fernández, C.; Padilla, A. A Systematic Review of Perception System and Simulators for Autonomous Vehicles Research. Sensors 2019, 19, 648. [Google Scholar] [CrossRef] [Green Version]
  62. Daniel, L.; Phippen, D.; Hoare, E.; Stove, A.; Cherniakov, M.; Gashinova, M. Low-THz radar, lidar and optical imaging through artificially generated fog. In Proceedings of the International Conference on Radar Systems 2017, Belfast, UK, 23–26 October 2017; pp. 1–4. [Google Scholar] [CrossRef]
  63. Rasshofer, R.H.; Spies, M.; Spies, H. Influences of weather phenomena on automotive laser radar systems. Adv. Radio Sci. 2011, 9, 49–60. [Google Scholar] [CrossRef] [Green Version]
  64. Xique, I.J.; Buller, W.; Fard, Z.B.; Dennis, E.; Hart, B. Evaluating Complementary Strengths and Weaknesses of ADAS Sensors. In Proceedings of the 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall), Chicago, IL, USA, 27–30 August 2018; pp. 1–5. [Google Scholar]
  65. Kutila, M.; Pyykonen, P.; Ritter, W.; Sawade, O.; Schaufele, B. Automotive LIDAR sensor development scenarios for harsh weather conditions. In Proceedings of the 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil, 1–4 November 2016; pp. 265–270. [Google Scholar] [CrossRef]
  66. Phillips, T.G.; Guenther, N.; McAree, P.R. When the Dust Settles: The Four Behaviors of LiDAR in the Presence of Fine Airborne Particulates. J. Field Robot. 2017, 34, 985–1009. [Google Scholar] [CrossRef]
  67. Wilson, J.; Sinha, P. Thermal Considerations For Designing Reliable and Safe Autonomous Vehicle Sensors. Available online: https://www.fierceelectronics.com/components/thermal-considerations-for-designing-reliable-and-safe-autonomous-vehicle-sensors (accessed on 25 September 2019).
  68. Hasirlioglu, S.; Riener, A.; Huber, W.; Wintersberger, P. Effects of exhaust gases on laser scanner data quality at low ambient temperatures. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA, 11–14 June 2017; pp. 1708–1713. [Google Scholar] [CrossRef]
  69. Hasirlioglu, S.; Kamann, A.; Doric, I.; Brandmeier, T. Test methodology for rain influence on automotive surround sensors. In Proceedings of the 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil, 1–4 November 2016; pp. 2242–2247. [Google Scholar] [CrossRef]
  70. Moorehead, S.; Simmons, R.; Apostolopoulos, D.D.; Whittaker, W.R.L. Autonomous Navigation Field Results of a Planetary Analog Robot in Antarctica. In Proceedings of the International Symposium on Artificial Intelligence, Robotics and Automation in Space, Noordwijk, The Netherlands, 1–3 June 1999. [Google Scholar]
  71. Pfeuffer, A.; Dietmayer, K. Optimal Sensor Data Fusion Architecture for Object Detection in Adverse Weather Conditions. arXiv 2018, arXiv:1807.02323. [Google Scholar]
  72. Sick. Available online: https://lidarnews.com/articles/top-5-considerations-for-choosing-lidar-for-outdoor-robots/ (accessed on 25 September 2019).
  73. Waterloo Autonomous Vehicles Lab. Available online: http://wavelab.uwaterloo.ca/?weblizar_portfolio=real-time-filtering-of-snow-from-lidar-point-clouds (accessed on 25 September 2019).
  74. Zhang, C.; Ang, M.H.; Rus, D. Robust LIDAR Localization for Autonomous Driving in Rain. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 1–5 October 2018; pp. 3409–3415. [Google Scholar] [CrossRef]
  75. Peynot, T.; Underwood, J.; Scheding, S. Towards reliable perception for Unmanned Ground Vehicles in challenging conditions. In Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, St. Louis, MO, USA, 10–15 October 2009; pp. 1170–1176. [Google Scholar] [CrossRef] [Green Version]
  76. Bijelic, M.; Gruber, T.; Ritter, W. A Benchmark for Lidar Sensors in Fog: Is Detection Breaking Down? In Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China, 26–30 June 2018. [Google Scholar] [CrossRef] [Green Version]
  77. Hasirlioglu, S.; Riener, A. A Model-based Approach to Simulate Rain Effects on Automotive Surround Sensor Data. In Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018; pp. 2609–2615. [Google Scholar] [CrossRef]
  78. Ijaz, M.; Ghassemlooy, Z.; Minh, H.L.; Rajbhandari, S.; Perez, J. Analysis of Fog and Smoke Attenuation in a Free Space Optical Communication Link. In Proceedings of the 2012 International Workshop on Optical Wireless Communications (IWOW), Pisa, Italy, 22 October 2012; pp. 1–3. [Google Scholar]
  79. Fersch, T.; Buhmann, A.; Koelpin, A.; Weigel, R. The Influence and of Rain on Small Aperture and LiDAR Sensors. In Proceedings of the German Microwave Conference (GeMiC) 2016, Bochum, Germany, 14–16 March 2016. [Google Scholar]
  80. Hadj-Bachir, M.; Souza, P. LIDAR Sensor Simulation in Adverse Weather Condition for Driving Assistance Development; hal-01998668; Hyper Articles en Ligne (HAL): Lyon, France, 2019. [Google Scholar]
  81. Kutila, M.; Pyykönen, P.; Holzhüter, H.; Colomb, M.; Duthon, P. Automotive LiDAR performance verification in fog and rain. In Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018; pp. 1695–1701. [Google Scholar] [CrossRef]
  82. Hasirlioglu, S.; Doric, I.; Lauerer, C.; Brandmeier, T. Modeling and simulation of rain for the test of automotive sensor systems. In Proceedings of the 2016 IEEE Intelligent Vehicles Symposium (IV), Gotenburg, Sweden, 19–22 June 2016; pp. 286–291. [Google Scholar] [CrossRef]
  83. Schönhuber, M. Available online: https://www.distrometer.at/2dvd/ (accessed on 13 September 2019).
  84. Jokela, M.; Kutila, M.; Pyykönen, P. Testing and Validation of Automotive Point-Cloud Sensors in Adverse Weather Conditions. Appl. Sci. 2019, 9, 2341. [Google Scholar] [CrossRef] [Green Version]
  85. Michaud, S.; Lalonde, J.F.; Giguere, P. Towards Characterizing the Behavior of LiDARs in Snowy Conditions. In Proceedings of the 7th Workshop on Planning, Perception and Navigation for Intelligent Vehicles, Hamburg, Germany, 28 September 2015. [Google Scholar]
  86. Sick. Available online: https://www.sick.com/ag/en/detection-and-ranging-solutions/2d-lidar-sensors/lms1xx/c/g91901 (accessed on 18 October 2019).
  87. O’Brien, M.E.; Fouche, D.G. Simulation of 3D Laser Radar Systems. Linc. Lab. J. 2005, 15, 37–60. [Google Scholar]
  88. McKnight, D.; Miles, R. Impact of reduced visibility conditions on laser based DP sensors. In Proceedings of the Dynamic Positioning Conference, Houston, TX, USA, 14–15 October 2014. [Google Scholar]
  89. Lingg, A.J.; Beck, S.W.; Stepanian, J.G.; Clifford, D.H. Method and Apparatus Crosstalk and Multipath Noise Reduction in A Lidar System. U.S. Patent 2019/0339393A1, 7 November 2019. [Google Scholar]
  90. Diehm, A.L.; Hammer, M.; Hebel, M.; Arens, M. Mitigation of crosstalk effects in multi-LiDAR configurations. In Proceedings of the Electro-Optical Remote Sensing XII, Berlin, Germany, 12–13 September 2018; Kamerman, G., Steinvall, O., Eds.; SPIE: Washington, DC, USA, 2018; Volume 10796. [Google Scholar] [CrossRef]
  91. Retterath, J.E.; Laumeyer, R.A. Methods and Apparatus for Array Based LiDAR Systems With Reduced Interference. U.S. Patent 10,203,399, 12 February 2019. [Google Scholar]
  92. Denham, M.; Gilliland, P.B.; Goldstein, B.M.; Musa, O. Crosstalk Mitigation Circuit for Lidar Pixel Receivers. U.S. Patent 2019/0317196A1, 17 October 2019. [Google Scholar]
  93. Hall, D.S. High Definition LiDAR System. U.S. Patent 7969558B2, 28 June 2011. [Google Scholar]
  94. Eichenholz, J.M.; LaChapelle, J.G. Detection of Crosstalk and Jamming Pulses With Lidar System. U.S. Patent 16/178,049, 2 May 2019. [Google Scholar]
  95. Huang, W.; Su, X. Design of a Fault Detection and Isolation System for Intelligent Vehicle Navigation System. Int. J. Navig. Obs. 2015, 2015, 1–19. [Google Scholar] [CrossRef] [Green Version]
  96. Qin, S.J.; Guiver, J.P. Sensor Validation Apparatus and Method. U.S. Patent 6594620B1, 15 July 2003. [Google Scholar]
  97. Song, J.; Fry, G.; Wu, C.; Parmer, G. CAML: Machine Learning-based Predictable, System-Level Anomaly Detection. In Proceedings of the 1st Workshop on Security and Dependability of Critical Embedded Real-Time Systems, in Conjunction with IEEE Real-Time Systems Symposium, Porto, Portugal, 29 November 2016. [Google Scholar]
  98. Breed, D.S. System and Method for Vehicle Diagnostics. U.S. Patent 7103460B1, 5 September 2006. [Google Scholar]
  99. Leslar, M.; Wang, J.; Hu, B. A Comparison of Two New Methods of Outlier Detection for Mobile Terrestrial Lidar Data. Proc. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2010, 38, 78–84. [Google Scholar]
  100. Khalastchi, E.; Kaminka, G.; Lin, R.; Kalech, M. Anomaly Detection Methods, Devices and Systems. U.S. Patent 2014/0149806A1, 29 May 2014. [Google Scholar]
  101. Tascione, D.; Bode, M. Autonomous Vehicle Diagnostic System. U.S. Patent 2018/0050704A1, 22 February 2018. [Google Scholar]
  102. Abt, T.L.; Hirsenkorn, K.; Isert, C.; Parolini, L.; Radler, S.; Rauch, S. Method, System, and Computer Program Product for Determining a Blockage of a Sensor of a Plurality of Sensors of an Ego Vehicle. EP 3511740A1, 17 July 2019. [Google Scholar]
  103. Mori, D.; Sugiura, H.; Hattori, Y. Adaptive Sensor Fault Detection and Isolation using Unscented Kalman Filter for Vehicle Positioning. In Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 27–30 October 2019; pp. 1298–1304. [Google Scholar] [CrossRef]
  104. Bruns, E.; Venator, M. Verfahren und Steuervorrichtung zum Erkennen einer Fehlfunktion zumindest eines Umfeldsensors eines Kraftfahrzeugs. DE 102018205322A1, 10 April 2019. [Google Scholar]
  105. Zhu, J.; Ferguson, D.I. System to Optimize Sensor Parameters in an Autonomous Vehicle. U.S. Patent 2018/0329423A1, 15 November 2018. [Google Scholar]
  106. Mehrdad, S.K. Verfahren Zum Verarbeiten Von Sensordaten, Anordnung Zur Verarbeitung Von Sensordaten, Anordnung Zum Verarbeiten Einer Anfrage Eines Ego-Fahrzeugs Für Ein Weiteres Fahrzeug, Rechner Zum Verarbeiten Von Sensordaten Sowie Rechner Zum Verarbeiten Einer Anfrage Eines Ego-Fahrzeugs Für Ein Weiteres Fahrzeug. DE 102018207658A1, 21 November 2019. [Google Scholar]
  107. Ghimire, S.; Prokhorov, D. Collaborative Multi-Agent Vehicle Fault Diagnostic System & Associated Methodology. WO 2012/148514 A1, 1 November 2012. [Google Scholar]
  108. Zhu, J.; Dolgov, D.A.; Urmson, C.P. Cross-Validating Sensors of an Autonomous Vehicle. U.S. Patent 9555740B1, 31 January 2017. [Google Scholar]
  109. Kim, H.S.; Seo, M.W.; Yang, J.W.; Kim, S.Y.; Yang, D.H.; Ko, B.C.; Seo, G.W. Apparatus and Method for Failure Diagnosis and Calibration of Sensors for advanced Driver Assistance Systems. U.S. Patent 10026239B2, 17 July 2018. [Google Scholar]
  110. Zhu, J.; Dolgov, D.; Urmson, C.P. Cross-Validating Sensors of an Autonomous Vehicle. U.S. Patent 9221396B1, 29 December 2015. [Google Scholar]
  111. Hartung, J.J.; Brink, P.; Lamb, J.; Miller, D.P. Autonomous Vehicle Platform and Safety Architecture. U.S. Patent 2017139411A1, 18 May 2017. [Google Scholar]
  112. Gao, B.; Coifman, B. Vehicle identification and GPS error detection from a LIDAR equipped probe vehicle. In Proceedings of the 2006 IEEE Intelligent Transportation Systems Conference, Toronto, ON, Canada, 17–20 September 2006; pp. 1537–1542. [Google Scholar] [CrossRef]
  113. Egnor, D.T.; Zbrozek, A.; Schultz, A. Methods and Systems for Compensating for Common Failures. U.S. Patent 9,195.232 B1, 24 November 2015. [Google Scholar]
  114. Ning, X.; Li, F.; Tian, G.; Wang, Y. An efficient outlier removal method for scattered point cloud data. PLoS ONE 2018, 13, e0201280. [Google Scholar] [CrossRef] [Green Version]
  115. Shin, B. Fault Tolerant Control and Localization for Autonomous Driving: Systems and Architecture; Technical Report UCB/EECS-2016-83; University of California at Berkeley: Berkeley, CA, USA, 2016. [Google Scholar]
  116. Sazara, C.; Nezafat, R.V.; Cetin, M. Offline reconstruction of missing vehicle trajectory data from 3D LIDAR. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA, 11–14 June 2017; pp. 792–797. [Google Scholar] [CrossRef] [Green Version]
  117. Avlas, L.N.; Berseth, E.N. Systems and methods for mitigating optical crosstalk in a light ranging and detection system. U.S. Patent 2020/0064452A1, 27 February 2020. [Google Scholar]
  118. El-Nayal, M.K.; Aly, M.M.; Fayed, H.A.; AbdelRassoul, R.A. Adaptive free space optic system based on visibility detector to overcome atmospheric attenuation. Results Phys. 2019, 14, 102392. [Google Scholar] [CrossRef]
  119. Colomb, M.; Duthon, P.; Laukkanen, S. Characteristics of Adverse Weather Conditions; Technical Report, DENSE247.eu; DENSE Consortium: Ulm, Germany, 2017. [Google Scholar]
  120. Hespel, L.; Riviere, N.; Huet, T.; Tanguy, B.; Ceolato, R. Performance evaluation of laser scanners through the atmosphere with adverse condition. In Electro-Optical Remote Sensing, Photonic Technologies, and Applications V; Kamerman, G.W., Steinvall, O., Bishop, G.J., Gonglewski, J.D., Lewis, K.L., Hollins, R.C., Merlet, T.J., Eds.; International Society for Optics and Photonics: Bellingham, WA, USA, 2011; Volume 8186, p. 818606. [Google Scholar] [CrossRef]
Sensors 20 03662 g001 550
Figure 1. Schematic illustration of the sense-plan-act cycle including a sensor FDIIR (fault detection, isolation, identification, and recovery) system that monitors the sensor activity to ensure a reliable environment perception.
Figure 1. Schematic illustration of the sense-plan-act cycle including a sensor FDIIR (fault detection, isolation, identification, and recovery) system that monitors the sensor activity to ensure a reliable environment perception.
Sensors 20 03662 g001
Sensors 20 03662 g002 550
Figure 2. Schematic illustration of sensor fault classes.
Figure 2. Schematic illustration of sensor fault classes.
Sensors 20 03662 g002
Sensors 20 03662 g003 550
Figure 3. Schematic illustration of FDII methods.
Figure 3. Schematic illustration of FDII methods.
Sensors 20 03662 g003
Sensors 20 03662 g004 550
Figure 4. Review process for FDIIR methods of automotive lidar sensors.
Figure 4. Review process for FDIIR methods of automotive lidar sensors.
Sensors 20 03662 g004
Sensors 20 03662 g005 550
Figure 5. Overview of BibTeX publications which were analysed (N = 95). (a) shows the counts of BibTeX classes. (b) shows the number of publications per year.
Figure 5. Overview of BibTeX publications which were analysed (N = 95). (a) shows the counts of BibTeX classes. (b) shows the number of publications per year.
Sensors 20 03662 g005
Sensors 20 03662 g006 550
Figure 6. Counts of mentioned faults in the literature.
Figure 6. Counts of mentioned faults in the literature.
Sensors 20 03662 g006
Sensors 20 03662 g007 550
Figure 7. FDII classes count with colour coding according to the mentioned fault class in the publication. Some papers present fault detection methods without mentioning a specific fault, as indicated by the “n/a” category.
Figure 7. FDII classes count with colour coding according to the mentioned fault class in the publication. Some papers present fault detection methods without mentioning a specific fault, as indicated by the “n/a” category.
Sensors 20 03662 g007
Sensors 20 03662 g008 550
Figure 8. Recovery class count colour coded by fault classes.
Figure 8. Recovery class count colour coded by fault classes.
Sensors 20 03662 g008
Sensors 20 03662 g009 550
Figure 9. Exemplary faults (Table 1) and their associated recovery class count.
Figure 9. Exemplary faults (Table 1) and their associated recovery class count.
Sensors 20 03662 g009
Table
Table 1. Classification of perception sensor faults including exemplary faults and related international safety standards.
Table 1. Classification of perception sensor faults including exemplary faults and related international safety standards.
Fault ClassExemplary FaultSafety Standard
defect subcomponentdefect transmitterISO 26262
defect receiver
defect processing unit
mechanical damage to sensor coverdeformationISO PAS 21448
scratch
crack
hole
missing cover
layer on sensordirt layer
water layer
ice layer
salt layer
snow layer
mounting issueposition change
vibration
security attacklaser hack
electronics hack
unfavourable environmental conditionlow/high temperature
dust
smoke
rain
fog
hail
electromagnetic interference
light disturbance
snow
crosstalkother active perception sensor
Table
Table 2. Classification of FDII methods for perception sensors including exemplary methods.
Table 2. Classification of FDII methods for perception sensors including exemplary methods.
FDII ClassExemplary Method
comparison to sensor modelobjects detected by the sensor model compared to
objects detected by the real sensor
monitoring sensor outputsignal analysis and plausibility check of sensor output
comparison to static ground-truthinfrastructure detected by the sensor compared to
ground-truth infrastructure in the environment
comparison to dynamic ground-truthroad users detected by another vehicle compared to
road user detected by the ego-vehicle
comparison to other sensor of same typecompare objects that are detected by the sensor
under observation with objects detected by another
sensor of the same type (two lidar sensors)
comparison to other sensor of different typecompare objects that are detected by the sensor
under observation with objects detected by another
sensor of a different type (a lidar and a radar sensors)
monitoring internal interfacesignal analysis and plausibility check of the output
of a single sensor interface
comparison of multiple interfacesa part of the sensor between sensor interfaces is
modelled; comparison of the output of the modelled
part with the output of the respective sensor interface
Table
Table 3. Classification of fault recovery methods for perception sensors including exemplary methods.
Table 3. Classification of fault recovery methods for perception sensors including exemplary methods.
Recovery ClassExemplary Method
software adaptionfilter pointcloud
mitigate faults in sensor data by averaging
hardware adaptionchange alignment of sensor cover
adjust sensor position and rotation
temperature regulationheating
cooling
cleaning of sensor coverair nozzle
fluid nozzle
wiper
Table
Table 4. Literature review of fault classes, FDII and recovery methods for automotive lidar.
Table 4. Literature review of fault classes, FDII and recovery methods for automotive lidar.
Fault ClassLiterature
defect subcomponent[26,27,28]
mechanical damage to sensor cover[29]
layer on sensor cover[29,30,31,32,33,34,35,36,37,38]
mounting issue[39,40,41]
security attack[15,16,42,43,44,45,46,47]
unfavourable environmental condition[16,28,30,41,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88]
crosstalk[17,18,63,89,90,91,92,93,94]
FDII Class
comparison to sensor model[43,95,96]
monitoring sensor output[18,26,33,38,41,43,46,47,53,60,84,90,97,98,99,100,101,102,103]
comparison to static ground-truth[104,105]
comparison to dynamic ground-truth[104,105,106,107]
comparison to other sensor of same type[38,108,109,110]
comparison to other sensor of different type[27,38,43,46,52,53,62,108,109,110,111,112]
monitoring internal interface-
comparison of multiple interfaces[113]
Recovery Class
software adaption[18,44,47,70,73,74,75,90,99,103,112,114,115,116]
hardware adaption[39]
temperature adaption[30,36,38,67,86]
cleaning of sensor cover[30,32,34,35,37,38]

Share and Cite

MDPI and ACS Style

Goelles, T.; Schlager, B.; Muckenhuber, S. Fault Detection, Isolation, Identification and Recovery (FDIIR) Methods for Automotive Perception Sensors Including a Detailed Literature Survey for Lidar. Sensors 2020, 20, 3662. https://doi.org/10.3390/s20133662

AMA Style

Goelles T, Schlager B, Muckenhuber S. Fault Detection, Isolation, Identification and Recovery (FDIIR) Methods for Automotive Perception Sensors Including a Detailed Literature Survey for Lidar. Sensors. 2020; 20(13):3662. https://doi.org/10.3390/s20133662

Chicago/Turabian Style

Goelles, Thomas, Birgit Schlager, and Stefan Muckenhuber. 2020. "Fault Detection, Isolation, Identification and Recovery (FDIIR) Methods for Automotive Perception Sensors Including a Detailed Literature Survey for Lidar" Sensors 20, no. 13: 3662. https://doi.org/10.3390/s20133662

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop