Online distraction detection for naturalistic driving dataset using kinematic motion models and a multiple model algorithm
Introduction
Driving safety is an increasing concern in both transportation research and the automotive industry. In 2013, a global status report by the World Health Organization indicated 1.24 million traffic-related fatalities (Organization et al., 2013). This report defined the leading cause of death for people ages – years as traffic-crashes. As about 70% of traffic-crashes are caused by human errors, one of important topics is the detection of distracted driving (Treat, 1980, Aberg and Rimmo, 1998).
Distracted driving occurs when a driver diverts his/her attention away from the driving task to focus on a secondary task (Wang et al., 2017a). For example, texting on a cellphone during driving is a common risky secondary task that causes a driver’s attention to both visually and manually divert from the main task of driving (Li et al., 2017). In 2015, the United States National Highway Traffic Safety Administrations found distracted driving to account for around 3477 fatalities (10% of overall fatalities) with an additional estimated 391,000 injuries in the United States (National Center for Statistics and Analysis, 2017). Furthermore, these numbers are likely under-reported due to difficulties in identifying driver distraction during crash investigation (Sundeen, 2006).
Various Advanced Driver Assistance Systems (ADAS) have been increasingly developed to prevent drivers from unsafe or dangerous situations relating to distracted driving. In general, ADAS typically follow two common approaches to detect distracted driving. The first most popular approach is to monitor drivers’ behavior directly via extra sensors installed within the cockpit of the vehicle, or via smartphone cameras, to capture how the driver’s head orientation is away from the driving direction. Within this approach, distraction behaviors are typically described as instances when drivers fail to focus their gaze towards the direction of driving for a pre-specified amount of time (Mihai et al., 2015, You et al., 2013, Xiao and Feng, 2016). Other forms of detecting distraction behaviors include monitoring drivers for a sign of fatigue via irregular blinking patterns or yawning (Eriksson and Papanikolopoulos, 2001, Zhu and Ji, 2004, Omidyeganeh et al., 2011). Moreover, time-spent-on-task has also been used as an indicator of distracted driving behaviors (Feng et al., 2009). ADAS using all above ways require using smartphone cameras. The issue of user-privacy still remains a discussion for the community. Therefore, implementation of the first type of approach is still limited in practice.
The second type of approach analyzes vehicle kinematics based on readily-available vehicle kinematic signals corresponding to speed, yaw rate, acceleration, etc. The principle is that the secondary tasks which distract the driver may have a direct influence on their control of the vehicle, and hence lead to abnormal vehicle kinematic signals. For example, Feng et al. (2017) identified aggressive driving behaviors based on unique characteristics of vehicle jerk in drivers’ gas and brake pedal operations. In ADAS research, kinematic signals have been used to detect unintentional drifting towards the boundary of the driving lane for developing lane departure warning (LDW) systems (Aksan et al., 2016, Jamson et al., 2008, Chen et al., 2018, Adell et al., 2011). Many models have been developed to predict the vehicle’s lateral trajectory (Angkititrakul et al., 2010, Wang et al., 2018, Narote et al., 2018) or its time to lane crossing (Cario et al., 2009). However, these methods mainly generate alarms when the vehicle is approaching the lane boundary, which may not warn the distracted driver in time. In contrast, our paper aims to detect distraction driving behaviors using a different anomaly detection strategy, that is, to firstly predict the vehicle kinematics under the normal driving, and then to trigger alarms when the directly measured vehicle kinematics are significantly different from the normal behavior predicted by the model. In this way, the ADAS can generate timely alarms to warn the distracted driver when their kinematic driving behaviors are significantly changed.
A key component of the proposed approach is the prediction of vehicle kinematics. Three different types of models are often used to predict vehicle kinematics: driving maneuver models, vehicle dynamic models, or state–space models (Lefèvre et al., 2014). Driving maneuver models predict vehicle kinematics based on different driving maneuvers. For example, Wang and Zhao (2018) used a hierarchical Dirichlet process with a Hidden Semi-Markov Model (HDP-HSMM) to learn normal maneuvers for making predictions. These models usually do not rely on physics models, which may generate less accurate predictions. However, a high prediction accuracy of vehicle kinematic signals is often required to distinguish the distracted driving behaviors from natural variations.
Vehicle dynamic models are often built based on Lagrange’s equations and consider dynamic force responses such as longitudinal tire forces or road banking angle (Rajamani, 2011). However, the parameters (such as slip angle) in these models vary under different weather and road conditions and can thus be intractable for real-time motion prediction (Pacejka, 2005, Cossalter, 2006).
State–space models (Schubert et al., 2008) track the vehicle kinematics based on typical kinematic physics equations. The Constant Velocity (CV) and Constant Acceleration (CA) models are often used when the vehicle is driven on a straight road. The Constant Turn Rate and Velocity (CTRV) and Constant Turn Rate and Acceleration (CTRA) models are applied when the vehicle is driven on a curvy road. Sun et al. (2015) detected lane-level irregular driving behaviors using GPS signals in a simulation dataset. While these models are able to provide predictions for a variety of scenarios, state–space models have seldom been implemented in ADAS for distraction detection with naturalistic vehicle kinematic data.
In this paper, an online algorithm is proposed to detect distracted driving behaviors based on vehicle kinematic prediction using state–space models. The proposed method involves two steps. One is to fuse multiple state–space models to predict vehicle kinematics. The other is to develop a control chart-based decision strategy to detect abnormal mean shifts of lateral speeds and prediction errors of lane positions to provide warnings of distracted driving. Our proposed algorithm aims to enhance ADAS by detecting distracted driving based on readily-available vehicle kinematic signals. This paper is organized as follows: Section 2 describes notation, data structure, and the formulation of the problem. Section 3 elaborates, step-by-step, the proposed distraction detection algorithm. Section 4 presents a case study which uses vehicle kinematic data to detect distraction behaviors. The paper will then conclude with a discussion.
Section snippets
Notations and problem formulation
A naturalistic driving dataset containing multivariate vehicle kinematic signals, including longitudinal speed, lateral speed, longitudinal acceleration, yaw rate and lane position based on cameras was used in this study (Sayer et al., 2011). All the signals were synchronized and resampled with a frequency of for easier analysis. Let denote the vehicle kinematic signals at time . Each is a dimensional vector of the measured motion signals at time .
Predict vehicle kinematics using state–space models
As a start, is predicted based on state–space models under different driving motion patterns . Each of the state–space models is a special case of the Kalman Filter (Kalman, 1960). Let denote the state vector, which is a dimensional vector of the true vehicle kinematic signals. A known transition matrix updates to as where is the linear transition matrix, and is the process noise that follows the Gaussian distribution . The covariance
Case study
In this section, the proposed method is applied to the vehicle kinematic data in two existing naturalistic driving datasets — the Integrated Vehicle-Based Safety Systems (IVBSS) (Sayer et al., 2011) and Safety Pilot Model Deployment (SPMD) datasets (Bezzina and Sayer, 2014).
The IVBSS program was designed to build and test an integrated in-vehicle crash warning system that includes forward crash warning, lane departure warning, curve speed warning, and lane change warning. From 2009 to 2010 a
Discussion
The paper presents a general methodology to detect the distraction behaviors that lead to abnormal driving kinematic signals. For the demonstration purpose, the method is validated by the detection performance of texting behaviors during high-speed driving on freeways (Feng et al., 2018). Additional efforts are expected to develop a comprehensive ADAS that is capable of detecting distractions under all circumstances.
Firstly, the current algorithm is trained based on vehicle kinematic signals
CRediT authorship contribution statement
Wenbo Sun: Conceptualization, Methodology, Software, Formal analysis, Data curation, Writing – original draft, Writing – review & editing. Matthew Aguirre: Conceptualization, Validation, Writing – original draft, Writing – review & editing. Jionghua (Judy) Jin: Conceptualization, Methodology, Writing – review & editing, Supervision, Funding acquisition. Fred Feng: Conceptualization, Methodology, Software, Investigation, Data curation, Writing - review & editing. Samer Rajab: Conceptualization,
Acknowledgments
The research is jointly supported by Honda R&D Americas, Inc. and Michigan Institute for Data Science (MIDAS) .
References (44)
- et al.
The effects of a driver assistance system for safe speed and safe distance–a real-life field study
Transp. Res. C
(2011) - et al.
An examination of teen drivers’ car-following behavior under naturalistic driving conditions: With and without an advanced driving assistance system
Accid. Anal. Prev.
(2020) - et al.
Impact of combined alignments on lane departure: A simulator study for mountainous freeways
Transp. Res. C
(2018) - et al.
Driver fatigue: a vision-based approach to automatic diagnosis
Transp. Res. C
(2001) - et al.
Can vehicle longitudinal jerk be used to identify aggressive drivers? an examination using naturalistic driving data
Accid. Anal. Prev.
(2017) - et al.
Potential benefits of an adaptive forward collision warning system
Transp. Res. C
(2008) - et al.
A review of recent advances in lane detection and departure warning system
Pattern Recognit.
(2018) - et al.
An integrated solution for lane level irregular driving detection on highways
Transp. Res. C
(2015) - et al.
Examining drivers’ eye glance patterns during distracted driving: Insights from scanning randomness and glance transition matrix
J. Saf. Res.
(2017) - et al.
A spectral power analysis of driving behavior changes during the transition from nondistraction to distraction
Traffic Injury Prev.
(2017)
Dimensions of aberrant driver behaviour
Ergonomics
Benefits from Heads-Up Lane Departure Warnings Predicts Safety in the Real-WorldTech. Report
On the use of stochastic driver behavior model in lane departure warning
IEEE Trans. Intell. Transp. Syst.
Safety Pilot Model Deployment: Test Conductor Team ReportReport No. DOT HS, 812
Design and Analysis of Gauge R&R Studies: Making Decisions with Confidence Intervals in Random and Mixed ANOVA Models, Vol. 17
Motorcycle Dynamics
Estimation of lead vehicle kinematics using camera-based data for driver distraction detection
Int. J. Automotive Eng.
An on-board system for detecting driver drowsiness based on multi-sensor data fusion using Dempster-Shafer theory
A General Method for Approximating Nonlinear Transformations of Probability DistributionsTechnical Report
Unscented filtering and nonlinear estimation
Proc. IEEE
A new approach to linear filtering and prediction problems
J. Basic Eng.
Cited by (11)
Transitioning to multi-dimensional estimation of visual distraction and its safety effects under automated driving: A spatiotemporal and directional estimation approach
2023, Transportation Research Part C: Emerging TechnologiesDistracted driving detection based on the fusion of deep learning and causal reasoning
2023, Information FusionCitation Excerpt :The method proposed by Sun et al. established vehicle motion patterns under normal driving by fusing multiple state space models, and then detects distracted driving online based on exponentially weighted moving average and cumulative sum charts. And the performance of the method was validated by identifying the texting behavior of drivers while driving at high speeds [19]. Behavioral analysis methods based on vehicle data mining generally require accurate real-time data.
Real-time fatigue driving detection system based on multi-module fusion
2022, Computers and Graphics (Pergamon)Citation Excerpt :Therefore, it can be considered that fatigue driving behavior is different from normal driving behavior [13]. We can evaluate whether the driver is in a fatigued driving state in real-time through the operation of the steering wheel [14], the speed, the track deviation, and the driving track of the vehicle [15]. The third method is to judge the fatigue state according to the driver’s facial features [9,16].
Driver distraction detection based on vehicle dynamics using naturalistic driving data
2022, Transportation Research Part C: Emerging TechnologiesCitation Excerpt :Among these studies, models based on NDS were found to be more accurate than ones based on driving simulator data. Recently, researchers developed a state-space model to detect visual-manual distraction tasks like texting (Sun, et al., 2021), which may not be the most appropriate as state-space model handles stable data well, but driving data is not always stable, especially when the driver is distracted. In addition to traditional machine learning methods, neural networks, as a powerful tool to solve nonlinear problems, were also used for driver distraction detection (Tango and Botta, 2013; Im et al., 2014; Yang et al., 2015; Ye et al., 2017).
Neuromorphic Driver Monitoring Systems: A Computationally Efficient Proof-of-Concept for Driver Distraction Detection
2023, IEEE Open Journal of Vehicular Technology