Online distraction detection for naturalistic driving dataset using kinematic motion models and a multiple model algorithm

https://doi.org/10.1016/j.trc.2021.103317Get rights and content

Highlights

  • Propose a general methodology to detect distracted behaviors based on vehicle kinematic data.

  • Implement the proposed method in ADAS to enhance driving safety.

  • Fuse multiple state–space models to track kinematic signals during normal driving.

  • Detect abnormal driving behaviors based on statistical quality control charts.

  • Validate the method using two naturalistic datasets (IVBSS & SPMD).

Abstract

Detecting distracted driving is important for developing Advanced Driver Assistance Systems and improving road safety. Most of the existing research analyzes drivers directly via video analysis techniques or by measuring cognitive load, however these approaches often require additional sensors to be installed in vehicles or equipped to drivers. Given that most distractions may have a direct influence on drivers’ control of vehicles, this paper proposes a new method to utilize available vehicle kinematic data for detecting distracted driving. The proposed method predicts vehicle kinematics by fusing multiple state–space models that capture different driving motion patterns under normal driving. An online monitoring scheme is developed by using Exponentially Weighted Moving Average (EWMA) and Cumulative Sum (CUSUM) charts, which detects abnormal mean shifts of lateral speeds and prediction errors of lane positions to provide warnings of distracted driving. A case study is presented based on two naturalistic driving datasets — the Integrated Vehicle-Based Safety Systems (IVBSS) and Safety Pilot Model Deployment (SPMD) datasets.

Introduction

Driving safety is an increasing concern in both transportation research and the automotive industry. In 2013, a global status report by the World Health Organization indicated 1.24 million traffic-related fatalities (Organization et al., 2013). This report defined the leading cause of death for people ages 1529 years as traffic-crashes. As about 70% of traffic-crashes are caused by human errors, one of important topics is the detection of distracted driving (Treat, 1980, Aberg and Rimmo, 1998).

Distracted driving occurs when a driver diverts his/her attention away from the driving task to focus on a secondary task (Wang et al., 2017a). For example, texting on a cellphone during driving is a common risky secondary task that causes a driver’s attention to both visually and manually divert from the main task of driving (Li et al., 2017). In 2015, the United States National Highway Traffic Safety Administrations found distracted driving to account for around 3477 fatalities (10% of overall fatalities) with an additional estimated 391,000 injuries in the United States (National Center for Statistics and Analysis, 2017). Furthermore, these numbers are likely under-reported due to difficulties in identifying driver distraction during crash investigation (Sundeen, 2006).

Various Advanced Driver Assistance Systems (ADAS) have been increasingly developed to prevent drivers from unsafe or dangerous situations relating to distracted driving. In general, ADAS typically follow two common approaches to detect distracted driving. The first most popular approach is to monitor drivers’ behavior directly via extra sensors installed within the cockpit of the vehicle, or via smartphone cameras, to capture how the driver’s head orientation is away from the driving direction. Within this approach, distraction behaviors are typically described as instances when drivers fail to focus their gaze towards the direction of driving for a pre-specified amount of time (Mihai et al., 2015, You et al., 2013, Xiao and Feng, 2016). Other forms of detecting distraction behaviors include monitoring drivers for a sign of fatigue via irregular blinking patterns or yawning (Eriksson and Papanikolopoulos, 2001, Zhu and Ji, 2004, Omidyeganeh et al., 2011). Moreover, time-spent-on-task has also been used as an indicator of distracted driving behaviors (Feng et al., 2009). ADAS using all above ways require using smartphone cameras. The issue of user-privacy still remains a discussion for the community. Therefore, implementation of the first type of approach is still limited in practice.

The second type of approach analyzes vehicle kinematics based on readily-available vehicle kinematic signals corresponding to speed, yaw rate, acceleration, etc. The principle is that the secondary tasks which distract the driver may have a direct influence on their control of the vehicle, and hence lead to abnormal vehicle kinematic signals. For example, Feng et al. (2017) identified aggressive driving behaviors based on unique characteristics of vehicle jerk in drivers’ gas and brake pedal operations. In ADAS research, kinematic signals have been used to detect unintentional drifting towards the boundary of the driving lane for developing lane departure warning (LDW) systems (Aksan et al., 2016, Jamson et al., 2008, Chen et al., 2018, Adell et al., 2011). Many models have been developed to predict the vehicle’s lateral trajectory (Angkititrakul et al., 2010, Wang et al., 2018, Narote et al., 2018) or its time to lane crossing (Cario et al., 2009). However, these methods mainly generate alarms when the vehicle is approaching the lane boundary, which may not warn the distracted driver in time. In contrast, our paper aims to detect distraction driving behaviors using a different anomaly detection strategy, that is, to firstly predict the vehicle kinematics under the normal driving, and then to trigger alarms when the directly measured vehicle kinematics are significantly different from the normal behavior predicted by the model. In this way, the ADAS can generate timely alarms to warn the distracted driver when their kinematic driving behaviors are significantly changed.

A key component of the proposed approach is the prediction of vehicle kinematics. Three different types of models are often used to predict vehicle kinematics: driving maneuver models, vehicle dynamic models, or state–space models (Lefèvre et al., 2014). Driving maneuver models predict vehicle kinematics based on different driving maneuvers. For example, Wang and Zhao (2018) used a hierarchical Dirichlet process with a Hidden Semi-Markov Model (HDP-HSMM) to learn normal maneuvers for making predictions. These models usually do not rely on physics models, which may generate less accurate predictions. However, a high prediction accuracy of vehicle kinematic signals is often required to distinguish the distracted driving behaviors from natural variations.

Vehicle dynamic models are often built based on Lagrange’s equations and consider dynamic force responses such as longitudinal tire forces or road banking angle (Rajamani, 2011). However, the parameters (such as slip angle) in these models vary under different weather and road conditions and can thus be intractable for real-time motion prediction (Pacejka, 2005, Cossalter, 2006).

State–space models (Schubert et al., 2008) track the vehicle kinematics based on typical kinematic physics equations. The Constant Velocity (CV) and Constant Acceleration (CA) models are often used when the vehicle is driven on a straight road. The Constant Turn Rate and Velocity (CTRV) and Constant Turn Rate and Acceleration (CTRA) models are applied when the vehicle is driven on a curvy road. Sun et al. (2015) detected lane-level irregular driving behaviors using GPS signals in a simulation dataset. While these models are able to provide predictions for a variety of scenarios, state–space models have seldom been implemented in ADAS for distraction detection with naturalistic vehicle kinematic data.

In this paper, an online algorithm is proposed to detect distracted driving behaviors based on vehicle kinematic prediction using state–space models. The proposed method involves two steps. One is to fuse multiple state–space models to predict vehicle kinematics. The other is to develop a control chart-based decision strategy to detect abnormal mean shifts of lateral speeds and prediction errors of lane positions to provide warnings of distracted driving. Our proposed algorithm aims to enhance ADAS by detecting distracted driving based on readily-available vehicle kinematic signals. This paper is organized as follows: Section 2 describes notation, data structure, and the formulation of the problem. Section 3 elaborates, step-by-step, the proposed distraction detection algorithm. Section 4 presents a case study which uses vehicle kinematic data to detect distraction behaviors. The paper will then conclude with a discussion.

Section snippets

Notations and problem formulation

A naturalistic driving dataset containing multivariate vehicle kinematic signals, including longitudinal speed, lateral speed, longitudinal acceleration, yaw rate and lane position based on cameras was used in this study (Sayer et al., 2011). All the signals were synchronized and resampled with a frequency of 10Hz for easier analysis. Let zt denote the vehicle kinematic signals at time t. Each zt=z1,t,,zq,tT is a qdimensional vector of the measured motion signals at time t. Zt1=z1,,zt1

Predict vehicle kinematics using state–space models

As a start, zt is predicted based on state–space models under different driving motion patterns 1,,C. Each of the state–space models is a special case of the Kalman Filter (Kalman, 1960). Let xtRp denote the state vector, which is a pdimensional vector of the true vehicle kinematic signals. A known transition matrix F updates xt1 to xt as xt=Fxt1+wt,where F is the p×p linear transition matrix, and wt is the process noise that follows the Gaussian distribution N(0,Q). The p×p covariance

Case study

In this section, the proposed method is applied to the vehicle kinematic data in two existing naturalistic driving datasets — the Integrated Vehicle-Based Safety Systems (IVBSS) (Sayer et al., 2011) and Safety Pilot Model Deployment (SPMD) datasets (Bezzina and Sayer, 2014).

The IVBSS program was designed to build and test an integrated in-vehicle crash warning system that includes forward crash warning, lane departure warning, curve speed warning, and lane change warning. From 2009 to 2010 a

Discussion

The paper presents a general methodology to detect the distraction behaviors that lead to abnormal driving kinematic signals. For the demonstration purpose, the method is validated by the detection performance of texting behaviors during high-speed driving on freeways (Feng et al., 2018). Additional efforts are expected to develop a comprehensive ADAS that is capable of detecting distractions under all circumstances.

Firstly, the current algorithm is trained based on vehicle kinematic signals

CRediT authorship contribution statement

Wenbo Sun: Conceptualization, Methodology, Software, Formal analysis, Data curation, Writing – original draft, Writing – review & editing. Matthew Aguirre: Conceptualization, Validation, Writing – original draft, Writing – review & editing. Jionghua (Judy) Jin: Conceptualization, Methodology, Writing – review & editing, Supervision, Funding acquisition. Fred Feng: Conceptualization, Methodology, Software, Investigation, Data curation, Writing - review & editing. Samer Rajab: Conceptualization,

Acknowledgments

The research is jointly supported by Honda R&D Americas, Inc. and Michigan Institute for Data Science (MIDAS) .

References (44)

  • AbergL. et al.

    Dimensions of aberrant driver behaviour

    Ergonomics

    (1998)
  • AksanN. et al.

    Benefits from Heads-Up Lane Departure Warnings Predicts Safety in the Real-WorldTech. Report

    (2016)
  • AngkititrakulP. et al.

    On the use of stochastic driver behavior model in lane departure warning

    IEEE Trans. Intell. Transp. Syst.

    (2010)
  • BezzinaD. et al.

    Safety Pilot Model Deployment: Test Conductor Team ReportReport No. DOT HS, 812

    (2014)
  • BurdickR.K. et al.

    Design and Analysis of Gauge R&R Studies: Making Decisions with Confidence Intervals in Random and Mixed ANOVA Models, Vol. 17

    (2005)
  • Cario, G., Casavola, A., Franze, G., Lupia, M., Brasili, G., it SpA, I., 2009. Predictive time-to-lane-crossing...
  • CossalterV.

    Motorcycle Dynamics

    (2006)
  • FengF. et al.

    Estimation of lead vehicle kinematics using camera-based data for driver distraction detection

    Int. J. Automotive Eng.

    (2018)
  • FengR. et al.

    An on-board system for detecting driver drowsiness based on multi-sensor data fusion using Dempster-Shafer theory

  • JulierS.J. et al.

    A General Method for Approximating Nonlinear Transformations of Probability DistributionsTechnical Report

    (1996)
  • JulierS.J. et al.

    Unscented filtering and nonlinear estimation

    Proc. IEEE

    (2004)
  • KalmanR.E.

    A new approach to linear filtering and prediction problems

    J. Basic Eng.

    (1960)
  • Cited by (11)

    • Distracted driving detection based on the fusion of deep learning and causal reasoning

      2023, Information Fusion
      Citation Excerpt :

      The method proposed by Sun et al. established vehicle motion patterns under normal driving by fusing multiple state space models, and then detects distracted driving online based on exponentially weighted moving average and cumulative sum charts. And the performance of the method was validated by identifying the texting behavior of drivers while driving at high speeds [19]. Behavioral analysis methods based on vehicle data mining generally require accurate real-time data.

    • Real-time fatigue driving detection system based on multi-module fusion

      2022, Computers and Graphics (Pergamon)
      Citation Excerpt :

      Therefore, it can be considered that fatigue driving behavior is different from normal driving behavior [13]. We can evaluate whether the driver is in a fatigued driving state in real-time through the operation of the steering wheel [14], the speed, the track deviation, and the driving track of the vehicle [15]. The third method is to judge the fatigue state according to the driver’s facial features [9,16].

    • Driver distraction detection based on vehicle dynamics using naturalistic driving data

      2022, Transportation Research Part C: Emerging Technologies
      Citation Excerpt :

      Among these studies, models based on NDS were found to be more accurate than ones based on driving simulator data. Recently, researchers developed a state-space model to detect visual-manual distraction tasks like texting (Sun, et al., 2021), which may not be the most appropriate as state-space model handles stable data well, but driving data is not always stable, especially when the driver is distracted. In addition to traditional machine learning methods, neural networks, as a powerful tool to solve nonlinear problems, were also used for driver distraction detection (Tango and Botta, 2013; Im et al., 2014; Yang et al., 2015; Ye et al., 2017).

    View all citing articles on Scopus
    View full text