Convolutional neural networks (CNNs) have achieved remarkable performance in driver drowsiness detection based on the extraction of deep features of drivers’ faces. However, the performance of driver drowsiness detection methods decreases sharply when complications, such as illumination changes in the cab, occlusions and shadows on the driver’s face, and variations in the driver’s head pose, occur. In addition, current driver drowsiness detection methods are not capable of distinguishing between driver states, such as talking versus yawning or blinking versus closing eyes. Therefore, technical challenges remain in driver drowsiness detection. In this article, we propose a novel and robust two-stream spatial_temporal graph convolutional network (2s-STGCN) for driver drowsiness detection to solve the above-mentioned challenges. To take advantage of the spatial and temporal features of the input data, we use a facial landmark detection method to extract the driver’s facial landmarks from real-time videos and then obtain the driver drowsiness detection result by 2s-STGCN. Unlike existing methods, our proposed method uses videos rather than consecutive video frames as processing units. This is the first effort to exploit these processing units in the field of driver drowsiness detection. Moreover, the two-stream framework not only models both the spatial and temporal features but also models both the first-order and second-order information simultaneously, thereby notably improving driver drowsiness detection. Extensive experiments have been performed on the yawn detection dataset (YawDD) and the National TsingHua University drowsy driver detection (NTHU-DDD) dataset. The experimental results validate the feasibility of the proposed method. This method achieves an average accuracy of 93.4% on the YawDD dataset and an average accuracy of 92.7% on the evaluation set of the NTHU-DDD dataset.

中文翻译：

---

用于驾驶员睡意检测的双流时空图卷积网络


卷积神经网络（CNN）在基于提取驾驶员面部深层特征的驾驶员睡意检测方面取得了显着的性能。然而，当驾驶室内的照明变化、驾驶员面部的遮挡和阴影以及驾驶员头部姿势的变化等复杂情况发生时，驾驶员困倦检测方法的性能急剧下降。此外，当前的驾驶员困倦检测方法无法区分驾驶员状态，例如说话与打哈欠或眨眼与闭眼。因此，驾驶员困倦检测仍然存在技术挑战。在本文中，我们提出了一种新颖且鲁棒的双流空间时间图卷积网络（2s-STGCN），用于驾驶员睡意检测，以解决上述挑战。为了利用输入数据的空间和时间特征，我们使用面部标志检测方法从实时视频中提取驾驶员的面部标志，然后通过2s-STGCN获得驾驶员睡意检测结果。与现有方法不同，我们提出的方法使用视频而不是连续视频帧作为处理单元。这是在驾驶员困倦检测领域利用这些处理单元的首次尝试。此外，双流框架不仅对空间和时间特征进行建模，还同时对一阶和二阶信息进行建模，从而显着改善了驾驶员困倦检测。在哈欠检测数据集（YawDD）和国立清华大学昏昏欲睡驾驶员检测（NTHU-DDD）数据集上进行了大量的实验。实验结果验证了所提方法的可行性。 该方法在 YawDD 数据集上的平均准确率达到 93.4%，在 NTHU-DDD 数据集的评估集上平均准确率达到 92.7%。