Skip to main content
SpringerLink
Log in

Temporal and spatial feature based approaches in drowsiness detection using deep learning technique

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Drowsiness is a term which seems to be very simple but for a moment, it becomes a critical issue for many drivers and workers while they are performing their duty. Many people’s lives may collapse into trouble because of drowsiness. Therefore, such a real-time system is needed which can be easy to develop and configure for early as well as accurate drowsiness detection. As per requisite, we have adopted a large realistic dataset which includes 30 h video of 60 different participants in three classes, i.e. alert, low vigilant and drowsy. In our proposed work, we have selected the videos with extreme classes, i.e. alert and drowsy only. Further, we have designed two different models based on temporal and spatial feature by employing computer vision as well as deep-learning approach. In one model, temporal features are obtained by computer vision techniques followed by long short-term memory (LSTM) and the second model adopts spatial features extraction through convolution neural network (CNN) followed by LSTM. Although the temporal model is more complex and has less accuracy than spatial model, in spite of this, the study shows that the temporal model is far better in terms of training time than spatial model by establishing the comparison using confusion metrics and Area under Curve (AUC)–Receiver-Operating Characteristic Curve (ROC) score.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Sleepy and unsafe: Data from Sleepy and unsafe May 2014. https://www.safetyandhealthmagazine.com/articles/10412-sleepy-and-unsafe-worker-fatigue (2014). Accessed 8 Jan 2020

  2. Wheaton, A.G., Shults, R.A., Chapman, D.P., Ford, E.S., Croft, J.B.: Drowsy driving and risk behaviors—10 states and Puerto Rico (2011–2012). Morbid Mortal Weekly Rep (MMWR) 63, 557–562 (2014)

    Google Scholar 

  3. P.-C. L. C. J. R. D., Wheaton, A.G., Chapman, D.P.: Drowsy driving 19 states and the district of Columbia (2009–2010). Morb. Mortal. Wkly. Rep. (MMWR) 63, 1033–1037 (2013)

  4. Sadeghniiat-Haghighi, K., Yazdi, Z.: Fatigue management in the workplace. Ind Psychiatry J 24, 1–12 (2015)

    Article  Google Scholar 

  5. Schmidt, E.A., Kincses, W.E., Scharuf, M., Haufe, S., Schubert, R., Curio, G.: Assessing Drivers’ Vigilance State During Monotonous Driving. University of Iowa (2007)

  6. Bak, M., Borkowski, P., de Stasio, C.: Transferability of ICT solutions for improving co-modality in passenger transport. In: Transport Research Arena (TRA) 5th Conference (2014)

  7. Forsman, P.M., Vila, B.J., Short, R.A., Mott, C.G., Van Dongen, H.P.A.: Efficient driver drowsiness detection at moderate levels of drowsiness. Accid. Anal. Prev. 50, 341–350 (2013)

    Article  Google Scholar 

  8. Reddy, B., Kim, Y.-H., Yun, S., Seo, C., Jang, J.: Real-time driver drowsiness detection for embedded system using model compression of deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 121–128 (2017)

  9. Tadesse, E., Sheng, W., Liu, M.: Driver drowsiness detection through HMM based dynamic modelling. In: International Conference on Robotics and Automation (ICRA), pp. 4003–4008. IEEE (2014)

  10. Liu, C.C., Hosking, S.G., Lenné, M.G.: Predicting driver drowsiness using vehicle measures: recent insights and future challenges. J Saf Res 40, 239–245 (2009)

    Article  Google Scholar 

  11. Manoharan, K., Daniel, P.: Survey on various lane and driver detection techniques based on image processing for hilly terrain. IET Image Proc. 12, 1511–1520 (2018)

    Article  Google Scholar 

  12. Simon, M., Schmidt, E.A., Kincses, W.E., Fritzsche, M., Bruns, A., Aufmuth, C., Bogdan, M., Rosenstiel, W., Schrauf, M.: EEG alpha spindle measures as indicators of driver fatigue under real traffic conditions. Clin. Neurophysiol. 122, 1168–1178 (2011)

    Article  Google Scholar 

  13. Massoz, Q., Langohr, T., François, C., Verly, J.G.: The ULg multimodality drowsiness database (called DROZY) and examples of use. In: Winter Conference on Applications of Computer Vision (WACV), pp. 1–7. IEEE, (2016)

  14. Svensson, U.: Blink behaviour based drowsiness detection. 0347-6049, (2004)

  15. Bergasa, L.M., Nuevo, J., Sotelo, M.A., Barea, R., Lopez, M.E.: Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Transport Syst 7, 63–77 (2006)

    Article  Google Scholar 

  16. Panda, S., Kolhekar, M.: Feature selection for driver drowsiness detection. In: Proceedings of International Conference on Computational Intelligence and Data Engineering, pp. 127–140. Springer, Singapore (2019)

  17. Huang, R., Wang, Y., Guo, L.: P-FDCN based eye state analysis for fatigue detection. In: 18th International Conference on Communication Technology (ICCT), pp. 1174–1178. IEEE (2018)

  18. Naqvi, R.A., Arsalan, M., Batchuluun, G., Yoon, H.S., Park, K.R.: Deep learning-based gaze detection system for automobile drivers using a NIR camera sensor. Sensors 18, 456 (2018)

    Article  Google Scholar 

  19. Wang, Y., Huang, R., Guo, L.: Eye gaze pattern analysis for fatigue detection based on GP-BCNN with ESM. Pattern Recogn. Lett. 123, 61–74 (2019)

    Article  Google Scholar 

  20. Alshaqaqi, B., Baquhaizel, A.S., Ouis, M.E.A., Boumehed, M., Ouamri, A., Keche, M.: Driver drowsiness detection system. In: 8th International Workshop on Systems, Signal Processing and their Applications (WoSSPA), pp.151–155. IEEE (2013)

  21. Ghoddoosian, R., Galib, M., Athitsos, V.: A realistic dataset and baseline temporal model for early drowsiness detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)

  22. RLDD: dataset created by The University of Texas at Arlington in 2019. https://sites.google.com/view/utarldd/home . Accessed 10 Jan 2020

  23. https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/

  24. How to Retrain Inception’s final layer for New Categories https://chromium.googlesource.com/external/github.com/tensorflow/tensorflow/+/r0.10/tensorflow/g3doc/how_tos/image_retraining/index.md. Accessed 5 Feb 2020

  25. Rosebrock, A.: Eye motion tracking, Jan 4, 2019 https://www.youtube.com/watch?v=kbdbZFT9NQI (2019). Accessed 20 Jan 2020

  26. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzz. Knowl. Based Syst. 6(02), 107–116 (1998)

    Article  Google Scholar 

  27. Weng, C.-H., Lai, Y.-H., Lai, S.-H.: Driver drowsiness detection via a hierarchical temporal deep belief network. In: Asian Conference on Computer Vision, pp. 117–133. Springer, Cham (2016)

  28. Mittal, A.: Understanding RNN and LSTM, Oct 12, 2019. https://towardsdatascience.com/understanding-rnn-and-lstm-f7cdf6dfc14e (2019). Accessed 20 Jan 2020

  29. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  30. Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J.: LSTM: a search space odyssey. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2222–2232 (2016)

    Article  MathSciNet  Google Scholar 

  31. Gandhi, N.: Drowsy driver detection system in video sequences using LSTM with CNN features, Jan 1, 2018. https://github.com/nishagandhi/DrowsyDriverDetection/blob/master/Drowsy-Driver-Detection.pdf. Accessed 24 Jan 2020

  32. Patil, A.: Real drowsiness detection using Viola–Jones algorithm in tensor flow. In: Machine Learning and Information Processing, pp. 317–329. Springer, Singapore (2020)

  33. Yang, S., Nian, F., Wang, Y., et al.: Real-time face attributes recognition via HPGC: horizontal pyramid global convolution. J. Real-Time Image Proc. 17, 1829–1840 (2020)

    Article  Google Scholar 

  34. Li, X., Wu, Y., Zhang, W., et al.: Deep learning methods in real-time image super-resolution: a survey. J. Real Time Image Proc. 17, 1885–1909 (2020)

    Article  Google Scholar 

  35. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826 (2016)

  36. Dong, N., et al.: Inception v3 based cervical cell classification combined with artificially extracted features. Appl. Soft. Comput. 93, 106311 (2020)

    Article  Google Scholar 

  37. Deng, J., Berg, A., Satheesh, S., Su, H., Khosla, A., Fei-Fei, L.: ILSVRC-2012. https://www.imagenet.org/challenges/LSVRC/2012/. Accessed 26 Jan 2020

  38. Shakeel, M.F., Bajwa, N.A., Anwaar, A.M., Sohail, A., Khan, A.: Detecting driver drowsiness in real time through deep learning based object detection. In: International Work-Conference on Artificial Neural Networks, pp. 283–296. Springer, Cham (2019)

  39. Tran, D., Do, H.M., Sheng, W., Bai, H., Chowdhary, G.: Real-time detection of distracted driving based on deep learning. IET Intell. Transport. Syst. 12(10), 1210–1219 (2018)

    Article  Google Scholar 

  40. Tümen, V., Yıldırım, Ö., Ergen, B.: Detection of driver drowsiness in driving environment using deep learning methods. In: 2018 Electric Electronics, Computer Science, Biomedical Engineering’s’ Meeting (EBBT), pp. 1–5. IEEE (2018)

Download references

Author information

Authors and Affiliations

  1. National Institute of Technology Silchar, Silchar, Assam, India

    Nageshwar Nath Pandey & Naresh Babu Muppalaneni

Authors
  1. Nageshwar Nath Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Naresh Babu Muppalaneni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nageshwar Nath Pandey.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pandey, N.N., Muppalaneni, N.B. Temporal and spatial feature based approaches in drowsiness detection using deep learning technique. J Real-Time Image Proc 18, 2287–2299 (2021). https://doi.org/10.1007/s11554-021-01114-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01114-x

Keywords

Search

Navigation