Sebastian Zepf
Javier Hernandez
Skip Abstract Section
Abstract
Driving can occupy a large portion of daily life and often can elicit negative emotional states like anger or stress, which can significantly impact road safety and long-term human health. In recent decades, the arrival of new tools to help recognize human affect has inspired increasing interest in how to develop emotion-aware systems for cars. To help researchers make needed advances in this area, this article provides a comprehensive literature survey of work addressing the problem of human emotion recognition in an automotive context. We systematically review the literature back to 2002 and identify 63 peer-review published articles on this topic. We overview each study’s methodology to measure and recognize emotions in the context of driving. Across the literature, we find a strong preference toward studying emotional states associated with high arousal and negative valence, monitoring the different states with cardiac, electrodermal activity, and speech signals, and using supervised machine learning to automatically infer the underlying human affective states. This article summarizes the existing work together with publicly available resources (e.g., datasets and tools) to help new researchers get started in this field. We also identify new research opportunities to help advance progress for improving driver emotion recognition.
- Fadel Adib, Hongzi Mao, Zachary Kabelac, Dina Katabi, and Robert C. Miller. 2015. Smart homes that monitor breathing and heart rate. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, NY, 837--846.Google Scholar
- Urvashi Agrawal, Shubhangi Giripunje, and Preeti Bajaj. 2013. Emotion and gesture recognition with soft computing tool for drivers assistance system in human centered transportation. In Proceedingsof the 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC’13). 4612--4616. DOI:https://doi.org/10.1109/SMC.2013.785Google ScholarDigital Library
- Ahmet Akbas. 2011. Evaluation of the physiological data indicating the dynamic stress level of drivers. Scientific Research and Essays 6, 2 (2011), 430--439. DOI:https://doi.org/10.5897/SRE10.943Google Scholar
- Ignacio Alvarez, Karmele Lopez de Ipiña, Shaundra B. Daily, and Juan E. Gilbert. 2012. Emotional adaptive vehicle user interfaces: Moderating negative effects of failed technology interactions while driving. In Adjunct Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 57--60.Google Scholar
- Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. OpenFace: A General-Purpose Face Recognition Library with Mobile Applications. CMU School of Computer Science, Pittsburgh, PA.Google Scholar
- Pongtep Angkititrakul, John H. L. Hansen, Sangjo Choi, Tyler Creek, Jeremy Hayes, Jeonghee Kim, Donggu Kwak, Levi T. Noecker, and Anhphuc Phan. 2009. UTDrive: The smart vehicle project. In In-Vehicle Corpus and Signal Processing for Driver Behavior, K. Takeda, J. H. L. Hangen, H. Erdogan, and H. Abut (Eds.). Springer, 55--67. DOI:https://doi.org/10.1007/978-0-387-79582-9Google Scholar
- Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, and Seth D. Pollak. 2019. Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest 20, 1 (2019), 1--68. DOI:https://doi.org/10.1177/1529100619832930Google ScholarCross Ref
- Ahahina Begum, Mobyen Uddin Ahmed, Ahmed Mobyen Uddin, Peter Funk, and Reno Filla. 2012. Mental state monitoring system for the professional drivers based on heart rate variability analysis and case-based reasoning. In Proceedings of the Federal Conference on Computer Science and Information Systems (FedSIS’12). 35--42. http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=8arnumber=6354476.Google Scholar
- Paul Boersma. 2014. The use of Praat in corpus research. In The Oxford Handbook of Corpus Phonology, J. Durand, U. Gut, and G. Kristoffersen (Eds.). Oxford Handbooks Online, 342--360. DOI:https://doi.org/10.1093/oxfordhb/9780199571932.013.016Google Scholar
- Gianluca Borghini, Laura Astolfi, Giovanni Vecchiato, Donatella Mattia, and Fabio Babiloni. 2014. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience and Biobehavioral Reviews 44 (2014), 58--75.Google ScholarCross Ref
- Hynek Boril, Pinar Boyraz, and John H. L. Hansen. 2012. Towards multimodal driver’s stress detection. In Digital Signal Processing for In-Vehicle Systems and Safety, J. H. L. Hansen, P. Boyraz, K. Takeda, and H. Abut (Eds.). Springer, 3--19. DOI:https://doi.org/10.1007/978-1-4419-9607-7Google Scholar
- Hynek Boril, Tristan Kleinschmidt, Pinar Boyraz, and John H. L. Hansen. 2010. Impact of cognitive load and frustration on drivers’ speech.Journal of the Acoustical Society of America 127 (Sept. 2010), 1996. DOI:https://doi.org/10.1121/1.3385171 arxiv:33168Google Scholar
- H. Boril, S. O. Sadjadi, and J. H. L. Hansen. 2011. UTDrive: Emotion and cognitive load classification for in-vehicle scenarios. In Proceedings of the 5th Biennial Workshop on Digital Signal Processing for In-Vehicle Systems (DSP’11). http://www.utd.edu/∼hynek/pdfs/BorilSadjadiHansen_DSP11.pdf.Google Scholar
- Wolfram Boucsein. 2012. Electrodermal Activity. Springer Science 8 Business Media.Google Scholar
- Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1 (1994), 49--59. DOI:https://doi.org/10.1016/0005-7916(94)90063-9 arxiv:0005-7916(93)E0016-ZGoogle ScholarCross Ref
- Johnell O. Brooks, Richard R. Goodenough, Matthew C. Crisler, Nathan D. Klein, Rebecca L. Alley, Beatrice L. Koon, William C. Logan, Jennifer H. Ogle, Richard A. Tyrrell, and Rebekkah F. Wills. 2010. Simulator sickness during driving simulation studies. Accident Analysis and Prevention 42, 3 (2010), 788--796. DOI:https://doi.org/10.1016/j.aap.2009.04.013Google ScholarCross Ref
- John T. Cacioppo and Louis G. Tassinary. 1990. Inferring psychological significance from physiological signals. American Psychologist 45, 1 (1990), 16--28. DOI:https://doi.org/10.1037/0003-066X.45.1.16Google ScholarCross Ref
- Hua Cai and Yingzi Lin. 2011. Modeling of operators emotion and task performance in a virtual driving environment. International Journal of Human Computer Studies 69, 9 (2011), 571--586. DOI:https://doi.org/10.1016/j.ijhcs.2011.05.003Google ScholarDigital Library
- Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2D pose estimation using part affinity fields. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). DOI:https://doi.org/10.1109/CVPR.2017.143 arxiv:1611.08050Google ScholarCross Ref
- Sailesh Conjeti, Rajiv Ranjan Singh, and Rahul Banerjee. 2012. Bio-inspired wearable computing architecture and physiological signal processing for on-road stress monitoring. Biomedical and Health Informatics 1, 0 (2012), 1--7.Google Scholar
- Albert C. Cruz and Alex Rinaldi. 2017. Video summarization for expression analysis of motor vehicle operators. In Proceedings of the International Conference on Universal Access in Human-Computer Interaction. 313--323. DOI:https://doi.org/10.1007/978-3-319-58706-6Google Scholar
- Yong Deng, Zhonghai Wu, Chao Hsien Chu, Qixun Zhang, and D. Frank Hsu. 2013. Sensor feature selection and combination for stress identification using combinatorial fusion. International Journal of Advanced Robotic Systems 10, 8 (2013), 306. DOI:https://doi.org/10.5772/56344Google ScholarCross Ref
- Ding Ding, Klaus Gebel, Philayrath Phongsavan, Adrian E. Bauman, and Dafna Merom. 2014. Driving: A road to unhealthy lifestyles and poor health outcomes. PloS One 9, 6 (June 2014), 1--5. DOI:https://doi.org/10.1371/journal.pone.0094602Google Scholar
- Monique Dittrich and Sebastian Zepf. 2019. Exploring the validity of methods to track emotions behind the wheel. In Proceedings of the International Conference on Persuasive Technology. 115--127.Google ScholarCross Ref
- Yanchao Dong, Zhencheng Hu, Keiichi Uchimura, and Nobuki Murayama. 2011. Driver inattention monitoring system for intelligent vehicles: A review. IEEE Transactions on Intelligent Transportation Systems 12, 2 (2011), 596--614. DOI:https://doi.org/10.1109/TITS.2010.2092770Google ScholarDigital Library
- Paul Ekman, Richard Davidson, Phoebe Ellsworth, Wallace V. Friesen, Robert Levenson, Harriet Oster, and Erika Rosenberg. 1992. Are there basic emotions? Psychological Review 99, 3 (1992), 550--553. DOI:https://doi.org/10.1037/0033-295X.99.3.550 arxiv:arXiv:1011.1669v3Google ScholarCross Ref
- Paul Ekman and Wallace V. Friesen. 1978. Facial Action Coding System: Investigator’s Guide. Consulting Psychologists Press.Google Scholar
- Neska El Haouij, Jean Michel Poggi, Raja Ghozi, Sylvie Sevestre-Ghalila, and Mériem Jaïdane. 2018. Random forest-based approach for physiological functional variable selection for driver’s stress level classification. Statistical Methods 8 Applications 28 (2018), 157--185. DOI:https://doi.org/10.1007/s10260-018-0423-5Google Scholar
- Luis Eudave and Miguel Valencia. 2017. Physiological response while driving in an immersive virtual environment. In Proceedings of the 2017 IEEE 14th International Conference on Wearable and Implantable Body Sensor Networks (BSN’17). 145--148. DOI:https://doi.org/10.1109/BSN.2017.7936028Google ScholarCross Ref
- Florian Eyben, Martin Wöllmer, Tony Poitschke, Björn Schuller, Christoph Blaschke, Berthold Färber, and Nhu Nguyen-Thien. 2010. Emotion on the road—Necessity, acceptance, and feasibility of affective computing in the car. Advances in Human-Computer Interaction 2010 (2010), Article 5.Google Scholar
- Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. OpenSMILE: The Munich versatile and fast open-source audio feature extractor. In Proceedings of ACM Multimedia. 1459--1462. DOI:https://doi.org/10.1145/1873951.1874246Google ScholarDigital Library
- Stephen H. Fairclough, Andrew J. Tattersall, and Kim Houston. 2006. Anxiety and performance in the British driving test. Transportation Research Part F: Traffic Psychology and Behaviour 9, 1 (2006), 43--52. DOI:https://doi.org/10.1016/j.trf.2005.08.004Google ScholarCross Ref
- Raul Fernandez and Rosalind W. Picard. 2003. Modeling driver’s speech under stress. Speech Communication 40 (2003), 145--149.Google ScholarDigital Library
- H. Guo, A, Yüce, and J.-P. Thiran. 2014. Detecting emotional stress from facial expressions for driving safety. In Proceedings of the IEEE International Conference on Image Processing (ICIP’14), Vol. 1. 5961--5965.Google ScholarCross Ref
- Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. 2014. Physiobank, PhysioToolkit, and PhysioNet. Retrieved May 12, 2020 from https://www.ahajournals.org/doi/full/10.1161/01.cir.101.23.e215.Google Scholar
- GPSBabel. 2019. Home Page. Retrieved May 12, 2020 from https://www.gpsbabel.org/.Google Scholar
- Michael Grimm, Kristian Kroschel, Helen Harris, Clifford Nass, Bjorn Björn Schuller, Gerhard Rigoll, and Tobias Moosmayr. 2007. On the necessity and feasibility of detecting a driver’s emotional state while driving. Affective Computing and Intelligent Interaction 4738 (2007), 126--138. DOI:https://doi.org/10.1007/978-3-540-74889-2_12Google ScholarDigital Library
- Markus Groth, Thorsten Hennig-Thurau, and Gianfranco Walsh. 2009. Customer reactions to emotional labor: The roles of employee acting strategies and customer detection accuracy. Academy of Management Journal 52, 5 (2009), 958--974. DOI:https://doi.org/10.5465/AMJ.2009.44634116Google ScholarCross Ref
- Oleg Gusikhin, Erica Klampfl, Dimitar Filev, and Yifan Chen. 2011. Emotive driver advisor system (EDAS). In Informatics in Control, Automation and Robotics. Lecture Notes in Electrical Engineering, Vol. Springer, 21--36. DOI:https://doi.org/10.1007/978-3-642-19539-6_2Google Scholar
- Markus Gutmann, Patrik Grausberg, and Kyandoghere Kyamakya. 2015. Detecting human driver’s physiological stress and emotions using sophisticated one-person cockpit vehicle simulator. In Proceedings of the 2015 Information Technologies in Innovation Business Conference (ITIB’15). 15--18. DOI:https://doi.org/10.1109/ITIB.2015.7355064Google ScholarCross Ref
- Mark A. Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations 11, 1 (2009), 10--18. DOI:https://doi.org/10.1145/1656274.1656278 arxiv:arXiv:1011.1669v3Google ScholarDigital Library
- Helen Harris and Clifford Nass. 2011. Emotion regulation for frustrating driving contexts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 749--752.Google ScholarDigital Library
- Jennifer A. Healey and Rosalind W. Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 6, 2 (2005), 156--166. DOI:https://doi.org/10.1109/TITS.2005.848368Google ScholarDigital Library
- Javier Hernandez, Zicheng Liu, Geoff Hulten, Dave DeBarr, Kyle Krum, and Zhengyou Zhang. 2013. Measuring the engagement level of TV viewers. In Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG’13). IEEE, Los Alamitos, CA, 1--7.Google ScholarCross Ref
- Javier Hernandez, Daniel McDuff, Xavier Benavides, Judith Amores, Pattie Maes, and Rosalind Picard. 2014. AutoEmotive: Bringing empathy to the driving experience to manage stress. In Proceedings of the 2014 Companion Publication on Designing Interactive Systems. 53--56.Google ScholarDigital Library
- Javier Hernandez, Rob R. Morris, and Rosalind W. Picard. 2011. Call center stress recognition with person-specific models. In Affective Computing and Intelligent Interaction. Lecture Notes in Computer Science, Vol. 6974, Springer, 125--134. DOI:https://doi.org/10.1007/978-3-642-24600-5_16Google ScholarCross Ref
- Stefan Hoch, Frank Althoff, G. McGlaun, and G. Rigoll. 2005. Bimodal fusion of emotional data in an automotive environment. In Proceedings of the IEEE International Conference on Acoustic, Speech, and Signal Processing. 1085--1088.Google Scholar
- Klas Ihme, Christina Dömeland, Maria Freese, and Meike Jipp. 2018. Frustration in the Face of the Driver: A Simulator Study on Facial Muscle Activity During Frustrated Driving. Interaction Studies. John Benjamins Publishing Company.Google Scholar
- Myounghoon Jeon. 2016. Don’t cry while you’re driving: Sad driving is as bad as angry driving. International Journal of Human-Computer Interaction 32, 10 (2016), 777--790. DOI:https://doi.org/10.1080/10447318.2016.1198524Google ScholarCross Ref
- Myounghoon Jeon, Jason Roberts, Parameshwaran Raman, Jung-Bin Yim, and Bruce N. Walker. 2011. Participatory design process for an in-vehicle affect detection and regulation system for various drivers. In Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility. 271--272.Google Scholar
- In Cheol Jeong, Dong Hee Lee, Shin Woo Park, Jae Il Ko, and Hyung Ro Yoon. 2007. Automobile driver’s stress index provision system that utilizes electrocardiogram. In Proceedings of the 2007 IEEE Intelligent Vehicles Symposium. 652--656. DOI:https://doi.org/10.1109/IVS.2007.4290190Google ScholarCross Ref
- Christian Jones and Ing Marie Jonsson. 2008. Using paralinguistic cues in speech to recognise emotions in older car drivers. In Affect and Emotion in Human-Computer Interaction. Lecture Notes in Computer Science, Vol. 4868. Springer, 229--240. DOI:https://doi.org/10.1007/978-3-540-85099-1_20Google Scholar
- Christian Martyn Jones and Ing-Marie Jonsson. 2005. Automatic recognition of affective cues in the speech of car drivers to allow appropriate responses. In Proceedings of the 17th Australia Conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future. 1--10.Google ScholarDigital Library
- Christian Martyn Jones and Ing Marie Jonsson. 2007. Performance analysis of acoustic emotion recognition for in-car conversational interfaces. In Universal Access in Human-Computer Interaction: Ambient Interaction. Lecture Notes in Computer Science, Vol. 4555. Springer, 411--420. DOI:https://doi.org/10.1007/978-3-540-73281-5_44Google Scholar
- Ing-Marie Jonsson, Clifford Nass, Helen Harris, and Leila Takayama. 2005. Matching in-car voice with driver state: Impact on attitude and driving performance. In Proceedings of the 3rd International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design. 173--180. DOI:https://doi.org/10.17077/drivingassessment.1158Google ScholarCross Ref
- O. Karaduman, H. Eren, H. Kurum, and M. Celenk. 2013. An effective variable selection algorithm for aggressive/calm driving detection via CAN bus. In Proceedings of the 2013 International Conference on Connected Vehicles and Expo (ICCVE’13). IEEE, Los Alamitos, CA, 586--591.Google Scholar
- Salman Karimi and Mohammad Hossein Sedaaghi. 2013. Robust emotional speech classification in the presence of babble noise. International Journal of Speech Technology 16, 2 (2013), 215--227. DOI:https://doi.org/10.1007/s10772-012-9176-yGoogle ScholarDigital Library
- Tomokazu Kato, Haruki Kawanaka, Md. Shoaib Bhuiyan, and Koji Oguri. 2011. Classification of positive and negative emotion evoked by traffic jam based on electrocardiogram (ECG) and pulse wave. In Proceedings of the 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC’11). IEEE, Los Alamitos, CA, 1217--1222.Google ScholarCross Ref
- Christos D. Katsis, N. Katertsidis, George Ganiatsas, and Dimitrios I. Fotiadis. 2008. Toward emotion recognition in car racing drivers: A biosignal processing approach. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 38, 3 (2008), 502--512. DOI:https://doi.org/10.1109/TSMCA.2008.918624Google Scholar
- Nobuo Kawaguchi and Shigeki Matsubara. 2001. Multimedia data collection of in-car speech communication. In Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech’01). 3--6.Google Scholar
- N. Keshan, P. V. Parimi, and I. Bichindaritz. 2015. Machine learning for stress detection from ECG signals in automobile drivers. In Proceedings of the 2015 IEEE International Conference on Big Data (IEEE Big Data’15). 2661--2669. DOI:https://doi.org/10.1109/BigData.2015.7364066Google ScholarDigital Library
- Abhiram Kolli, Alireza Fasih, Fadi Al Machot, and Kyandoghere Kyamakya. 2011. Non-intrusive car driver’s emotion recognition using thermal camera. In Proceedings of the 3rd International Workshop on Nonlinear Dynamics and Synchronization (INDS’11) and the 16th International Symposium on Theoretical Electrical Engineering (ISTET’11). IEEE, Los Alamitos, CA, 1--5.Google ScholarCross Ref
- Arun Sai Krishnan, Xiping Hu, Jun-Qi Deng, Li Zhou, Edith C.-H. Ngai, Xitong Li, Victor C. M. Leung, and Yu-kwong Kwok. 2015. Towards in time music mood-mapping for drivers: A novel approach. In Proceedings of the 5th ACM Symposium on Development and Analysis of Intelligent Vehicular Networks and Applications. 59--66. DOI:https://doi.org/10.1145/2815347.2815352Google ScholarDigital Library
- H. Leng, Y. Lin, and L. A. Zanzi. 2007. An experimental study on physiological parameters toward driver emotion recognition. Ergonomics and Health Aspects of Work with Computers 4566 (2007), 237--246. DOI:https://doi.org/10.1007/978-3-540-73333-1_30Google ScholarCross Ref
- Linda J. Levine. 1997. Reconstructing memory for emotions. Journal of Experimental Psychology: General 126, 2 (1997), 165.Google ScholarCross Ref
- Y. Lin, H. Leng, G. Yang, and H. Cai. 2007. An intelligent noninvasive sensor for driver pulse wave measurement. IEEE Sensors Journal 7, 5 (2007), 790--799. DOI:https://doi.org/10.1109/JSEN.2007.894923Google ScholarCross Ref
- C. Lisetti and F. Nasoz. 2005. Affective intelligent car interfaces with emotion recognition. Proceedings of 11th International Conference on Human Computer Interaction. 1--10. https://www.eurecom.fr/fr/publication/1797/download/mm-lisech-050722.pdf.Google Scholar
- Andreas Löcken, Klas Ihme, and Anirudh Unni. 2017. Towards designing affect-aware systems for mitigating the effects of in-vehicle frustration. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Adjunct (AutomotiveUI’17). 88--93. DOI:https://doi.org/10.1145/3131726.3131744Google ScholarDigital Library
- Zhiyi Ma, Marwa Mahmoud, Peter Robinson, Eduardo Dias, and Lee Skrypchuk. 2017. Automatic detection of a driver’s complex mental states. In Proceedings of the International Conference on Computational Science and Its Applications. 678--691.Google ScholarCross Ref
- Diana MacLean, Asta Roseway, and Mary Czerwinski. 2013. MoodWings. In Proceedings of the 6th International Conference on Pervasive Technologies Related to Assistive Environments (PETRA’13). DOI:https://doi.org/10.1145/2504335.2504406Google Scholar
- Marek Malik, J. Thomas Bigger, A. John Camm, Robert E. Kleiger, Alberto Malliani, Arthur J. Moss, and Peter J. Schwartz. 1996. Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. European Heart Journal 17, 3 (1996), 354--381.Google ScholarCross Ref
- L. Malta, P. Angkititrakul, C. Miyajima, and K. Takeda. 2008. Multi-modal real-world driving data collection, transcription, and integration using Bayesian network. In Proceedings of the 2008 IEEE Intelligent Vehicles Symposium. 150--155. DOI:https://doi.org/10.1109/IVS.2008.4621141Google ScholarCross Ref
- Lucas Malta, Chiyomi Miyajima, Norihide Kitaoka, and Kazuya Takeda. 2011. Analysis of real-world driver’s frustration. IEEE Transactions on Intelligent Transportation Systems 12, 1 (2011), 109--118. DOI:https://doi.org/10.1109/TITS.2010.2070839Google ScholarDigital Library
- D. McNair, M. Lorr, and L. F. Droppleman. 1991. POMS: Profile of Mood States. Educational and Industrial Testing Service, San Diego, CA.Google Scholar
- Jolieke Mesken, Marjan P. Hagenzieker, Talib Rothengatter, and Dick de Waard. 2007. Frequency, determinants, and consequences of different drivers’ emotions: An on-the-road study using self-reports, (observed) behaviour, and physiology. Transportation Research Part F: Traffic Psychology and Behaviour 10, 6 (2007), 458--475. DOI:https://doi.org/10.1016/j.trf.2007.05.001Google ScholarCross Ref
- Tsuyoshi Moriyama. 2012. Face analysis of aggressive moods in automobile driving using mutual subspace method. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR’12). 2898--2901.Google Scholar
- Nermine Munla, Mohamad Khalil, Ahmad Shahin, and Azzam Mourad. 2015. Driver stress level detection using HRV analysis. In Proceedings of the 2015 International Conference on Advances in Biomedical Engineering (ICABME’15). 61--64. DOI:https://doi.org/10.1109/ICABME.2015.7323251.Google ScholarCross Ref
- Kevin P. Murphy. 2009. The Bayes Net Toolbox for Matlab. Retrieved May 12, 2020 from https://www.cs.utah.edu/∼tch/notes/matlab/bnt/docs/bnt_pre_sf.htmlGoogle Scholar
- Fatma Nasoz, Christine L. Lisetti, and Athanasios V. Vasilakos. 2010. Affectively intelligent and adaptive car interfaces. Information Sciences 180, 20 (2010), 3817--3836. DOI:https://doi.org/10.1016/j.ins.2010.06.034Google ScholarDigital Library
- Fatma Nasoz, Onur Ozyer, Christine L. Lisetti, and Neal Finkelstein. 2002. Multimodal affective driver interfaces for future cars. In Proceedings of the 10th ACM International Conference on Multimedia. 319--322. DOI:https://doi.org/10.1145/641007.641074Google ScholarDigital Library
- Clifford Nass, Ing-Marie Jonsson, Helen Harris, Ben Reaves, Jack Endo, Scott Brave, and Leila Takayama. 2005. Improving automotive safety by pairing driver emotion and car voice emotion. In Proceedings of CHI’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA’05). 1973. DOI:https://doi.org/10.1145/1056808.1057070Google ScholarDigital Library
- Khairun Nisa’Minhad, Sawal Hamid Md. Ali, Jonathan Ooi Shi Khai, and Siti Anom Ahmad. 2016. Human emotion classifications for automotive driver using skin conductance response signal. In Proceedings of the 2016 International Conference on Advances in Electrical, Electronic, and Systems Engineering (ICAEES’16). IEEE, Los Alamitos, CA, 371--375.Google ScholarCross Ref
- M. Oehl, F. W. Siebert, T.-K. Tews, R. Höger, and H.-R. Pfister. 2011. Improving human-machine interaction: A non invasive approach to detect emotions in car drivers. In Human-Computer Interaction: Towards Mobile and Intelligent Interaction Environments. Lecture Notes in Computer Science, Vol. 6763. Springer. 577--585. DOI:https://doi.org/10.1007/978-3-642-21616-9_65Google Scholar
- Jonathan Shi Khai Ooi and Siti Anom Ahmad. 2016. Driver emotion recognition framework based on electrodermal activity measurements during simulated driving conditions. In Proceedings of the Conference on Biomedical Engineering and Sciences. DOI:https://doi.org/10.1109/IECBES.2016.7843475Google ScholarCross Ref
- Pablo Enrique Paredes, Nur Al Huda Hamdan, Dav Clark, Carrie Cai, Wendy Ju, and James A. Landay. 2017. Evaluating in-car movements in the design of mindful commute interventions: Exploratory study. Journal of Medical Internet Research 19, 12 (2017), e372. DOI:https://doi.org/10.2196/jmir.6983Google ScholarCross Ref
- Pablo E. Paredes, Francisco Ordonez, Wendy Ju, and James A. Landay. 2018. Fast and furious: Detecting stress with a car steering wheel. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--12.Google Scholar
- Thomas D. Parsons and Christopher G. Courtney. 2016. Interactions between threat and executive control in a virtual reality Stroop task. IEEE Transactions on Affective Computing 9, 1 (2016), 66--75. DOI:https://doi.org/10.1109/TAFFC.2016.2569086Google ScholarDigital Library
- M. Paschero, G. Del Vescovo, L. Benucci, A. Rizzi, M. Santello, G. Fabbri, and F. M. Frattale Mascioli. 2012. A real time classifier for emotion and stress recognition in a vehicle driver. In Proceedings of the IEEE International Symposium on Industrial Electronics. 1690--1695. DOI:https://doi.org/10.1109/ISIE.2012.6237345Google Scholar
- Rosalind W. Picard. 1997. Affective Computing. MIT Press, Cambridge, MA.Google ScholarDigital Library
- Ming Zher Poh, Daniel J. McDuff, and Rosalind W. Picard. 2011. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering 58, 1 (2011), 7--11. DOI:https://doi.org/10.1109/TBME.2010.2086456Google ScholarCross Ref
- Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion 37 (2017), 98--125. DOI:https://doi.org/10.1016/j.inffus.2017.02.003Google ScholarDigital Library
- Hamidur Rahman, Shaibal Barua, and Begum Shahina. 2015. Intelligent driver monitoring based on physiological sensor signals: Application using camera. In Proceedings of the IEEE Conference on Intelligent Transportation Systems (ITSC’15). 2637--2642. DOI:https://doi.org/10.1109/ITSC.2015.424Google ScholarDigital Library
- Genaro Rebolledo-Mendez, Angelica Reyes, Sebastian Paszkowicz, Mari Carmen Domingo, and Lee Skrypchuk. 2014. Developing a body sensor network to detect emotions during driving. IEEE Transactions on Intelligent Transportation Systems 15, 4 (2014), 1850--1854. DOI:https://doi.org/10.1109/TITS.2014.2335151Google ScholarCross Ref
- Andreas Riener, Alois Ferscha, and Mohamed Aly. 2009. Heart on the road: HRV analysis for monitoring a driver’s affective state. In Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI’09). 99--106. DOI:https://doi.org/10.1145/1620509.1620529Google ScholarDigital Library
- George Rigas, Yorgos Goletsis, and Dimitrios I. Fotiadis. 2012. Real-time driver’s stress event detection. IEEE Transactions on Intelligent Transportation Systems 13, 1 (2012), 221--234. DOI:https://doi.org/10.1109/TITS.2011.2168215Google ScholarDigital Library
- Ognjen Rudovic, Jaeryoung Lee, Miles Dai, Björn Schuller, and Rosalind W. Picard. 2018. Personalized machine learning for robot perception of affect and engagement in autism therapy. Science Robotics 3, 19 (2018), eaao6760.Google Scholar
- Daniele Ruscio, Luca Bascetta, Alessandro Gabrielli, Matteo Matteucci, and Lorenzo Mussone. 2017. Collection and comparison of driver/passenger physiologic and behavioural data in simulation and on-road driving. In Proceedings of the IEEE International Conference on Models and Technologies for Intelligent Transportation Systems. 403--408.Google ScholarCross Ref
- James A. Russell. 1980. A circumplex model of affect. Personality and Social Psychology 39 (1980), 1161--1178.Google ScholarCross Ref
- Aaqib Saeed and Stojan Trajanovski. 2017. Personalized driver stress detection with multi-task neural networks using physiological signals. In Proceedings of the Conference on Neural Information Processing Systems. http://arxiv.org/abs/1711.06116Google Scholar
- Arun Sahayadhas, Kenneth Sundaraj, and Murugappan Murugappan. 2012. Detecting driver drowsiness based on sensors: A review. Sensors (Switzerland) 12, 12 (2012), 16937--16953. DOI:https://doi.org/10.3390/s121216937Google ScholarCross Ref
- Bjorn Schuller, Felix Friedmann, and Florian Eyben. 2013. Automatic recognition of physiological parameters in the human voice: Heart rate and skin conductance. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’13). 7219--7223. DOI:https://doi.org/10.1109/ICASSP.2013.6639064Google ScholarCross Ref
- Björn Schuller, Manfred Lang, and Gerhard Rigoll. 2006. Recognition of spontaneous emotions by speech within automotive environment. Tagungsband Fortschritte der Akustik (DAGA’06). 57--58. http://www.mmk.ei.tum.de/publ/pdf/06/06sch5.pdf.Google Scholar
- Björn Schuller, Gerhard Rigoll, and Manfred Lang. 2004. Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture. Acoustics, Speech, and Signal Processing 1 (2004), 577--580. DOI:https://doi.org/10.1109/ICASSP.2004.1326051Google Scholar
- Bjoern Schuller, Matthias Wimmer, Dejan Arsic, Tobias Moosmayr, and Gerhard Rigoll. 2008. Detection of security related affect and behaviour in passenger transport. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH’08). 265--268.Google ScholarCross Ref
- Bjoern W. Schuller. 2008. Speaker, noise, and acoustic space adaptation for emotion recognition in the automotive environment. In Proceedings of the ITG Conference on Voice Communication (SprachKommunikation’08). 1--4. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5759973Google Scholar
- Liping Shen, Minjuan Wang, and Ruimin Shen. 2009. Affective e-learning: Using emotional data to improve learning in pervasive learning environment related work and the pervasive e-learning platform. Educational Technology 8 Society 12 (2009), 176--189. DOI:https://doi.org/citeulike-article-id:7412147Google Scholar
- Saul Shiffman, Arthur A. Stone, and Michael R. Hufford. 2008. Ecological momentary assessment. Annual Review of Clinical Psychology 4, 1 (2008), 1--32. DOI:https://doi.org/10.1146/annurev.clinpsy.3.022806.091415Google ScholarCross Ref
- Felix W. Siebert, Michael Oehl, and H.-R. Pfister. 2010. The measurement of grip-strength in automobiles: A new approach to detect driver’s emotions. In Advances in Human Factors, Ergonomics, and Safety in Manufacturing and Service Industry, W. Karwowski and G. Salvendy (Eds.). CRC Press, Boca Raton, FL, 775--782.Google Scholar
- Mohamad Hoseyn Sigari, Mahmood Fathy, and Mohsen Soryani. 2013. A driver face monitoring system for fatigue and distraction detection. International Journal of Vehicular Technology 2013 (2013), 73--100. DOI:https://doi.org/10.1155/2013/263983 arxiv:263983Google ScholarCross Ref
- Rajiv Ranjan Singh and Rahul Banerjee. 2010. Multi-parametric analysis of sensory data collected from automotive drivers for building a safety-critical wearable computing system. In Proceedings of the 2010 International Conference on Computer Engineering and Technology (ICCET’10), Vol. 1.355--360. DOI:https://doi.org/10.1109/ICCET.2010.5486110Google ScholarCross Ref
- Rajiv Ranjan Singh, Sailesh Conjeti, and Rahul Banerjee. 2011. An approach for real-time stress-trend detection using physiological signals in wearable computing systems for automotive drivers. In Proceedings of the 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC’11). 1477--1482. DOI:https://doi.org/10.1109/ITSC.2011.6082900Google ScholarCross Ref
- Rajiv Ranjan Singh, Sailesh Conjeti, and Rahul Banerjee. 2012. Biosignal based on-road stress monitoring for automotive drivers. In Proceedings of the 2012 National Conference on Communications (NCC’12). 8--9. DOI:https://doi.org/10.1109/NCC.2012.6176845Google ScholarCross Ref
- Rajiv Ranjan Singh, Sailesh Conjeti, and Rahul Banerjee. 2013. A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals. Biomedical Signal Processing and Control 8, 6 (2013), 740--754. DOI:https://doi.org/10.1016/j.bspc.2013.06.014Google ScholarCross Ref
- Kare Sjölander. 2004. The snack sound toolkit. http://www.speech.kth.se/snack.Google Scholar
- Kåre Sjölander and Jonas Beskow. 2000. Wavesurfer—An open source speech tool. Interspeech 4 (2000), 464--467.Google Scholar
- SourceForge. 2012. The Open Racing Car Simulator (TORCS). Retrieved May 12, 2020 from http://torcs.sourceforge.net/.Google Scholar
- SourceForge. 2019. The BioSig Project. Retrieved May 12, 2020 from http://biosig.sourceforge.net/.Google Scholar
- Olga Sourina, Yisi Liu, Qiang Wang, and Minh Khoa Nguyen. 2011. EEG-based personalized digital experience. In Universal Access in Human-Computer Interaction: Users Diversity. Lecture Notes in Computer Science, Vol. 6766. Springer, 591--599. DOI:https://doi.org/10.1007/978-3-642-21663-3_64Google Scholar
- Ronnie Taib, Jeremy Tederry, and Benjamin Itzstein. 2014. Quantifying driver frustration to improve road safety. In Proceedings of the Extended Abstracts of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI EA’14). 1777--1782. DOI:https://doi.org/10.1145/2559206.2581258Google ScholarDigital Library
- Ashish Tawari and Mohan Trivedi. 2010. Speech emotion analysis in noisy real-world environment. In Proceedings of the International Conference on Pattern Recognition. DOI:https://doi.org/10.1109/ICPR.2010.1132Google ScholarDigital Library
- Ashish Tawari and Mohan M. Trivedi. 2010. Speech based emotion classification framework for driver assistance system. In Proceedings of the IEEE Intelligent Vehicles Symposium. 174--178. DOI:https://doi.org/10.1109/IVS.2010.5547956Google Scholar
- Ashish Tawari and Mohan Manubhai Trivedi. 2010. Speech emotion analysis: Exploring the role of context. IEEE Transactions on Multimedia 12, 6 (2010), 502--509. DOI:https://doi.org/10.1109/TMM.2010.2058095Google ScholarDigital Library
- OpenCV Team. 2019. Open Source Computer Vision Library. Retrieved May 12, 2020 from https://opencv.org/.Google Scholar
- Masaharu Terasaki, Youlchl Klshimoto, and Alto Koga. 1992. Construction of a multiple mood scale. Japanese Journal of Psychology 62, 6 (1992), 350--356. DOI:https://doi.org/10.4992/jjpsy.62.350Google ScholarCross Ref
- Tessa Karina Tews, Michael Oehl, Felix W. Siebert, Rainer Höger, and Helmut Faasch. 2011. Emotional human-machine interaction: Cues from facial expressions. In Human Interface and the Management of Information: Interacting with Information. Lecture Notes in Computer Science, Vol. 6771. Springer, 641--650. DOI:https://doi.org/10.1007/978-3-642-21793-7_73Google Scholar
- M. Tischler, C. Peter, M. Wimmer, and J. Voskamp. 2007. Application of emotion recognition methods in automotive research. In Proceedings of the Workshop on Emotion and Computing—Current Research and Future Impact. 50--55. http://ias.cs.tum.edu/_media/spezial/bib/tischler07application.pdf.Google Scholar
- Geoffrey Underwood, Peter Chapman, Sharon Wright, and David Crundall. 1999. Anger while driving. Transportation Research Part F: Traffic Psychology and Behaviour 2, 1 (1999), 55--68. DOI:https://doi.org/10.1016/S1369-8478(99)00006-6Google ScholarCross Ref
- Paul Viola and M. J. Jones. 2004. Robust real-time face detection. International Journal of Computer Vision 57, 2 (2004), 137--154. DOI:https://doi.org/10.1023/B:VISI.0000013087.49260.fb arxiv:arXiv:1011.1669v3Google ScholarDigital Library
- Jinjun Wang and Yihong Gong. 2008. Recognition of multiple drivers’ emotional state. In Proceedings of the 2008 19th International Conference on Pattern Recognition. 1--4. DOI:https://doi.org/10.1109/ICPR.2008.4761904Google ScholarCross Ref
- Jeen Shing Wang, Che Wei Lin, and Ya Ting C. Yang. 2013. A k-nearest-neighbor classifier with heart rate variability feature-based transformation algorithm for driving stress recognition. Neurocomputing 116 (2013), 136--143. DOI:https://doi.org/10.1016/j.neucom.2011.10.047Google ScholarCross Ref
- D. Watson, L. A. Clark, and A. Tellegen. 1988. Development and validation of brief measures of positive and negative affect: The PANAS scales.Journal of Personality and Social Psychology 54, 6 (1988), 1063--1070.Google Scholar
- Frank H. Wilhelm and Paul Grossman. 2010. Emotions beyond the laboratory: Theoretical fundaments, study design, and analytic strategies for advanced ambulatory assessment. Biological Psychology 84, 3 (2010), 552--569. DOI:https://doi.org/10.1016/j.biopsycho.2010.01.017Google ScholarCross Ref
- Kenton Williams, José Acevedo Flores, and Joshua Peters. 2014. Affective robot influence on driver adherence to safety, cognitive load reduction and sociability. In Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI’14). 1--8. DOI:https://doi.org/10.1145/2667317.2667342Google ScholarDigital Library
- Mingmin Zhao, Fadel Adib, and Dina Katabi. 2016. Emotion recognition using wireless signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, New York, NY, 95--108.Google ScholarDigital Library
Index Terms
- Driver Emotion Recognition for Intelligent Vehicles: A Survey
Recommendations
Towards Empathetic Car Interfaces: Emotional Triggers while Driving
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing SystemsMonitoring the emotions of drivers can play a critical role to reduce road accidents and enable novel driver-car interactions. To help understand the possibilities, this work systematically studies the in-road triggers that may lead to different ...
Improving automotive safety by pairing driver emotion and car voice emotion
CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing SystemsThis study examines whether characteristics of a car voice can affect driver performance and affect. In a 2 (driver emotion: happy or upset) x 2 (car voice emotion: energetic vs. subdued) experimental study, participants (N=40) had emotion induced ...
Emotion Recognition Using Physiological Signals
MIDI '15: Proceedings of the Mulitimedia, Interaction, Design and InnnovationIn this paper the problem of emotion recognition using physiological signals is presented. Firstly the problems with acquisition of physiological signals related to specific human emotions are described. It is not a trivial problem to elicit real ...
Comments