Abstract
Personality analysis is an important area of research in several fields, including psychology, psychiatry, and neuroscience. With the recent dramatic improvements in machine learning, it has also become a popular research area in computer science. While the current computational methods are able to interpret behavioral cues (e.g., facial expressions, gesture, and voice) to estimate the level of (apparent) personality traits, accessible assessment tools are still substandard for practical use, not to mention the need for fast and accurate methods for such analyses. In this study, we present multimodal deep architectures to estimate the Big Five personality traits from (temporal) audio-visual cues and transcribed speech. Furthermore, for a detailed analysis of personality traits, we have collected a new audio-visual dataset, namely: Self-presentation and Induced Behavior Archive for Personality Analysis (SIAP). In contrast to the available datasets, SIAP introduces recordings of induced behavior in addition to self-presentation (speech) videos. With thorough experiments on SIAP and ChaLearn LAP First Impressions datasets, we systematically assess the reliability of different behavioral modalities and their combined use. Furthermore, we investigate the characteristics and discriminative power of induced behavior for personality analysis, showing that the induced behavior indeed includes signs of personality traits.
Similar content being viewed by others
Data Availability Statement
The collected SIAP dataset will be made available for research purposes upon requests with a signed EULA.
References
Abadi MK, Correa JAM, Wache J, Yang H, Patras I, Sebe N (2015) Inference of personality traits and affect schedule by analysis of spontaneous reactions to affective videos. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), Ljubljana, pp 1–8. https://doi.org/10.1109/FG.2015.7163100
Abele AE, Wojciszke B (2007) Agency and communion from the perspective of self versus others. J Pers Soc Psychol 93(5):751
Alam F, Riccardi G (2014) Predicting personality traits using multimodal information. In: Proceedings of the 2014 ACM multi media on workshop on computational personality recognition, pp 15–18
Alam F, Stepanov EA, Riccardi G (2013) Personality traits recognition on social network-facebook. In: International AAAI conference on weblogs and social media
Almaev TR, Valstar MF (2013) Local Gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: Humaine association conference on affective computing and intelligent interaction, pp 356–361. IEEE
Aran O, Gatica-Perez D (2013) Cross-domain personality prediction: from video blogs to small group meetings. In: Proceedings of the 15th ACM on international conference on multimodal interaction, pp 127–130
Ashton MC, Lee K (2007) Empirical, theoretical, and practical advantages of the Hexaco model of personality structure. Personality Soc Psychol Rev 11(2):150–166
Baltrusaitis T, Zadeh A, Lim YC, Morency LP (2018) Openface 2.0: facial behavior analysis toolkit. In: Proceedings of the 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp 59–66. IEEE
Bartko JJ (1966) The intraclass correlation coefficient as a measure of reliability. Psychol Rep 19(1):3–11
Biel JI, Aran O, Gatica-Perez D (2011) You are known by how you vlog: personality impressions and nonverbal behavior in youtube. In: Fifth international AAAI conference on weblogs and social media
Biel JI, Gatica-Perez D (2012) The youtube lens: crowdsourced personality impressions and audiovisual analysis of vlogs. IEEE Trans Multimedia 15(1):41–55
Boyette LL, Korver-Nieberg N, Meijer C, de Haan L (2014) Genetic risk and outcome of psychosis investigators: quality of life in patients with psychotic disorders: impact of symptoms, personality, and attachment. J Nerv Ment Dis 202(1):64–69
Boyette LL, Korver-Nieberg N, Verweij K, Meijer C, Dingemans P, Cahn W, de Haan L, Kahn R, de Haan L, van Os J, Wiersma D, Bruggeman R, Cahn W, Meijer C, Myin-Germeys I (2013) Associations between the five-factor model personality traits and psychotic experiences in patients with psychotic disorders, their siblings and controls. Psychiatry Res 210(2):491–497
Bromet E, Andrade LH, Hwang I, Sampson NA, Alonso J, De Girolamo G, De Graaf R, Demyttenaere K, Hu C, Iwata N, Karam A, Kaur J, Kostyuchenko S, Lépine JP, Levinson D, Matschinger H, Mora M, Browne M, Posada-Villa J, Viana M, Williams D, Kessler R (2011) Cross-national epidemiology of DSM-IV major depressive episode. BMC Med 9(1):90
Butcher JN, Graham JR, Williams CL, Ben-Porath YS (1990) Development and use of the MMPI-2 content scales. University of Minnesota Press, Minneapolis
Cao Z, Simon T, Wei SE, Sheikh Y (2017) Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7291–7299
Celiktutan O, Gunes H (2014) Continuous prediction of perceived traits and social dimensions in space and time. In: 2014 IEEE international conference on image processing (ICIP), pp 4196–4200. IEEE
Celiktutan O, Gunes H (2015) Automatic prediction of impressions in time and across varying context: personality, attractiveness and likeability. IEEE Trans Affect Comput 8(1):29–42
Chandrashekar P (2018) Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. Mhealth 4:6
Compton MT, Bakeman R, Alolayan Y, Balducci PM, Bernardini F, Broussard B, Crisafio A, Cristofaro S, Amar P, Johnson S, Wan CR (2015) Personality domains, duration of untreated psychosis, functioning, and symptom severity in first-episode psychosis. Schizophr Res 168(1–2):113–119
Corr PJ, Matthews G (2009) The Cambridge handbook of personality psychology. Cambridge University Press, New York
Costa PT Jr, Terracciano A, McCrae RR (2001) Gender differences in personality traits across cultures: robust and surprising findings. J Pers Soc Psychol 81(2):322
Cucurull G, Rodríguez P, Yazici VO, Gonfaus JM, Roca FX, Gonzàlez J (2018) Deep inference of personality traits by integrating image and word use in social networks. arXiv preprint arXiv:1802.06757
Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
Eddine Bekhouche S, Dornaika F, Ouafi A, Taleb-Ahmed A (2017) Personality traits and job candidate screening via analyzing facial videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 10–13
Escalante HJ, Kaya H, Salah AA, Escalera S, Güç Y, Güçlü U, Baró X, Guyon I, Jacques JCS, Madadi M, Ayache S, Viegas E, Gurpinar F, Wicaksana AS, Liem C, Van Gerven MA, Van Lier R (2020) Modeling, recognizing, and explaining apparent personality from videos. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.2973984
Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp 445–450. ACM
Farnadi G, Sushmita S, Sitaraman G, Ton N, De Cock M, Davalos S (2014) A multivariate regression approach to personality impression recognition of vloggers. In: Proceedings of the ACM multimedia workshop on computational personality recognition, pp 1–6
Feingold A (1994) Gender differences in personality: a meta-analysis. Psychol Bull 116(3):429
Google cloud speech-to-text. https://cloud.google.com/speech-to-text. Accessed 2019-09-15
Giannakopoulos T (2015) pyaudioanalysis: An open-source python library for audio signal analysis. PloS ONE 10(12):e0144610
Goldberg LR (1990) An alternative“ description of personality”: the big-five factor structure. J Pers Soc Psychol 59(6):1216
Gosling SD, Rentfrow PJ, Swann WB Jr (2003) A very brief measure of the big-five personality domains. J Res Pers 37(6):504–528
Graziano WG, Eisenberg N (1997) Agreeableness: a dimension of personality. In: Handbook of personality psychology, pp 795–824. Elsevier
Güçlütürk Y, Güçlü U, Baro X, Escalante HJ, Guyon I, Escalera S, Van Gerven MA, Van Lier R (2017) Multimodal first impression analysis with deep residual networks. IEEE Trans Affect Comput 9(3):316–329
Güçlütürk Y, Güçlü U, van Gerven MA, van Lier R (2016) Deep impression: audiovisual deep residual networks for multimodal apparent personality trait recognition. In: European conference on computer vision, pp 349–358. Springer
Guntuku SC, Qiu L, Roy S, Lin W, Jakhetiya V (2015) Do others perceive you as you want them to? modeling personality based on selfies. In: Proceedings of the 1st international workshop on affect & sentiment in multimedia, pp 21–26
Gürpınar F, Kaya H, Salah AA (2016) Combining deep facial and ambient features for first impression estimation. In: European Conference on Computer Vision, pp. 372–385. Springer
Gürpinar F, Kaya H, Salah AA (2016) Multimodal fusion of audio, scene, and face features for first impression estimation. In: Proceedings of the international conference on pattern recognition (ICPR), pp 43–48. IEEE
Hara K, Kataoka H, Satoh Y (2018) Can spatiotemporal 3D CNNS retrace the history of 2D CNNS and imagenet? In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 6546–6555
Hasan A (2013) The Turkish adaptation of the ten-item personality inventory. Nöro Psikiyatri Arşivi 50(4):312
Hassan S, Akhtar N, Yılmaz AK (2016) Impact of the conscientiousness as personality trait on both job and organizational performance. J Manag Sci 10(1):1–14
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778. Retrieved from https://repository.upenn.edu/asc_papers/43. Accessed 6 Oct 2020
Huggingface: huggingface/pytorch-transformers (2019). https://github.com/huggingface/pytorch-transformers. Accessed 29 July 2019
Husain MI, Carvalho AF (2020) The importance of assessing personality traits and disorders in clinical trials of major depressive disorder. Braz J Psychiatry 42(1):3–4
Kay W, Carreira J, Simonyan K, Zhang B, Hillier C, Vijayanarasimhan S, Viola F, Green T, Back T, Natsev P, Suleyman M, Zisserman A (2017) The kinetics human action video dataset. arXiv preprint arXiv:1705.06950
Kaya H, Gurpinar F, Ali Salah A (2017) Multi-modal score fusion and decision trees for explainable automatic job candidate screening from video CVS. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 1–9
Keltner D, Ekman P, Gonzaga GC, Beer J (2003) Facial expressions of emotion. In: Davidson RJ, Scherer KR, Goldsmith HH (eds) Handbook of affective sciences. Oxford University Press, New York, pp 415–432
Khan AA, Jacobson KC, Gardner CO, Prescott CA, Kendler KS (2005) Personality and comorbidity of common psychiatric disorders. Br J Psychiatry 186(3):190–196
Kim B, Joo YH, Kim SY, Lim JH, Kim EO (2011) Personality traits and affective morbidity in patients with bipolar I disorder: the five-factor model perspective. Psychiatry Res 185(1–2):135–140
Kim B, Lim JH, Kim SY, Joo YH (2012) Comparative study of personality traits in patients with bipolar I and II disorder from the five-factor model perspective. Psychiatry Investig 9(4):347
Kotov R, Gamez W, Schmidt F, Watson D (2010) Linking “big” personality traits to anxiety, depressive, and substance use disorders: a meta-analysis. Psychol Bull 136(5):768
Krippendorff, K (2011). Computing Krippendorff’s alpha-reliability. Departmental Papers (ASC) https://repository.upenn.edu/asc_papers/43. Accessed 6 Oct 2020
Lai G, Chang WC, Yang Y, Liu H (2018) Modeling long-and short-term temporal patterns with deep neural networks. In: Proceedings of the 41st international ACM SIGIR conference on research & development in information retrieval, pp 95–104. ACM
Lai S, Xu L, Liu K, Zhao J (2015) Recurrent convolutional neural networks for text classification. In: Proceedings of the 29th AAAI conference on artificial intelligence
Lecomte T, Spidel A, Leclerc C, MacEwan GW, Greaves C, Bentall RP (2008) Predictors and profiles of treatment non-adherence and engagement in services problems in early psychosis. Schizophr Res 102(1–3):295–302
Lysaker PH, Davis LW (2004) Social function in schizophrenia and schizoaffective disorder: associations with personality, symptoms and neurocognition. Health Qual Life Outcomes 2(1):15
van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9:2579–2605
McCrae RR, John OP (1992) An introduction to the five-factor model and its applications. J Pers 60(2):175–215
Mulder RT (2002) Personality pathology and treatment outcome in major depression: a review. Am J Psychiatry 159(3):359–371
Odić A, Tkalčič M, Tasič J, Košir A (2013) Personality and social context: impact on emotion induction from movies. In: Extended proceedings of the conference on user modeling, adaptation, and personalization
Ohi K, Shimada T, Nitta Y, Kihara H, Okubo H, Uehara T, Kawasaki Y (2016) The five-factor model personality traits in schizophrenia: a meta-analysis. Psychiatry Res 240:34–41
Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: British machine vision conference. British Machine Vision Association
Pedregon CA, Farley RL, Davis A, Wood JM, Clark RD (2012) Social desirability, personality questionnaires, and the “better than average” effect. Personality Individ Differ 52(2):213–217
Ponce-López V, Chen B, Oliu M, Corneanu C, Clapés A, Guyon I, Baró X, Escalante HJ, Escalera S (2016) Chalearn lap 2016: first round challenge on first impressions-dataset and results. In: Proceedings of the European conference on computer vision, pp 400–418. Springer
Qiu L, Lu J, Yang S, Qu W, Zhu T (2015) What does your selfie say about you? Comput Hum Behav 52:443–449
Quilty LC, De Fruyt F, Rolland JP, Kennedy SH, Rouillon PF, Bagby RM (2008) Dimensional personality traits and treatment outcome in patients with major depressive disorder. J Affect Disord 108(3):241–250
Riggio HR, Riggio RE (2002) Emotional expressiveness, extraversion, and neuroticism: a meta-analysis. J Nonverbal Behav 26(4):195–218
Salem H, Ruiz A, Hernandez S, Wahid K, Cao F, Karnes B, Beasley S, Sanches M, Ashtari E, Pigott T (2019) Borderline personality features in inpatients with bipolar disorder: impact on course and machine learning model use to predict rapid readmission. J Psychiatr Pract 25(4):279–289
Segalin C, Cheng DS, Cristani M (2017) Social profiling through image understanding: personality inference using convolutional neural networks. Comput Vis Image Underst 156:34–50
Selçuk E, Günaydin G, Sümer N, Uysal A (2005) Yetişkin bağlanma boyutlan için yeni bir ölçüm: Yakın ilişkilerde yaşantılar envanteri-II’nin Türk örnekleminde psikometrik açıdan değerlendirilmesi. Türk Psikoloji Yazıları
Silveira Jacques Junior JC, Güçlütürk Y, Pérez M, Güçlü U, Andujar C, Baró X, Escalante HJ, Guyon I, Van Gerven MA, Van Lier R, Escalera S (2019) First impressions: a survey on vision-based apparent personality trait analysis. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2019.2930058
Smeland OB, Wang Y, Lo MT, Li W, Frei O, Witoelar A, Tesli M, Hinds DA, Tung JY, Djurovic S, Chen CH, Dale AM, Andreassen OA (2017) Identification of genetic loci shared between schizophrenia and the big five personality traits. Sci Rep 7(1):1–9
Sparkman DJ, Eidelman S, Dueweke AR, Marin MS, Dominguez B (2019) Open to diversity: openness to experience predicts beliefs in multiculturalism and colorblindness through perspective taking. J Individ Diff 40(1):1–12. https://doi.org/10.1027/1614-0001/a000270
Spearman C (1961) The proof and measurement of association between two things. In: Jenkins JJ, Paterson DG (eds) Studies in individual differences: the search for intelligence. Appleton-Century-Crofts, pp 45–58. https://doi.org/10.1037/11491-005
Subramaniam A, Patel V, Mishra A, Balasubramanian P, Mittal A (2016) Bi-modal first impressions recognition using temporally ordered deep audio and stochastic visual features. In: European conference on computer vision, pp 337–348. Springer
Valente F, Kim S, Motlicek P (2012) Annotation and recognition of personality traits in spoken conversations from the ami meetings corpus. In: Annual conference of the international speech communication association
Ventura C, Masip D, Lapedriza A (2017) Interpreting CNN models for apparent personality trait regression. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 55–63
Vielzeuf V, Lechervy A, Pateux S, Jurie F (2018) Centralnet: a multilayer approach for multimodal fusion. In: Proceedings of the European conference on computer vision workshops, pp 575–589
Vinciarelli A, Mohammadi G (2014) A survey of personality computing. IEEE Trans Affect Comput 5(3):273–291
Wang J, Yang Y, Mao J, Huang Z, Huang C, Xu W (2016) CNN-RNN: a unified framework for multi-label image classification. In: The IEEE conference on computer vision and pattern recognition (CVPR)
Wauthia E, Lefebvre L, Huet K, Blekic W, El Bouragui K, Rossignol M (2019) Examining the hierarchical influences of the big-five dimensions and anxiety sensitivity on anxiety symptoms in children. Front Psychol 10:1185
Wei XS, Zhang CL, Zhang H, Wu J (2017) Deep bimodal regression of apparent personality traits from short video sequences. IEEE Trans Affect Comput 9(3):303–315
Weisberg YJ, DeYoung CG, Hirsh JB (2011) Gender differences in personality across the ten aspects of the big five. Front Psychol 2:178
Wicaksana AS, Liem CC (2017) Human-explainable features for job candidate screening prediction. In: IEEE Conference on computer vision and pattern recognition workshops (CVPRW), pp 1664–1669. IEEE
Willis J, Todorov A (2006) First impressions: Making up your mind after a 100-ms exposure to a face. Psychol Sci 17(7):592–598
Wu X, He H, Shi L, Xia Y, Zuang K, Feng Q, Zhang Y, Ren Z, Wei D, Qiu J (2019) Personality traits are related with dynamic functional connectivity in major depression disorder: a resting-state analysis. J Affect Disord 245:1032–1042
Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1492–1500
Yan Y, Nie J, Huang L, Li Z, Cao Q, Wei Z (2016) Exploring relationship between face and trustworthy impression using mid-level facial features. In: International conference on multimedia modeling, pp 540–549. Springer
Yik MS, Russell JA (2001) Predicting the big two of affect from the big five of personality. J Res Pers 35(3):247–277
Zillig LMP, Hemenover SH, Dienstbier RA (2002) What do we assess when we assess a big 5 trait? a content analysis of the affective, behavioral, and cognitive processes represented in big 5 personality inventories. Pers Soc Psychol Bull 28(6):847–858
Acknowledgements
Authors would like to thank Mr. Alper Dağgez, Ms. Merve Nur Doğanlı and Prof. Gül Günaydin for their contributions to this study.
Funding
Not applicable.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflict of interest.
Ethical approval
The current study received the required ethical approval from the Ethics Committee of the university. Each participant of the experiment was informed about the procedure, and signed a consent form.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Giritlioğlu, D., Mandira, B., Yilmaz, S.F. et al. Multimodal analysis of personality traits on videos of self-presentation and induced behavior. J Multimodal User Interfaces 15, 337–358 (2021). https://doi.org/10.1007/s12193-020-00347-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-020-00347-7