Skip to main content
Log in

Multimodal analysis of personality traits on videos of self-presentation and induced behavior

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Personality analysis is an important area of research in several fields, including psychology, psychiatry, and neuroscience. With the recent dramatic improvements in machine learning, it has also become a popular research area in computer science. While the current computational methods are able to interpret behavioral cues (e.g., facial expressions, gesture, and voice) to estimate the level of (apparent) personality traits, accessible assessment tools are still substandard for practical use, not to mention the need for fast and accurate methods for such analyses. In this study, we present multimodal deep architectures to estimate the Big Five personality traits from (temporal) audio-visual cues and transcribed speech. Furthermore, for a detailed analysis of personality traits, we have collected a new audio-visual dataset, namely: Self-presentation and Induced Behavior Archive for Personality Analysis (SIAP). In contrast to the available datasets, SIAP introduces recordings of induced behavior in addition to self-presentation (speech) videos. With thorough experiments on SIAP and ChaLearn LAP First Impressions datasets, we systematically assess the reliability of different behavioral modalities and their combined use. Furthermore, we investigate the characteristics and discriminative power of induced behavior for personality analysis, showing that the induced behavior indeed includes signs of personality traits.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability Statement

The collected SIAP dataset will be made available for research purposes upon requests with a signed EULA.

References

  1. Abadi MK, Correa JAM, Wache J, Yang H, Patras I, Sebe N (2015) Inference of personality traits and affect schedule by analysis of spontaneous reactions to affective videos. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), Ljubljana, pp 1–8. https://doi.org/10.1109/FG.2015.7163100

  2. Abele AE, Wojciszke B (2007) Agency and communion from the perspective of self versus others. J Pers Soc Psychol 93(5):751

    Article  Google Scholar 

  3. Alam F, Riccardi G (2014) Predicting personality traits using multimodal information. In: Proceedings of the 2014 ACM multi media on workshop on computational personality recognition, pp 15–18

  4. Alam F, Stepanov EA, Riccardi G (2013) Personality traits recognition on social network-facebook. In: International AAAI conference on weblogs and social media

  5. Almaev TR, Valstar MF (2013) Local Gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: Humaine association conference on affective computing and intelligent interaction, pp 356–361. IEEE

  6. Aran O, Gatica-Perez D (2013) Cross-domain personality prediction: from video blogs to small group meetings. In: Proceedings of the 15th ACM on international conference on multimodal interaction, pp 127–130

  7. Ashton MC, Lee K (2007) Empirical, theoretical, and practical advantages of the Hexaco model of personality structure. Personality Soc Psychol Rev 11(2):150–166

    Article  Google Scholar 

  8. Baltrusaitis T, Zadeh A, Lim YC, Morency LP (2018) Openface 2.0: facial behavior analysis toolkit. In: Proceedings of the 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp 59–66. IEEE

  9. Bartko JJ (1966) The intraclass correlation coefficient as a measure of reliability. Psychol Rep 19(1):3–11

    Article  Google Scholar 

  10. Biel JI, Aran O, Gatica-Perez D (2011) You are known by how you vlog: personality impressions and nonverbal behavior in youtube. In: Fifth international AAAI conference on weblogs and social media

  11. Biel JI, Gatica-Perez D (2012) The youtube lens: crowdsourced personality impressions and audiovisual analysis of vlogs. IEEE Trans Multimedia 15(1):41–55

    Article  Google Scholar 

  12. Boyette LL, Korver-Nieberg N, Meijer C, de Haan L (2014) Genetic risk and outcome of psychosis investigators: quality of life in patients with psychotic disorders: impact of symptoms, personality, and attachment. J Nerv Ment Dis 202(1):64–69

    Article  Google Scholar 

  13. Boyette LL, Korver-Nieberg N, Verweij K, Meijer C, Dingemans P, Cahn W, de Haan L, Kahn R, de Haan L, van Os J, Wiersma D, Bruggeman R, Cahn W, Meijer C, Myin-Germeys I (2013) Associations between the five-factor model personality traits and psychotic experiences in patients with psychotic disorders, their siblings and controls. Psychiatry Res 210(2):491–497

    Article  Google Scholar 

  14. Bromet E, Andrade LH, Hwang I, Sampson NA, Alonso J, De Girolamo G, De Graaf R, Demyttenaere K, Hu C, Iwata N, Karam A, Kaur J, Kostyuchenko S, Lépine JP, Levinson D, Matschinger H, Mora M, Browne M, Posada-Villa J, Viana M, Williams D, Kessler R (2011) Cross-national epidemiology of DSM-IV major depressive episode. BMC Med 9(1):90

    Article  Google Scholar 

  15. Butcher JN, Graham JR, Williams CL, Ben-Porath YS (1990) Development and use of the MMPI-2 content scales. University of Minnesota Press, Minneapolis

    Google Scholar 

  16. Cao Z, Simon T, Wei SE, Sheikh Y (2017) Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7291–7299

  17. Celiktutan O, Gunes H (2014) Continuous prediction of perceived traits and social dimensions in space and time. In: 2014 IEEE international conference on image processing (ICIP), pp 4196–4200. IEEE

  18. Celiktutan O, Gunes H (2015) Automatic prediction of impressions in time and across varying context: personality, attractiveness and likeability. IEEE Trans Affect Comput 8(1):29–42

    Article  Google Scholar 

  19. Chandrashekar P (2018) Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. Mhealth 4:6

    Article  Google Scholar 

  20. Compton MT, Bakeman R, Alolayan Y, Balducci PM, Bernardini F, Broussard B, Crisafio A, Cristofaro S, Amar P, Johnson S, Wan CR (2015) Personality domains, duration of untreated psychosis, functioning, and symptom severity in first-episode psychosis. Schizophr Res 168(1–2):113–119

    Article  Google Scholar 

  21. Corr PJ, Matthews G (2009) The Cambridge handbook of personality psychology. Cambridge University Press, New York

    Book  Google Scholar 

  22. Costa PT Jr, Terracciano A, McCrae RR (2001) Gender differences in personality traits across cultures: robust and surprising findings. J Pers Soc Psychol 81(2):322

    Article  Google Scholar 

  23. Cucurull G, Rodríguez P, Yazici VO, Gonfaus JM, Roca FX, Gonzàlez J (2018) Deep inference of personality traits by integrating image and word use in social networks. arXiv preprint arXiv:1802.06757

  24. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  25. Eddine Bekhouche S, Dornaika F, Ouafi A, Taleb-Ahmed A (2017) Personality traits and job candidate screening via analyzing facial videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 10–13

  26. Escalante HJ, Kaya H, Salah AA, Escalera S, Güç Y, Güçlü U, Baró X, Guyon I, Jacques JCS, Madadi M, Ayache S, Viegas E, Gurpinar F, Wicaksana AS, Liem C, Van Gerven MA, Van Lier R (2020) Modeling, recognizing, and explaining apparent personality from videos. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.2973984

    Article  Google Scholar 

  27. Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp 445–450. ACM

  28. Farnadi G, Sushmita S, Sitaraman G, Ton N, De Cock M, Davalos S (2014) A multivariate regression approach to personality impression recognition of vloggers. In: Proceedings of the ACM multimedia workshop on computational personality recognition, pp 1–6

  29. Feingold A (1994) Gender differences in personality: a meta-analysis. Psychol Bull 116(3):429

    Article  Google Scholar 

  30. Google cloud speech-to-text. https://cloud.google.com/speech-to-text. Accessed 2019-09-15

  31. Giannakopoulos T (2015) pyaudioanalysis: An open-source python library for audio signal analysis. PloS ONE 10(12):e0144610

    Article  Google Scholar 

  32. Goldberg LR (1990) An alternative“ description of personality”: the big-five factor structure. J Pers Soc Psychol 59(6):1216

    Article  Google Scholar 

  33. Gosling SD, Rentfrow PJ, Swann WB Jr (2003) A very brief measure of the big-five personality domains. J Res Pers 37(6):504–528

    Article  Google Scholar 

  34. Graziano WG, Eisenberg N (1997) Agreeableness: a dimension of personality. In: Handbook of personality psychology, pp 795–824. Elsevier

  35. Güçlütürk Y, Güçlü U, Baro X, Escalante HJ, Guyon I, Escalera S, Van Gerven MA, Van Lier R (2017) Multimodal first impression analysis with deep residual networks. IEEE Trans Affect Comput 9(3):316–329

    Article  Google Scholar 

  36. Güçlütürk Y, Güçlü U, van Gerven MA, van Lier R (2016) Deep impression: audiovisual deep residual networks for multimodal apparent personality trait recognition. In: European conference on computer vision, pp 349–358. Springer

  37. Guntuku SC, Qiu L, Roy S, Lin W, Jakhetiya V (2015) Do others perceive you as you want them to? modeling personality based on selfies. In: Proceedings of the 1st international workshop on affect & sentiment in multimedia, pp 21–26

  38. Gürpınar F, Kaya H, Salah AA (2016) Combining deep facial and ambient features for first impression estimation. In: European Conference on Computer Vision, pp. 372–385. Springer

  39. Gürpinar F, Kaya H, Salah AA (2016) Multimodal fusion of audio, scene, and face features for first impression estimation. In: Proceedings of the international conference on pattern recognition (ICPR), pp 43–48. IEEE

  40. Hara K, Kataoka H, Satoh Y (2018) Can spatiotemporal 3D CNNS retrace the history of 2D CNNS and imagenet? In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 6546–6555

  41. Hasan A (2013) The Turkish adaptation of the ten-item personality inventory. Nöro Psikiyatri Arşivi 50(4):312

    Article  Google Scholar 

  42. Hassan S, Akhtar N, Yılmaz AK (2016) Impact of the conscientiousness as personality trait on both job and organizational performance. J Manag Sci 10(1):1–14

    Google Scholar 

  43. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778. Retrieved from https://repository.upenn.edu/asc_papers/43. Accessed 6 Oct 2020

  44. Huggingface: huggingface/pytorch-transformers (2019). https://github.com/huggingface/pytorch-transformers. Accessed 29 July 2019

  45. Husain MI, Carvalho AF (2020) The importance of assessing personality traits and disorders in clinical trials of major depressive disorder. Braz J Psychiatry 42(1):3–4

    Article  Google Scholar 

  46. Kay W, Carreira J, Simonyan K, Zhang B, Hillier C, Vijayanarasimhan S, Viola F, Green T, Back T, Natsev P, Suleyman M, Zisserman A (2017) The kinetics human action video dataset. arXiv preprint arXiv:1705.06950

  47. Kaya H, Gurpinar F, Ali Salah A (2017) Multi-modal score fusion and decision trees for explainable automatic job candidate screening from video CVS. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 1–9

  48. Keltner D, Ekman P, Gonzaga GC, Beer J (2003) Facial expressions of emotion. In: Davidson RJ, Scherer KR, Goldsmith HH (eds) Handbook of affective sciences. Oxford University Press, New York, pp 415–432

    Google Scholar 

  49. Khan AA, Jacobson KC, Gardner CO, Prescott CA, Kendler KS (2005) Personality and comorbidity of common psychiatric disorders. Br J Psychiatry 186(3):190–196

    Article  Google Scholar 

  50. Kim B, Joo YH, Kim SY, Lim JH, Kim EO (2011) Personality traits and affective morbidity in patients with bipolar I disorder: the five-factor model perspective. Psychiatry Res 185(1–2):135–140

    Article  Google Scholar 

  51. Kim B, Lim JH, Kim SY, Joo YH (2012) Comparative study of personality traits in patients with bipolar I and II disorder from the five-factor model perspective. Psychiatry Investig 9(4):347

    Article  Google Scholar 

  52. Kotov R, Gamez W, Schmidt F, Watson D (2010) Linking “big” personality traits to anxiety, depressive, and substance use disorders: a meta-analysis. Psychol Bull 136(5):768

    Article  Google Scholar 

  53. Krippendorff, K (2011). Computing Krippendorff’s alpha-reliability. Departmental Papers (ASC) https://repository.upenn.edu/asc_papers/43. Accessed 6 Oct 2020

  54. Lai G, Chang WC, Yang Y, Liu H (2018) Modeling long-and short-term temporal patterns with deep neural networks. In: Proceedings of the 41st international ACM SIGIR conference on research & development in information retrieval, pp 95–104. ACM

  55. Lai S, Xu L, Liu K, Zhao J (2015) Recurrent convolutional neural networks for text classification. In: Proceedings of the 29th AAAI conference on artificial intelligence

  56. Lecomte T, Spidel A, Leclerc C, MacEwan GW, Greaves C, Bentall RP (2008) Predictors and profiles of treatment non-adherence and engagement in services problems in early psychosis. Schizophr Res 102(1–3):295–302

    Article  Google Scholar 

  57. Lysaker PH, Davis LW (2004) Social function in schizophrenia and schizoaffective disorder: associations with personality, symptoms and neurocognition. Health Qual Life Outcomes 2(1):15

    Article  Google Scholar 

  58. van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9:2579–2605

    MATH  Google Scholar 

  59. McCrae RR, John OP (1992) An introduction to the five-factor model and its applications. J Pers 60(2):175–215

    Article  Google Scholar 

  60. Mulder RT (2002) Personality pathology and treatment outcome in major depression: a review. Am J Psychiatry 159(3):359–371

    Article  Google Scholar 

  61. Odić A, Tkalčič M, Tasič J, Košir A (2013) Personality and social context: impact on emotion induction from movies. In: Extended proceedings of the conference on user modeling, adaptation, and personalization

  62. Ohi K, Shimada T, Nitta Y, Kihara H, Okubo H, Uehara T, Kawasaki Y (2016) The five-factor model personality traits in schizophrenia: a meta-analysis. Psychiatry Res 240:34–41

    Article  Google Scholar 

  63. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: British machine vision conference. British Machine Vision Association

  64. Pedregon CA, Farley RL, Davis A, Wood JM, Clark RD (2012) Social desirability, personality questionnaires, and the “better than average” effect. Personality Individ Differ 52(2):213–217

    Article  Google Scholar 

  65. Ponce-López V, Chen B, Oliu M, Corneanu C, Clapés A, Guyon I, Baró X, Escalante HJ, Escalera S (2016) Chalearn lap 2016: first round challenge on first impressions-dataset and results. In: Proceedings of the European conference on computer vision, pp 400–418. Springer

  66. Qiu L, Lu J, Yang S, Qu W, Zhu T (2015) What does your selfie say about you? Comput Hum Behav 52:443–449

    Article  Google Scholar 

  67. Quilty LC, De Fruyt F, Rolland JP, Kennedy SH, Rouillon PF, Bagby RM (2008) Dimensional personality traits and treatment outcome in patients with major depressive disorder. J Affect Disord 108(3):241–250

    Article  Google Scholar 

  68. Riggio HR, Riggio RE (2002) Emotional expressiveness, extraversion, and neuroticism: a meta-analysis. J Nonverbal Behav 26(4):195–218

    Article  Google Scholar 

  69. Salem H, Ruiz A, Hernandez S, Wahid K, Cao F, Karnes B, Beasley S, Sanches M, Ashtari E, Pigott T (2019) Borderline personality features in inpatients with bipolar disorder: impact on course and machine learning model use to predict rapid readmission. J Psychiatr Pract 25(4):279–289

    Article  Google Scholar 

  70. Segalin C, Cheng DS, Cristani M (2017) Social profiling through image understanding: personality inference using convolutional neural networks. Comput Vis Image Underst 156:34–50

    Article  Google Scholar 

  71. Selçuk E, Günaydin G, Sümer N, Uysal A (2005) Yetişkin bağlanma boyutlan için yeni bir ölçüm: Yakın ilişkilerde yaşantılar envanteri-II’nin Türk örnekleminde psikometrik açıdan değerlendirilmesi. Türk Psikoloji Yazıları

  72. Silveira Jacques Junior JC, Güçlütürk Y, Pérez M, Güçlü U, Andujar C, Baró X, Escalante HJ, Guyon I, Van Gerven MA, Van Lier R, Escalera S (2019) First impressions: a survey on vision-based apparent personality trait analysis. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2019.2930058

  73. Smeland OB, Wang Y, Lo MT, Li W, Frei O, Witoelar A, Tesli M, Hinds DA, Tung JY, Djurovic S, Chen CH, Dale AM, Andreassen OA (2017) Identification of genetic loci shared between schizophrenia and the big five personality traits. Sci Rep 7(1):1–9

    Article  Google Scholar 

  74. Sparkman DJ, Eidelman S, Dueweke AR, Marin MS, Dominguez B (2019) Open to diversity: openness to experience predicts beliefs in multiculturalism and colorblindness through perspective taking. J Individ Diff 40(1):1–12. https://doi.org/10.1027/1614-0001/a000270

    Article  Google Scholar 

  75. Spearman C (1961) The proof and measurement of association between two things. In: Jenkins JJ, Paterson DG (eds) Studies in individual differences: the search for intelligence. Appleton-Century-Crofts, pp 45–58. https://doi.org/10.1037/11491-005

  76. Subramaniam A, Patel V, Mishra A, Balasubramanian P, Mittal A (2016) Bi-modal first impressions recognition using temporally ordered deep audio and stochastic visual features. In: European conference on computer vision, pp 337–348. Springer

  77. Valente F, Kim S, Motlicek P (2012) Annotation and recognition of personality traits in spoken conversations from the ami meetings corpus. In: Annual conference of the international speech communication association

  78. Ventura C, Masip D, Lapedriza A (2017) Interpreting CNN models for apparent personality trait regression. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 55–63

  79. Vielzeuf V, Lechervy A, Pateux S, Jurie F (2018) Centralnet: a multilayer approach for multimodal fusion. In: Proceedings of the European conference on computer vision workshops, pp 575–589

  80. Vinciarelli A, Mohammadi G (2014) A survey of personality computing. IEEE Trans Affect Comput 5(3):273–291

    Article  Google Scholar 

  81. Wang J, Yang Y, Mao J, Huang Z, Huang C, Xu W (2016) CNN-RNN: a unified framework for multi-label image classification. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  82. Wauthia E, Lefebvre L, Huet K, Blekic W, El Bouragui K, Rossignol M (2019) Examining the hierarchical influences of the big-five dimensions and anxiety sensitivity on anxiety symptoms in children. Front Psychol 10:1185

    Article  Google Scholar 

  83. Wei XS, Zhang CL, Zhang H, Wu J (2017) Deep bimodal regression of apparent personality traits from short video sequences. IEEE Trans Affect Comput 9(3):303–315

    Article  Google Scholar 

  84. Weisberg YJ, DeYoung CG, Hirsh JB (2011) Gender differences in personality across the ten aspects of the big five. Front Psychol 2:178

    Article  Google Scholar 

  85. Wicaksana AS, Liem CC (2017) Human-explainable features for job candidate screening prediction. In: IEEE Conference on computer vision and pattern recognition workshops (CVPRW), pp 1664–1669. IEEE

  86. Willis J, Todorov A (2006) First impressions: Making up your mind after a 100-ms exposure to a face. Psychol Sci 17(7):592–598

    Article  Google Scholar 

  87. Wu X, He H, Shi L, Xia Y, Zuang K, Feng Q, Zhang Y, Ren Z, Wei D, Qiu J (2019) Personality traits are related with dynamic functional connectivity in major depression disorder: a resting-state analysis. J Affect Disord 245:1032–1042

    Article  Google Scholar 

  88. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1492–1500

  89. Yan Y, Nie J, Huang L, Li Z, Cao Q, Wei Z (2016) Exploring relationship between face and trustworthy impression using mid-level facial features. In: International conference on multimedia modeling, pp 540–549. Springer

  90. Yik MS, Russell JA (2001) Predicting the big two of affect from the big five of personality. J Res Pers 35(3):247–277

    Article  Google Scholar 

  91. Zillig LMP, Hemenover SH, Dienstbier RA (2002) What do we assess when we assess a big 5 trait? a content analysis of the affective, behavioral, and cognitive processes represented in big 5 personality inventories. Pers Soc Psychol Bull 28(6):847–858

    Article  Google Scholar 

Download references

Acknowledgements

Authors would like to thank Mr. Alper Dağgez, Ms. Merve Nur Doğanlı and Prof. Gül Günaydin for their contributions to this study.

Funding

Not applicable.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dersu Giritlioğlu.

Ethics declarations

Conflict of interest

The authors have no conflict of interest.

Ethical approval

The current study received the required ethical approval from the Ethics Committee of the university. Each participant of the experiment was informed about the procedure, and signed a consent form.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Giritlioğlu, D., Mandira, B., Yilmaz, S.F. et al. Multimodal analysis of personality traits on videos of self-presentation and induced behavior. J Multimodal User Interfaces 15, 337–358 (2021). https://doi.org/10.1007/s12193-020-00347-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-020-00347-7

Keywords

Navigation