Skip to main content
SpringerLink
Log in

Analyzing Connections Between User Attributes, Images, and Text

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

This work explores the relationship between a person’s demographic/psychological traits (e.g., gender and personality) and self-identity images and captions. We use a dataset of images and captions provided by N ≈ 1350 individuals, and we automatically extract features from both the images and captions. We identify several visual and textual properties that show reliable relationships with individual differences between participants. The automated techniques presented here allow us to draw interesting conclusions from our data that would be difficult to identify manually, and these techniques are extensible to other large datasets. Additionally, we consider the task of predicting gender and personality using both single modality features and multimodal features. We show that a multimodal predictive approach outperforms purely visual methods and purely textual methods. We believe that our work on the relationship between user characteristics and user data has relevance in online settings, where users upload billions of images each day.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. This data was collected under IRB approval at UT Austin.

  2. For prediction results, we use a slightly different version of dominance (Dominance = 0.76y + 0.32s), as formulated in Machajdik and Hanbury [59].

  3. Available at https://code.google.com/archive/p/word2vec/.

References

  1. Meeker M. 2014. Internet trends 2014–Code conference.

  2. Wendlandt L, Mihalcea R, Boyd R, Pennebaker J. Multimodal analysis and prediction of latent user dimensions. Proceedings of the 9th international conference on social informatics (SocInfo 2017). Oxford, UK; 2017. p. 323–340.

  3. Boyd RL. Psychological text analysis in the digital humanities. Data analytics in digital humanities. Springer; 2017. p. 161–189.

  4. Coppersmith G, Dredze M, Harman C, Hollingshead K. From ADHD to SAD: analyzing the language of mental health on twitter through self-reported diagnoses. Proceedings of the 2nd workshop on computational linguistics and clinical psychology: from linguistic signal to clinical reality; 2015. p. 1–10.

  5. Conover M, Gonçalves B, Ratkiewicz J, Flammini A, Menczer F. Predicting the political alignment of twitter users. Proceedings of 3rd IEEE conference on social computing (SocialCom); 2011. p. 192–199.

  6. Cohen R, Ruths D. Classifying political orientation on Twitter: it’s not easy! Proceedings of the seventh international AAAI conference on weblogs and social media (ICWSM 2013); 2013 . p. 91–99.

  7. van der Goot R, Ljubesić N, Matroos I, Nissim M, Plank B. Bleaching text: abstract features for cross-lingual gender prediction. Proceedings of the 56th annual meeting of the association for computational linguistics; 2018. p. 383–389.

  8. Ciccone G, Sultan A, Laporte L, Egyed-Zsigmond E, Alhamzeh A, Granitzer M. Stacked gender prediction from tweet texts and images notebook for PAN at CLEF 2018. CLEF 2018 - conference and labs of the evaluation; 2018. p. 11.

  9. Mukherjee A, Liu B. Improving gender classification of blog authors. Proceedings of the 2010 conference on empirical methods in natural language processing; 2010. p. 207–217.

  10. Rao D, Yarowsky D, Shreevats A, Gupta M. Classifying latent user attributes in twitter. Proceedings of the 2nd international workshop on search and mining user-generated contents; 2010 . p. 37–44.

  11. Burger JD, Henderson J, Kim G, Zarrella G. Discriminating gender on Twitter. Proceedings of the conference on empirical methods in natural language processing; 2011. p. 1301–1309.

  12. Van Durme B. Streaming analysis of discourse participants. Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning; 2012. p. 48–58.

  13. Volkova S, Yarowsky D. Improving gender prediction of social media users via weighted annotator rationales. NeurIPS workshop on personalization; 2014.

  14. Volkova S, Bachrach Y, Armstrong M, Sharma V. Inferring latent user properties from texts published in social media. AAAI conference on artificial intelligence; 2015. p. 4296–4297.

  15. Pennacchiotti M, Popescu AM. Democrats, republicans and starbucks afficinados: user classification in twitter. Proceedings of the 17th ACM SIGKDD international conference on knowledge discovery and data mining; 2011. p. 430–438.

  16. Eisenstein J, Smith NA, Xing EP. Discovering sociolinguistic associations with structured sparsity. Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies; 2011. p. 1365–1374.

  17. Rao D, Paul M, Fink C, Yarowsky D, Oates T, Coppersmith G. Hierarchical Bayesian models for latent attribute detection in social media. International AAAI conference on weblogs and social media; 2011. p. 598–601.

  18. Li Y, Yang L, Xu B, Wang J, Lin H. Improving user attribute classification with text and social network attention. Cognitive Comput 2019;11(4):459–468.

    Article  Google Scholar 

  19. Favaretto RM, Knob P, Musse SR, Vilanova F, Costa ÂB. Detecting personality and emotion traits in crowds from video sequences. Machine Vision and Applications 2019;30(5):999–1012.

    Article  Google Scholar 

  20. Al-Ghadir AI, Azmi AM. A study of arabic social media users—posting behavior and author’s gender prediction. Cognitive Computation 2019;11(1):71–86.

    Article  Google Scholar 

  21. Favaretto RM, Knob P, Musse SR, Vilanova F, Costa ÂB. Detecting personality and emotion traits in crowds from video sequences. Machine Vision and Applications 2019;30(5):999–1012.

    Article  Google Scholar 

  22. An G, Levitan SI, Hirschberg J, Levitan R. Deep personality recognition for deception detection. Interspeech; 2018 . p. 421–425.

  23. Moreno DRJ, Gomez JC, Almanza-Ojeda DL, Ibarra-Manzano MA. Prediction of personality traits in twitter users with latent features. 2019 international conference on electronics, communications and computers; 2019. p. 176–181.

  24. Bose R, Dey RK, Roy S, Sarddar D. Analyzing political sentiment using twitter data. Information and communication technology for intelligent systems. Singapore: Springer; 2019 . p. 427–436.

  25. Volkova S, Durme BV. Online Bayesian models for personal analytics in social media. AAAI conference on artificial intelligence; 2015. p. 2325–2331.

  26. Seabrook EM, Kern ML, Fulcher BD, Rickard NS. Predicting depression from language-based emotion dynamics: longitudinal analysis of Facebook and Twitter status updates. J Med Internet Res 2018 May;20(5):e168.

  27. Riordan B, Wade H, Upal A. Detecting sociostructural beliefs about group status differences in online discussions. Proceedings of the joint workshop on social dynamics and personal attributes in social media; 2014. p. 1–6.

  28. Gottipati S, Qiu M, Yang L, Zhu F, Jiang J. An integrated model for user attribute discovery: a case study on political affiliation identification. Advances in knowledge discovery and data mining. vol. 8443 of lecture notes in computer science. In: Tseng V, Ho T, Zhou Z H, Chen A P, and Kao H Y, editors. Springer; 2014 . p. 434–446.

  29. Schwartz HA, Eichstaedt JC, Kern ML, Dziurzynski L, Ramones SM, Agrawal M, et al. Personality, gender, and age in the language of social media: the open vocabulary approach. PLOS ONE 2013;8(9):1–16.

    Article  Google Scholar 

  30. Chang J, Rosenn I, Backstrom L, Marlow C. ePluribus: ethnicity on social networks. Proceedings of the fourth international AAAI conference on weblogs and social media; 2010. p. 18–25.

  31. Mohammady E, Culotta A. Using county demographics to infer attributes of twitter users. Proceedings of the joint workshop on social dynamics and personal attributes in social media; 2014. p. 7–16.

  32. Yang SH, Long B, Smola A, Sadagopan N, Zheng Z, Zha H. Like like alike: joint friendship and interest propagation in social networks. Proceedings of the 20th international conference on World Wide Web. WWW ’11; 2011. p. 537–546.

  33. Gong NZ, Talwalkar A, Mackey LW, Huang L, Shin ECR, Stefanov E, et al. Predicting links and inferring attributes using a social-attribute network (SAN). The 6th SNA-KDD workshop; 2012.

  34. Filippova K. User demographics and language in an implicit social network. Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL); 2012. p. 1478– 1488.

  35. Nguyen D, Gravel R, Trieschnigg D, Meder T. How old do you think I am? A study of language and age in Twitter. Proceedings of the AAAI conference on weblogs and social media (ICWSM); 2013. p. 439–448.

  36. Bergsma S, Post M, Yarowsky D. Stylometric analysis of scientific articles. Proceedings of the North American association of computational linguistics. Montrea, CA; 2012 . p. 327–337.

  37. Bergsma S, Dredze M, Durme BV, Wilson T, Yarowsky D. Broadly improving user classification via communication-based name and location clustering on twitter. Proceedings of the conference of the North American chapter of the association for computational linguistics: human language technologies; 2013 . p. 1010–1019.

  38. Eisenstein J, O’Connor B, Smith NA, Xing EP. A latent variable model for geographic lexical variation. Proceedings of the 2010 conference on empirical methods in natural language processing. EMNLP ’10; 2010. p. 1277–1287.

  39. Kelly EL, Conley JJ. Personality and compatibility: a prospective analysis of marital stability and marital satisfaction. J Personality Social Psychol 1987;52(1):27.

    Article  Google Scholar 

  40. Roberts B, Kuncel N, Shiner R, Caspi A, Goldberg L. The power of personality: the comparative validity of personality traits, socioeconomic status, and cognitive ability for predicting important life outcomes. Perspectives on Psychological Science 2007;4(2):313–345.

    Article  Google Scholar 

  41. Park G, Schwartz HA, Eichstaedt JC, Kern ML, Kosinski M, Stillwell DJ, et al. 2014. Automatic personality assessment through social media language. Journal of Personality and Social Psychology.

  42. Pennebaker JW, King LA. Linguistic styles: language use as an individual difference. Journal of Personality and Social Psychology 1999;77(6):1296.

    Article  Google Scholar 

  43. Mairesse F, Walker MA, Mehl MR, Moore RK. Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of Artificial Intelligence Research 2007;30:457–500.

    Article  Google Scholar 

  44. Whitty MT, Doodson J, Creese S, Hodges DA. Picture tells a thousand words: what Facebook and Twitter images convey about our personality. Personality and Individual Differences 2018;133:109–114.

    Article  Google Scholar 

  45. Lay A, Ferwerda B. Predicting users’ personality based on their ‘liked’ images on Instagram. The 23rd international on intelligent user interfaces, March 7-11, 2018; 2018.

  46. Newman ML, Groom CJ, Handelman LD, Pennebaker JW. Gender differences in language use: an analysis of 14,000 text samples. Discourse Processes 2008;45(3):211–236.

    Article  Google Scholar 

  47. You Q, Bhatia S, Sun T, Luo J. The eyes of the beholder: gender prediction using images posted in online social networks. IEEE; 2014. p. 1026–1030.

  48. Zhang D, Islam MM, Lu G. A review on automatic image annotation techniques. Pattern Recognition 2012; 45(1):346–362.

    Article  Google Scholar 

  49. Hossain M, Sohel F, Shiratuddin MF, Laga H. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CSUR) 2019;51(6):118.

    Article  Google Scholar 

  50. Mithun NC, Panda R, Papalexakis EE, Roy-Chowdhury AK. Webly supervised joint embedding for cross-modal image-text retrieval. Proceedings of the 26th ACM international conference on multimedia. MM ’18. New York: ACM; 2018 . p. 1856–1864.

  51. Johnson J, Karpathy A, Fei-Fei L. Densecap: fully convolutional localization networks for dense captioning. Proceedings of the IEEE conference on computer vision and pattern recognition; 2016. p. 4565–4574.

  52. McCrae RR, John OP. An introduction to the five-factor model and its applications. Journal of Personality 1992;60(2):175– 215.

    Article  Google Scholar 

  53. John OP, Srivastava S. The Big Five trait taxonomy: history, measurement, and theoretical perspectives. Handbook of Personality: Theory and Research 1999;2(1999):102–138.

    Google Scholar 

  54. Yoder PJ, Blackford JU, Waller NG, Kim G. Enhancing power while controlling family-wise error: an illustration of the issues using electrocortical studies. Journal of Clinical and Experimental Neuropsychology 2004;26 (3):320–331.

    Article  Google Scholar 

  55. Redi M, Quercia D, Graham L, Gosling S. Like partying? Your face says it all. Predicting the ambiance of places with profile pictures. Ninth international AAAI conference on web and social media; 2015.

  56. Khouw N. 2002. The meaning of color for gender. Colors Matters–Research.

  57. Van De Weijer J, Schmid C, Verbeek J, Larlus D. Learning color names for real-world applications. IEEE Transactions on Image Processing 2009;18(7):1512–1523.

  58. Valdez P, Mehrabian A. Effects of color on emotions. Journal of Experimental Psychology: General 1994; 123(4):394.

    Article  Google Scholar 

  59. Machajdik J, Hanbury A. Affective image classification using features inspired by psychology and art theory. Proceedings of the 18th ACM international conference on multimedia. ACM; 2010. p. 83–92.

  60. Lovato P, Bicego M, Segalin C, Perina A, Sebe N, Cristani M. Faved! Biometrics: tell me which image you like and I’ll tell you who you are. IEEE Transactions on Information Forensics and Security 2014;9 (3):364–374.

    Article  Google Scholar 

  61. Gosling SD, Ko SJ, Mannarelli T, Morris MEA. Room with a cue: personality judgments based on offices and bedrooms. Journal of Personality and Social Psychology 2002;82(3):379.

    Article  Google Scholar 

  62. Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A. Learning deep features for scene recognition using places database. Advances in neural information processing systems; 2014 . p. 487–495.

  63. Mathias M, Benenson R, Pedersoli M, Van Gool L. Face detection without bells and whistles. European conference on computer vision. Springer; 2014. p. 720–735.

  64. Gosling SD, Craik KH, Martin NR, Pryor MR. Material attributes of personal living spaces. Home Cultures 2005;2(1):51–87.

    Article  Google Scholar 

  65. Fellbaum C. 1998. Wordnet. Wiley Online Library.

  66. Ciaramita M, Johnson M. Supersense tagging of unknown nouns in WordNet. Proceedings of the 2003 conference on empirical methods in natural language processing. Association for Computational Linguistics; 2003. p. 168–175.

  67. Bentivogli L, Forner P, Magnini B, Pianta E. Revising the WordNet domains hierarchy: semantics, coverage and balancing. Proceedings of the workshop on multilingual linguistic ressources. Association for Computational Linguistics; 2004. p. 101–108.

  68. Finkel JR, Grenager T, Manning C. Incorporating non-local information into information extraction systems by Gibbs sampling. Proceedings of the 43rd annual meeting on association for computational linguistics; 2005. p. 363–370.

  69. Li JJ, Nenkova A. Fast and accurate prediction of sentence specificity. AAAI; 2015. p. 2281–2287.

  70. Coltheart M. The MRC psycholinguistic database. The Quarterly Journal of Experimental Psychology 1981; 33(4):497–505.

    Article  Google Scholar 

  71. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems; 2013. p. 3111–3119.

  72. Oberlander J, Nowson S. Whose thumb is it anyway?: classifying author personality from weblog text. COLING/ACL; 2006 . p. 627–634.

Download references

Acknowledgments

We would like to thank Samuel Gosling for helping with the dataset collection, Shibamouli Lahiri for providing the code to calculate readability features, and Steven R. Wilson for providing the code to implement the Mairesse et al. paper that we use for prediction comparison.

Funding

This material is based in part upon work supported by the National Science Foundation (#1344257), the John Templeton Foundation (#48503), the Michigan Institute for Data Science, and DARPA (grant #HR001117S0026-AIDA-FP-045). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation, the John Templeton Foundation, the Michigan Institute for Data Science, or DARPA.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Michigan, Bob and Betty Beyster Building, 2260 Hayward Street, Ann Arbor, MI, 48109-2121, USA

    Laura Burdick & Rada Mihalcea

  2. Department of Psychology, Lancaster University, Lancaster, LA1 4YF, UK

    Ryan L. Boyd

  3. Department of Psychology, The University of Texas at Austin, SEA 4.208, 108 E. Dean Keeton Stop A8000, Austin, TX, 78712-1043, USA

    James W. Pennebaker

Authors
  1. Laura Burdick
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rada Mihalcea
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ryan L. Boyd
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. James W. Pennebaker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura Burdick.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflicts of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Burdick, L., Mihalcea, R., Boyd, R.L. et al. Analyzing Connections Between User Attributes, Images, and Text. Cogn Comput 13, 241–260 (2021). https://doi.org/10.1007/s12559-019-09695-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-019-09695-3

Keywords

Search

Navigation