Skip to main content
SpringerLink
Log in

Affective interaction: Using emotions as a user interface in games

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Affective computing, as a novel paradigm, adds a new feature into the human-computer interactions era that brings emotions into the process of design and evaluation. Such condition influences digital games’ not only structural dimension but also content and evaluation areas. This study aims to evaluate the effect of using emotions in the form of facial expressions as a User Interface (UI) and the input of a system in virtual scenarios. After studying the findings and methods of previous researches, a conceptual framework for Affective User Interface Design and Evaluation (AFFUIDE) has been proposed. Then, a game in the genre of shoot ‘em up (called Facial Battle) in four interaction modes has been developed. The evaluation of this study has been performed in terms of the players’ performance and experience. For this purpose, 18 male students of the Multimedia Faculty of Tabriz Islamic Art University took part in the experiments, who were asked to play all four modes of the game. Then, after analyzing the normalization of collected data, one-way ANOVA and U Mann Whitney tests were used for the statistical analysis. The findings indicate that the Traditional UI is the most usable and the Full Affective UI has the best sense of fun and user experience among other methods. However, the hybrid mode (affective shooting), is not only usable as traditional mode, but it also works better in terms of user experience compared to other modes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Andersson AT (2016) Facial feature tracking and head pose tracking as input for platform games

  2. Bahreini K, van der Vegt W, Westera W (2019) A fuzzy logic approach to reliable real-time recognition of facial emotions. Multimed Tools Appl 78:18943–18966

    Article  Google Scholar 

  3. Bannon L (2011) Reimagining HCI: toward a more human-centered perspective. Interactions 18:50–57

    Article  Google Scholar 

  4. Benyon D (2019) Designing User Experience: a guide to HCI, UX and interaction design. Pearson, London

  5. Bernays R, Mone J, Yau P et al (2012) Lost in the dark: emotion adaption. In: Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, New York, pp 79–80

  6. Blom PM, Bakkes S, Tan CT et al (2014) Towards personalised gaming via facial expression recognition. In: Tenth Artificial Intelligence and Interactive Digital Entertainment Conference

  7. Bogdanovych A, Trescak T, Simoff S (2016) What makes virtual agents believable? Conn Sci 28:83–108

    Article  Google Scholar 

  8. Bowman ND, Pietschmann D, Liebold B (2017) The golden (hands) rule: Exploring user experiences with gamepad and natural-user interfaces in popular video games. J Gaming Virtual Worlds 9:71–85

    Article  Google Scholar 

  9. Brooke J (1996) SUS-A quick and dirty usability scale. Usability Eval Ind 189:4–7

    Google Scholar 

  10. Browne C, Yannakakis GN, Colton S (2012) Guest editorial: Special issue on computational aesthetics in games. IEEE Trans Comput Intell AI Games 4:149–151

    Article  Google Scholar 

  11. Burns A, Tulip J (2017) Detecting flow in games using facial expressions. In: 2017 IEEE Conference on Computational Intelligence, Games (CIG). IEEE, Piscataway, pp 45–52

  12. Cai L, Liu B, Yu J, Zhang J (2017) Human behaviors modeling in multi-agent virtual environment. Multimed Tools Appl 76:5851–5871

    Article  Google Scholar 

  13. Caschera MC, Ferri F, Grifoni P (2016) Sentiment analysis from textual to multimodal features in digital environments. In: Proceedings of the 8th International Conference on Management of Digital EcoSystems. pp 137–144

  14. Čereković A, Pandžić IS (2011) Multimodal behavior realization for embodied conversational agents. Multimed Tools Appl 54:143–164

    Article  Google Scholar 

  15. Collins K, Kapralos B, Kanev K (2014) Smart table computer interaction interfaces with integrated sound

  16. De Byl P (2015) A conceptual affective design framework for the use of emotions in computer game design. Cyberpsychology J Psychosoc Res Cybersp 9

  17. de Peuter A (2014) Development of an adaptive game by runtime adjustment of game parameters to EEG measurements

  18. Dekker A, Champion E (2007) Please Biofeed the Zombies: Enhancing the Gameplay and Display of a Horror Game Using Biofeedback. In: DiGRA Conference. pp 43–52

  19. Dewitt A, Bresin R (2007) Sound design for affective interaction. In: International Conference on Affective Computing and Intelligent Interaction. Springer, Berlin, pp 523–533

  20. Drachen A, Nacke LE, Yannakakis G, Pedersen AL (2010) Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. In: Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games. ACM, New York, pp 49–54

  21. Dudley JJ, Kristensson PO (2018) A review of user interface design for interactive machine learning. ACM Trans Interact Intell Syst 8:8

    Article  Google Scholar 

  22. Dzedzickis A, Kaklauskas A, Bucinskas V (2020) Human emotion recognition: Review of sensors and methods. Sensors 20:592

    Article  Google Scholar 

  23. Ekman P (1999) Basic emotions. Handb Cogn Emot :45–60

  24. Ekman P, Friesen WV (1977) Facial action coding system

  25. Fu LP, Landay J, Nebeling M et al (2018) Redefining natural user interface. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, p SIG19

  26. Ghosh M, Kundu T, Ghosh D, Sarkar R (2019) Feature selection for facial emotion recognition using late hill-climbing based memetic algorithm. Multimed Tools Appl :1–27

  27. Gilleade K, Dix A, Allanson J (2005) Affective videogames and modes of affective gaming: assist me, challenge me, emote me. DiGRA 2005 Chang Views–Worlds Play

  28. Guthier B, Dörner R, Martinez HP (2016) Affective computing in games. In: Entertainment Computing and Serious Games. Springer, Berlin pp 402–441

  29. Hamdy S, King D (2018) Affective games: a multimodal classification system. In: 19th annual European GAME-ON Conference (GAME-ON’2018) on Simulation and AI in Computer Games. EUROSIS, Ghent

  30. Hassenzahl M, Burmester M, Koller F (2003) AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In: Mensch & computer 2003. Springer, Berlin, pp 187–196

  31. Högberg J, Hamari J, Wästlund E (2019) Gameful Experience Questionnaire (GAMEFULQUEST): an instrument for measuring the perceived gamefulness of system use. User Model User-adapt Interact 1–42

  32. Hubbard JA (2001) Emotion expression processes in children’s peer interaction: The role of peer rejection, aggression, and gender. Child Dev 72:1426–1438

    Article  Google Scholar 

  33. i Badia SB, Quintero LV, Cameirão MS et al (2018) Toward emotionally adaptive virtual reality for mental health applications. IEEE J Biomed Health Inform 23:1877–1887

    Article  Google Scholar 

  34. Ilves M, Gizatdinova Y, Surakka V, Vankka E (2014) Head movement and facial expressions as game input. Entertain Comput 5:147–156

    Article  Google Scholar 

  35. Ip B (2011) Narrative structures in computer and video games: Part 1: Context, definitions, and initial findings. Games Cult 6:103–134

    Article  Google Scholar 

  36. Jaimes A, Sebe N (2007) Multimodal human–computer interaction: A survey. Comput Vis Image Underst 108:116–134

    Article  Google Scholar 

  37. Ji H, Yun Y, Lee S et al (2018) An adaptable UI/UX considering user’s cognitive and behavior information in distributed environment. Cluster Comput 21:1045–1058

    Article  Google Scholar 

  38. Johnson D, Wiles J (2003) Effective affective user interface design in games. Ergonomics 46:1332–1345

    Article  Google Scholar 

  39. Kleinginna PR, Kleinginna AM (1981) A categorized list of emotion definitions, with suggestions for a consensual definition. Motiv Emot 5:345–379

    Article  Google Scholar 

  40. Korn O, Blatz M, Rees A et al (2017) Procedural content generation for game props? a study on the effects on user experience. Comput Entertain 15:1–15

    Article  Google Scholar 

  41. Kromand D (2007) Avatar Categorization. In: DiGRA Conference

  42. Landowska A, Miler J (2016) Limitations of emotion recognition in software user experience evaluation context. In: 2016 Federated Conference on Computer Science and Systems I (FedCSIS). IEEE, Piscataway, pp 1631–1640

  43. Lang PJ (1995) The emotion probe: studies of motivation and attention. Am Psychol 50:372

    Article  Google Scholar 

  44. Lankes M, Riegler S, Weiss A et al (2008) Facial expressions as game input with different emotional feedback conditions. In: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology. ACM, New York, pp 253–256

  45. Lim MY, Dias J, Aylett R, Paiva A (2012) Creating adaptive affective autonomous NPCs. Auton Agent Multi Agent Syst 24:287–311

    Article  Google Scholar 

  46. Liu C, Agrawal P, Sarkar N, Chen S (2009) Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback. Int J Hum Comput Interact 25:506–529

    Article  Google Scholar 

  47. Mandel T (1997) The elements of user interface design. Wiley, Hoboken

  48. Mandryk RL, Inkpen KM, Calvert TW (2006) Using psychophysiological techniques to measure user experience with entertainment technologies. Behav Inf Technol 25:141–158

    Article  Google Scholar 

  49. Mansilla WA (2006) Interactive dramaturgy by generating acousmêtre in a virtual environment. In: International Conference on Entertainment Computing. Springer, Berlin, pp 90–95

  50. McDuff D, Mahmoud A, Mavadati M et al (2016) AFFDEX SDK: a cross-platform real-time multi-face expression recognition toolkit. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. ACM, New York, pp 3723–3726

  51. Mohammadi HS, Pirbabaei E, Sisi MJ, Sekhavat YA (2018) ExerBrain: A Comparison of Positive and Negative Reinforcement in Attention Training Using BCI based Computer Games. In: 2018 2nd National and 1st International Digital Games Research Conference: Trends, Technologies, and Applications (DGRC). IEEE, Piscataway, pp 167–171

  52. Müller N, Eska B, Schäffer R et al (2018) Arch’n’Smile: A Jump’n’Run Game Using Facial Expression Recognition Control For Entertaining Children During Car Journeys. In: Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia. ACM, New York, pp 335–339

  53. Mungra D, Agrawal A, Sharma P et al (2019) PRATIT: a CNN-based emotion recognition system using histogram equalization and data augmentation. Multimed Tools Appl :1–23

  54. Ng YY, Khong CW (2014) A review of affective user-centered design for video games. In: 2014 3rd International Conference on User Science and Engineering (i-USEr). IEEE, Piscataway, pp 79–84

  55. Obaid M, Han C, Billinghurst M (2008) Feed the Fish: an affect-aware game. In: Proceedings of the 5th Australasian Conference on Interactive Entertainment. ACM, New York, p 6

  56. Pan S, Brdiczka O, Carenini G et al (2016) What’s Hot in Intelligent User Interfaces. In: Thirtieth AAAI Conference on Artificial Intelligence

  57. Partala T (2005) Affective information in human-computer interaction. Tampere University Press, Tampere

  58. Pawar S, Tam F, Plass JL (2020) 14 Emerging Design Factors in Game-Based Learning: Emotional Design, Musical Score, and Game Mechanics Design. Handb Game-Based Learn 347

  59. Picard RW (1995) Affective Computing-MIT Media Laboratory Perceptual Computing Section Technical Report No. 321. Cambridge, MA 2139

  60. Picard RW (2000) Affective computing. MIT Press, Cambridge

    Book  Google Scholar 

  61. Plutchik RE, Conte HR (1997) Circumplex models of personality and emotions. American Psychological Association, Washington, D.C.

  62. Preece J, Rogers Y, Sharp H et al (1994) Interface metaphors and conceptual models. Human-Computer Interation 9

  63. Prema P, Ramadoss B (2008) A Classification Scheme for Game Input and Output. World Acad Sci Eng Technol Int J Comput Electr Autom Control Inf Eng 2:1870–1876

    Google Scholar 

  64. Przybylski AK, Weinstein N, Murayama K et al (2012) The ideal self at play: The appeal of video games that let you be all you can be. Psychol Sci 23:69–76

    Article  Google Scholar 

  65. Raheel A, Anwar SM, Majid M (2019) Emotion recognition in response to traditional and tactile enhanced multimedia using electroencephalography. Multimed Tools Appl 78:13971–13985

    Article  Google Scholar 

  66. Robison J, McQuiggan S, Lester J (2009) Evaluating the consequences of affective feedback in intelligent tutoring systems. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops. IEEE, Piscataway, pp 1–6

  67. Ruan S, Chen L, Sun J, Chen G (2009) Study on the change of physiological signals during playing body-controlled games. In: Proceedings of the International Conference on Advances in Computer Enterntainment Technology. ACM, New York, pp 349–352

  68. Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39:1161

    Article  Google Scholar 

  69. Sawyer R, Smith A, Rowe J et al (2017) Enhancing student models in game-based learning with facial expression recognition. In: Proceedings of the 25th conference on user modeling, adaptation and personalization. ACM, New York, pp 192–201

  70. Sekhavat YA (2019) Battle of minds: a new interaction approach in BCI games through competitive reinforcement. Multimed Tools Appl :1–16

  71. Sekhavat YA (2020) Collaboration or battle between minds? An attention training game through collaborative and competitive reinforcement. Entertain Comput 100360

  72. Sekhavat YA, Roohi S, Mohammadi HS, Yannakakis GN (2020) Play with One’s Feelings: A Study on Emotion Awareness for Player Experience. IEEE Trans Games

  73. Sharma M, Jalal AS, Khan A (2019) Emotion recognition using facial expression by fusing key points descriptor and texture features. Multimed Tools Appl 78:16195–16219

    Article  Google Scholar 

  74. Shaw M (1986) An input-output model of interactive systems. In: Proceedings of the SIGCHI conference on Human factors in computing systems. pp 261–273

  75. Shilling R, Zyda M, Wardynski EC (2002) Introducing emotion into military simulation and videogame design: America’s Army: Operations and VIRTE. In: Proceedings of the GameOn Conference

  76. Song W, Cai X, Xi Y et al (2017) Real-time single camera natural user interface engine development. Multimed Tools Appl 76:11159–11175

    Article  Google Scholar 

  77. Souza AD, Souza RD (2019) Affective interaction based hybrid approach for emotion detection using machine learning. In: 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT). IEEE, Piscataway, pp 337–342

  78. Takatalo J, Häkkinen J, Kaistinen J, Nyman G (2010) Presence, involvement, and flow in digital games. In: Evaluating user experience in games. Springer, Berlin, pp 23–46

  79. Turečková Š (2016) Using player’s facial emotional expressions as a game input: Effects on narrative engagement

  80. Valli A (2008) The design of natural interaction. Multimed Tools Appl 38:295–305

    Article  Google Scholar 

  81. Von Ahn L, Dabbish L (2008) Designing games with a purpose. Commun ACM 51:58–67

    Article  Google Scholar 

  82. Werbach K, Hunter D (2012) For the win: How game thinking can revolutionize your business. Wharton Digital Press

  83. Xie L, Jia J, Meng H et al (2015) Expressive talking avatar synthesis and animation. Multimed Tools Appl 74:9845–9848

    Article  Google Scholar 

  84. Yannakakis GN, Togelius J (2015) Experience-driven procedural content generation. In: 2015 International Conference on Affective Computing and Interaction I (ACII). IEEE, Piscataway, pp 519–525

  85. Yeh Y, Lai SC, Lin C-W (2016) The dynamic influence of emotions on game-based creativity: An integrated analysis of emotional valence, activation strength, and regulation focus. Comput Human Behav 55:817–825

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Multimedia, Tabriz Islamic Art University, Tabriz, Iran

    Yoones A. Sekhavat, Milad Jafari Sisi & Samad Roohi

Authors
  1. Yoones A. Sekhavat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Milad Jafari Sisi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Samad Roohi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yoones A. Sekhavat.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This paper is an excerpt from the master’s thesis of Milad Jafari Sisi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sekhavat, Y.A., Sisi, M.J. & Roohi, S. Affective interaction: Using emotions as a user interface in games. Multimed Tools Appl 80, 5225–5253 (2021). https://doi.org/10.1007/s11042-020-10006-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10006-4

Keywords

Search

Navigation