Empirical and modeling study of emotional state dynamics in social videogame paradigms

Abstract

The objective of this work was to study the dynamics of human emotional states in the process of social interaction in a virtual environment. The previously developed for this purpose prototypes of the virtual actor (NPC) and its virtual environment simulator “Teleport” underwent significant re-design and modification. The experimental platform was re-implemented and used in experiments with college student participants, combining electromyography, emotion recognition in facial recordings and model-based game log analysis in a social videogame paradigm. Participants interacted with two virtual actors implemented based on the eBICA cognitive architecture (Samsonovich, 2013, 2018). Positive correlations were found between eBICA model predictions and participant affects extracted from their facial expressions and facial muscle activity. Affective dynamics of social phenomena, such as the establishment of partnership or an act of betrayal, were characterized and found consistent with the model predictions. Other findings include a gradually developing emotional reaction, possibly due to the integration of appraisals of game events. Overall, obtained results confirm the eBICA model, suggesting its further extension and refinement.

Introduction

The development of artificial intelligence of the next generations implies the provision of natural human interaction with intellectual agents at the social level, including emotional contact. Therefore, it is not surprising that today increasing attention is paid to the recognition and expression of affects by artifacts. At the same time, the question of choosing an adequate emotional response remains in the background, despite its paramount importance. In order to allow a virtual character or robot to be psychologically realistic, socially acceptable and effective in collaboration with a user, it must respond adequately to the emotionally meaningful actions of other agents. A promising approach to implementation of this capability in artifacts is based on biologically inspired cognitive architectures (BICA) Laird et al., 2017, Samsonovich & De Jong, 2005, Chella et al., 2011, that should be augmented with a suitable model of emotional intelligence. However, at the moment the models of natural affective behavior are not comprehensive enough for their wide acceptance by the world scientific and engineering community, as evidenced by the achieved level Breazeal, 2003, Gratch and Marsella, 2004, Leontiev, 2002, Simonov, 1982, Fominykh, 2006, Ilyin, 2007.

The aim of this work is to study the dynamics of human affective states in the process of social interaction in a virtual environment. To do this, we used the platform we created earlier - the game “Teleport” (Azarnov, Chubarov, & Samsonovich, 2018) (see below) that enables anonymous social interaction of actors of different natures - both humans and automatons - with an empirical proof of their indistinguishability, which ensures that the virtual actor behaves naturally. The platform has been substantially revised and redesigned. The paradigm of this game requires collaborative interaction between actors-participants, who are a human and two machines in this case. The data on the dynamics of participant’s affective states in the chosen paradigm allows us to measure semantic correlates of participants' affects and confirm or correct the model formulated earlier (Samsonovich, 2013, Samsonovich, 2018a, Samsonovich, 2018b). The sign and intensity of the emotional experience of the participant can be established based on the analysis of activity of two muscles: M. Zygomaticus Major and M. Corrugator Supercilii. The validity of this analysis and the rationale of the specific choice of muscles are confirmed by both classical and recent work (Fridlund and Cacioppo, 1986, Golland et al., 2018, Philip et al., 2018, Rohr et al., 2018), indicating that the activity of the first muscle is positively correlated with the intensity of experienced joy (which indicates a positive valence), and the second one – with anger (which indicates a negative valence). Electromyography (EMG) was used to record the activity of these muscles and, consequently, emotional experience.

The key purpose of the chosen scheme for the embodiment of Virtual Actors is to build and use a semantic map Lövheim, 2012, Plutchik, 1982, Russell, 1980, Samsonovich & Ascoli, 2013, Samsonovich et al., 2010 as an element of the cognitive architecture of eBICA (Samsonovich, 2013, Samsonovich, 2018a), as illustrated in Fig. 1. In this scheme, the agent chooses actions taking into account both traditional rational factors (for example, expediency in terms of achieving the goal) and socio-emotional factors (for example, the agreement of the appraisal of the selected action and the object of the action). The latter is determined on the basis of the so-called moral schemes (Samsonovich, 2013, Samsonovich, 2018a, Samsonovich, 2018b), which determine probabilities of particular behaviors of the agent.

The virtual environment setup (Azarnov, Chubarov, & Samsonovich, 2018) consists of a virtual stage, in which three avatars are allocated. The stage includes a rescue zone “the tower” (the left component in Fig. 2) and an action zone “the platform” (the right component in Fig. 2). Each avatar can be controlled by a human player or a virtual actor. The platform has two teleporter terminals (circles on the platform, Fig. 2). Actors can be moved from the platform to the tower by means of teleportation, as described below. The game session consists of a sequence of logically identical rounds. Each round has a fixed limited duration, and may terminate earlier, if certain conditions are met, as explained below. Following the termination, a new round starts automatically, until the 10-minute session time limit expires. Initially, all actors are placed on the platform at random locations.

Each actor located on the platform has the following available behaviors: greet, ask for help, or thank another actor, kick another actor, activate or de-activate a remote teleporter (available from a teleporter only; cannot activate own teleporter), and take off (perform own teleportation, available from an active teleport only). The take-off action moves the actor from the platform to the tower. When on the tower, the actor may perform the following two actions, each of which leads to the termination of the round: save a selected actor located on the platform or escape alone. The round also terminates automatically whenever two actors reach the tower, or the 2-minute time limit expires. Upon termination, all actors located on the tower win, others lose. The virtual environment also includes the clock, the score meter, and checkboxes that allow participants to indicate, who is their partner at the current moment.