Automated affect classification and task difficulty adaptation in a competitive scenario based on physiological linkage: An exploratory study

https://doi.org/10.1016/j.ijhcs.2021.102673Get rights and content

Highlights

  • In multi-user scenarios, difficulty generally adapted based on performance.

  • Could alternatively be adapted based on physiological measurements of both users.

  • Automated classification of human psychological states in competitive scenario.

  • Based on individual physiological responses and physiological linkage.

  • Task difficulty then dynamically adapted based on classified human states.

Abstract

In competitive and cooperative scenarios, task difficulty should be dynamically adapted to suit people with different abilities. State-of-the-art difficulty adaptation methods for such scenarios are based on task performance, which conveys little information about user-specific factors such as workload. Thus, we present an exploratory study of automated affect recognition and task difficulty adaptation in a competitive scenario based on physiological linkage (covariation of participants’ physiological responses). Classification algorithms were developed in an open-loop study where 16 pairs played a competitive game while 5 physiological responses were measured: respiration, skin conductance, electrocardiogram, and 2 facial electromyograms. Physiological and performance data were used to classify four self-reported variables (enjoyment, valence, arousal, perceived difficulty) into two or three classes. The highest classification accuracies were obtained for perceived difficulty: 84.3% for two-class and 60.5% for three-class classification. As a proof of concept, the developed classifiers were used in a small closed-loop study to dynamically adapt game difficulty. While this closed-loop study found no clear advantages of physiology-based adaptation, it demonstrated the technical feasibility of such real-time adaptation. In the long term, physiology-based task adaptation could enhance competition and cooperation in many multi-user settings (e.g., education, manufacturing, exercise).

Introduction

Competition and cooperation between multiple interacting humans are popular in many areas of human-machine interaction. Perhaps most famously, computer games tend to include a multiplayer mode that lets several players compete or cooperate with each other (Chanel et al., 2012). However, there are also many serious examples of competitive and cooperative human-machine interaction scenarios. For example, students can cooperate or compete in educational scenarios to learn different topics (Zhou et al., 2020), patients in technology-assisted language therapy can work with each other to relearn words (Grechuta et al., 2016), and stroke survivors in technology-assisted motor rehabilitation (Baur et al., 2018; Goršič, et al., 2017) as well as overweight adults in technology-assisted weight loss programs (Esakia, et al., 2020) can compete or cooperate with each other for increased motivation and exercise intensity. While such multi-user scenarios have many potential benefits, they also present new challenges. One challenge is that such scenarios often include two users with different skills and abilities – for example, a severely impaired patient working with a mildly impaired patient. In such cases, the task difficulty should be intelligently balanced so that both users remain engaged by the task, usefully contribute to it, and derive maximum benefit from it. But how can this adaptation be performed most effectively?

In single-user scenarios, difficulty adaptation is usually performed based on either task performance or physiological responses. Performance is a task-specific concept – for example, the score in a computer game or the number of successfully completed actions in a rehabilitation exercise. It can be easily measured and can serve as the basis for simple, intuitive adaptation rules (e.g., if score is high, make the game harder). However, although performance is a good indicator of a person's capabilities, it does not provide precise information regarding their internal, subjective state. For example, a person can exhibit acceptable performance but at the cost of high workload that may lead to stress and fatigue (Watson et al., 1996). Therefore, many studies have instead focused on physiological measurements such as the electrocardiogram (ECG), respiration and skin conductance, which provide an estimate of the person's cognitive and affective (emotional) state, allowing difficulty to be adapted in a more personalized manner (e.g., if player workload is high, reduce game difficulty) (Novak et al., 2012). Assessment of a user's cognitive and affective states from physiological responses and scenario adaptation in response to these user states falls under the field of affective computing (Picard et al., 2001) and is used for diverse purposes such as attention and workload recognition in drivers (Fan et al., 2018), computer game difficulty adaptation (Liu et al., 2009), and adaptive automation in flight (Wilson and Russell, 2007). Studies of single-user scenarios have shown that physiology-based adaptation often outperforms performance-based adaptation (Bian et al., 2019; Liu et al., 2009; Xu et al., 2018); thus, physiology-based adaptation might also be promising for multi-user scenarios.

To date, task difficulty adaptation in competitive and cooperative scenarios has been almost exclusively based on task performance (Baldwin et al., 2013; Chih-Yueh et al., 2013; Goršič et al., 2017; Vicencio-Moreira et al., 2014) rather than physiological measurements. However, given the demonstrated advantages of physiology-based adaptation over performance-based adaptation in single-user scenarios, such physiology-based adaptation should be investigated in multi-user scenarios as well. In our previous study, we took the first step by preliminarily evaluating a physiology-based adaptation method for a competitive scenario, but that method only adapted difficulty to suit one person based on that person's physiological responses and ignored the other person (Darzi et al., 2017). Nonetheless, it demonstrated the feasibility of physiology-based difficulty adaptation for multi-user scenarios. The goal of the current study was thus to adapt the difficulty of a competitive scenario to suit both participants based on their physiological responses.

In a multi-user scenario, we could study each participant's physiological responses individually and use this as a basis for difficulty adaptation, as done in our previous study. However, we can also study the similarity (degree of synchronization or mutual variation) of the participants' physiological responses, thus obtaining information related specifically to the interaction between participants. This is referred to as physiological linkage (or synchronization), and is not simply due to participants perceiving the same stimuli (Haataja et al., 2018; Pérez et al., 2017). The degree of linkage increases with the amount of cooperation (Ahonen et al., 2016; Hu et al., 2018; Szymanski et al., 2017), the intensity of competition (Liu et al., 2017; Spapé et al., 2013) or simply the degree of shared attention (Chênes et al., 2013; Muszynski et al., 2018); thus, it may be able to provide additional information that would complement information obtained from individual physiological responses, as suggested by previous studies (Chanel et al., 2016). However, despite the unique potential of physiological linkage, it has never been used for task difficulty adaptation. Most applications have been limited to group-level correlational studies (Delaherche et al., 2012; Reidsma et al., 2010; Tschacher et al., 2014); one research group did perform automated classification of movie highlights based on spectators’ physiological linkage, but this is a fundamentally different application (Chênes et al., 2013; Muszynski et al., 2018).

This paper, to the best of our knowledge, thus presents the first use of two participants’ physiological responses for automated affect classification and consequent dynamic difficulty adaptation in a competitive scenario. Each participant's individual physiological responses, different metrics of physiological linkage, and task performance were measured during different competitive game conditions and used to train affect classification algorithms. Different performance- and physiology-based classifiers were then compared with regard to both offline accuracy and their effect on user experience in a closed-loop real-time difficulty adaptation study. Our plan was originally to obtain a large sample of participants for the closed-loop study and thoroughly evaluate the effectiveness of such closed-loop adaptation; however, due to the COVID-19 pandemic, data collection had to be terminated early, and the study is thus presented as an exploratory proof-of-concept. Our research questions were:

  • RQ1: How accurately can physiological measurements classify human cognitive and affective states in a competitive scenario? This has been previously extensively explored in single-user scenarios, where physiological responses are most commonly classified into either two classes (e.g., low/high enjoyment) or three classes (e.g., low/medium/high stress) (Novak et al., 2012), but not in competitive scenarios. As a baseline for qualitative comparison, the same classification was also done with a few simple performance measurements, which are much easier to obtain and analyze.

  • RQ2: Does the addition of physiological linkage information allow more accurate affect classification than using only individual physiological responses? Physiological linkage calculation requires more signal processing than using only individual responses, as both participants’ physiological responses must be time-synchronized and analyzed together. Thus, adding it is useful only if it results in higher classification accuracy.

  • RQ3: (Preliminarily) Does physiology-based difficulty adaptation result in a positive user experience? High offline classification accuracy is not guaranteed to transfer to accurate real-time difficulty adaptation (Fairclough et al., 2015; McCrea et al., 2017). For example, there is more potential for erroneous adaptation decisions since artefacts that can be easily manually removed in offline processing cannot be quickly removed during gameplay itself; at the same time, users may be able to compensate for erroneous decisions made by the system by adapting their own behavior (Fairclough and Lotte, 2020). Our original goal was to compare the effectiveness of physiology-based adaptation to that of performance-based adaptation similarly to what has previously been done in single-user studies (Bian et al., 2019; Liu et al., 2009; Xu et al., 2018); however, as data collection was interrupted by COVID-19, we have limited ourselves to demonstrating the technical feasibility of providing a positive user experience using physiology-based adaptation. Performance-based adaptation was included as a baseline for qualitative comparison.

A preliminary version of this study was published as a 2019 conference paper (Darzi and Novak, 2019). It used the same study setup (hardware and competitive game) as the current paper. The differences between this paper and the preliminary conference version are as follows:

  • -

    The conference version included no closed-loop adaptation, though such adaptation was mentioned briefly as a future step. The current version includes closed-loop adaptation based on individual physiological responses and physiological linkage, with simple performance-based adaptation also included as a baseline for comparison.

  • -

    The conference version included only preliminary open-loop classification (two classes, only a subset of the classifiers and features used in this paper). The current version includes both two- and three-class classification with a larger set of classifiers and features. Furthermore, the current version describes the most informative features in classification.

  • -

    The conference version included more extensive analysis of group-level differences using analyses of variance to verify that different game difficulty levels induce different physiological responses. This has been omitted from the current paper, as we wished to focus more on classification and adaptation.

  • -

    The conference version included a somewhat smaller open-loop participant sample (12 pairs vs. 16 in the current paper).

Section snippets

Methodology

This paper describes two related studies: an open-loop study used to train affect classification algorithms for a competitive scenario based on two participants’ physiological responses, and a closed-loop study where difficulty adaptation based on this affect classification was compared to performance-based adaptation. Both studies were approved by the University of Wyoming Institutional Review Board. Section 2.1 presents the study hardware and setup, which was identical for both studies.

Open-loop study

Table 3 presents the mean 2-class classification accuracies for five combinations of three feature sets. The highest accuracy (84.3%) was obtained for classification of perceived difficulty using the combination of all feature sets. The combination of individual physiological features and physiological linkage features as well as the combination of all feature sets yielded the highest accuracy for enjoyment and valence as well; for arousal, the highest accuracy was obtained with a combination

Open-loop classification accuracies

Tables 3 and 4 show classification accuracies for both two-class and three-class classification of different cognitive and affective states (perceived difficulty, enjoyment, valence, arousal) using different feature sets. Both the two-class and three-class classification accuracies are comparable to those seen in single-user studies. Our highest reported accuracies based on only physiological responses were 81.1% for two-class and 59.5% for three-class classification; for comparison, our 2012

Conclusion

To our knowledge, this paper presents the first time that two participants’ physiological responses (including physiological linkage) were used to classify the participants’ cognitive and affective states in a competitive scenario and dynamically adapt the difficulty of that scenario based on the classified states. In the open-loop part of the paper, different combinations of physiological and performance data were used to classify four self-reported variables related to cognitive and affective

CRediT authorship contribution statement

Ali Darzi: Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing. Domen Novak: Conceptualization, Data curation, Funding acquisition, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

Research supported by the National Science Foundation under grants no. 1717705 and 2007908 as well as by the National Institute of General Medical Sciences of the National Institutes of Health under grant no. 2P20GM103432.

References (55)

  • C. Szymanski et al.

    Teams on the same wavelength perform better: Inter-brain phase synchronization constitutes a neural substrate for social facilitation

    Neuroimage

    (2017)
  • I. Van Diest et al.

    Anxiety and respiratory variability

    Physiol. Behav.

    (2006)
  • A.R. Watson et al.

    Effects of task difficulty on pilot workload

    Comput. Ind. Eng.

    (1996)
  • F. Zhou et al.

    Affect prediction from physiological measures via visual stimuli

    Int. J. Hum. Comput. Stud.

    (2011)
  • L. Ahonen et al.

    Cognitive collaboration found in cardiac physiology: Study in classroom environment

    PLoS One

    (2016)
  • Alessio, S. M. (2016). Digital signal processing and spectral analysis for scientists....
  • A. Baldwin et al.

    A framework of dynamic difficulty adjustment in competitive multiplayer video games

  • K. Baur et al.

    Trends in robot-assisted and virtual reality-assisted neuromuscular therapy: a systematic review of health-related multiplayer games

    J. Neuroeng. Rehabil.

    (2018)
  • D. Bian et al.

    Design of a physiology-based adaptive virtual reality driving platform for individuals with ASD

    ACM Transact. Access. Comput.

    (2019)
  • W. Boucsein

    Electrodermal Activity

    (2012)
  • G. Chanel et al.

    Grand challenge problem 2: adaptive awareness for social regulation of emotions in online collaborative learning environments

    Grand Challenge Problems in Technology-Enhanced Learning II

    (2016)
  • C. Chênes et al.

    Highlight detection in movie scenes through inter-users, physiological linkage

    Soc. Media Retr.

    (2013)
  • C. Chih-Yueh et al.

    Evenly matched competitive strategies: dynamic difficulty adaptation in a game-based learning system

    Res. Pract. Technol. Enhanc. Learn.

    (2013)
  • A. Darzi et al.

    Difficulty adaptation in a competitive arm rehabilitation game using real-time control of arm electromyogram and respiration

  • A. Darzi et al.

    Using physiological linkage for patient state assessment in a competitive rehabilitation game

  • A. Darzi et al.

    Classification of multiple psychological dimensions of computer game players using physiology, performance and personality characteristics

    Front. Neurosci.

    (2019)
  • E. Delaherche et al.

    Interpersonal synchrony: a survey of evaluation methods across disciplines

    IEEE Trans. Affective Comput.

    (2012)
  • Cited by (14)

    View all citing articles on Scopus
    View full text