Automated affect classification and task difficulty adaptation in a competitive scenario based on physiological linkage: An exploratory study
Introduction
Competition and cooperation between multiple interacting humans are popular in many areas of human-machine interaction. Perhaps most famously, computer games tend to include a multiplayer mode that lets several players compete or cooperate with each other (Chanel et al., 2012). However, there are also many serious examples of competitive and cooperative human-machine interaction scenarios. For example, students can cooperate or compete in educational scenarios to learn different topics (Zhou et al., 2020), patients in technology-assisted language therapy can work with each other to relearn words (Grechuta et al., 2016), and stroke survivors in technology-assisted motor rehabilitation (Baur et al., 2018; Goršič, et al., 2017) as well as overweight adults in technology-assisted weight loss programs (Esakia, et al., 2020) can compete or cooperate with each other for increased motivation and exercise intensity. While such multi-user scenarios have many potential benefits, they also present new challenges. One challenge is that such scenarios often include two users with different skills and abilities – for example, a severely impaired patient working with a mildly impaired patient. In such cases, the task difficulty should be intelligently balanced so that both users remain engaged by the task, usefully contribute to it, and derive maximum benefit from it. But how can this adaptation be performed most effectively?
In single-user scenarios, difficulty adaptation is usually performed based on either task performance or physiological responses. Performance is a task-specific concept – for example, the score in a computer game or the number of successfully completed actions in a rehabilitation exercise. It can be easily measured and can serve as the basis for simple, intuitive adaptation rules (e.g., if score is high, make the game harder). However, although performance is a good indicator of a person's capabilities, it does not provide precise information regarding their internal, subjective state. For example, a person can exhibit acceptable performance but at the cost of high workload that may lead to stress and fatigue (Watson et al., 1996). Therefore, many studies have instead focused on physiological measurements such as the electrocardiogram (ECG), respiration and skin conductance, which provide an estimate of the person's cognitive and affective (emotional) state, allowing difficulty to be adapted in a more personalized manner (e.g., if player workload is high, reduce game difficulty) (Novak et al., 2012). Assessment of a user's cognitive and affective states from physiological responses and scenario adaptation in response to these user states falls under the field of affective computing (Picard et al., 2001) and is used for diverse purposes such as attention and workload recognition in drivers (Fan et al., 2018), computer game difficulty adaptation (Liu et al., 2009), and adaptive automation in flight (Wilson and Russell, 2007). Studies of single-user scenarios have shown that physiology-based adaptation often outperforms performance-based adaptation (Bian et al., 2019; Liu et al., 2009; Xu et al., 2018); thus, physiology-based adaptation might also be promising for multi-user scenarios.
To date, task difficulty adaptation in competitive and cooperative scenarios has been almost exclusively based on task performance (Baldwin et al., 2013; Chih-Yueh et al., 2013; Goršič et al., 2017; Vicencio-Moreira et al., 2014) rather than physiological measurements. However, given the demonstrated advantages of physiology-based adaptation over performance-based adaptation in single-user scenarios, such physiology-based adaptation should be investigated in multi-user scenarios as well. In our previous study, we took the first step by preliminarily evaluating a physiology-based adaptation method for a competitive scenario, but that method only adapted difficulty to suit one person based on that person's physiological responses and ignored the other person (Darzi et al., 2017). Nonetheless, it demonstrated the feasibility of physiology-based difficulty adaptation for multi-user scenarios. The goal of the current study was thus to adapt the difficulty of a competitive scenario to suit both participants based on their physiological responses.
In a multi-user scenario, we could study each participant's physiological responses individually and use this as a basis for difficulty adaptation, as done in our previous study. However, we can also study the similarity (degree of synchronization or mutual variation) of the participants' physiological responses, thus obtaining information related specifically to the interaction between participants. This is referred to as physiological linkage (or synchronization), and is not simply due to participants perceiving the same stimuli (Haataja et al., 2018; Pérez et al., 2017). The degree of linkage increases with the amount of cooperation (Ahonen et al., 2016; Hu et al., 2018; Szymanski et al., 2017), the intensity of competition (Liu et al., 2017; Spapé et al., 2013) or simply the degree of shared attention (Chênes et al., 2013; Muszynski et al., 2018); thus, it may be able to provide additional information that would complement information obtained from individual physiological responses, as suggested by previous studies (Chanel et al., 2016). However, despite the unique potential of physiological linkage, it has never been used for task difficulty adaptation. Most applications have been limited to group-level correlational studies (Delaherche et al., 2012; Reidsma et al., 2010; Tschacher et al., 2014); one research group did perform automated classification of movie highlights based on spectators’ physiological linkage, but this is a fundamentally different application (Chênes et al., 2013; Muszynski et al., 2018).
This paper, to the best of our knowledge, thus presents the first use of two participants’ physiological responses for automated affect classification and consequent dynamic difficulty adaptation in a competitive scenario. Each participant's individual physiological responses, different metrics of physiological linkage, and task performance were measured during different competitive game conditions and used to train affect classification algorithms. Different performance- and physiology-based classifiers were then compared with regard to both offline accuracy and their effect on user experience in a closed-loop real-time difficulty adaptation study. Our plan was originally to obtain a large sample of participants for the closed-loop study and thoroughly evaluate the effectiveness of such closed-loop adaptation; however, due to the COVID-19 pandemic, data collection had to be terminated early, and the study is thus presented as an exploratory proof-of-concept. Our research questions were:
RQ1: How accurately can physiological measurements classify human cognitive and affective states in a competitive scenario? This has been previously extensively explored in single-user scenarios, where physiological responses are most commonly classified into either two classes (e.g., low/high enjoyment) or three classes (e.g., low/medium/high stress) (Novak et al., 2012), but not in competitive scenarios. As a baseline for qualitative comparison, the same classification was also done with a few simple performance measurements, which are much easier to obtain and analyze.
RQ2: Does the addition of physiological linkage information allow more accurate affect classification than using only individual physiological responses? Physiological linkage calculation requires more signal processing than using only individual responses, as both participants’ physiological responses must be time-synchronized and analyzed together. Thus, adding it is useful only if it results in higher classification accuracy.
RQ3: (Preliminarily) Does physiology-based difficulty adaptation result in a positive user experience? High offline classification accuracy is not guaranteed to transfer to accurate real-time difficulty adaptation (Fairclough et al., 2015; McCrea et al., 2017). For example, there is more potential for erroneous adaptation decisions since artefacts that can be easily manually removed in offline processing cannot be quickly removed during gameplay itself; at the same time, users may be able to compensate for erroneous decisions made by the system by adapting their own behavior (Fairclough and Lotte, 2020). Our original goal was to compare the effectiveness of physiology-based adaptation to that of performance-based adaptation similarly to what has previously been done in single-user studies (Bian et al., 2019; Liu et al., 2009; Xu et al., 2018); however, as data collection was interrupted by COVID-19, we have limited ourselves to demonstrating the technical feasibility of providing a positive user experience using physiology-based adaptation. Performance-based adaptation was included as a baseline for qualitative comparison.
A preliminary version of this study was published as a 2019 conference paper (Darzi and Novak, 2019). It used the same study setup (hardware and competitive game) as the current paper. The differences between this paper and the preliminary conference version are as follows:
- -
The conference version included no closed-loop adaptation, though such adaptation was mentioned briefly as a future step. The current version includes closed-loop adaptation based on individual physiological responses and physiological linkage, with simple performance-based adaptation also included as a baseline for comparison.
- -
The conference version included only preliminary open-loop classification (two classes, only a subset of the classifiers and features used in this paper). The current version includes both two- and three-class classification with a larger set of classifiers and features. Furthermore, the current version describes the most informative features in classification.
- -
The conference version included more extensive analysis of group-level differences using analyses of variance to verify that different game difficulty levels induce different physiological responses. This has been omitted from the current paper, as we wished to focus more on classification and adaptation.
- -
The conference version included a somewhat smaller open-loop participant sample (12 pairs vs. 16 in the current paper).
Section snippets
Methodology
This paper describes two related studies: an open-loop study used to train affect classification algorithms for a competitive scenario based on two participants’ physiological responses, and a closed-loop study where difficulty adaptation based on this affect classification was compared to performance-based adaptation. Both studies were approved by the University of Wyoming Institutional Review Board. Section 2.1 presents the study hardware and setup, which was identical for both studies.
Open-loop study
Table 3 presents the mean 2-class classification accuracies for five combinations of three feature sets. The highest accuracy (84.3%) was obtained for classification of perceived difficulty using the combination of all feature sets. The combination of individual physiological features and physiological linkage features as well as the combination of all feature sets yielded the highest accuracy for enjoyment and valence as well; for arousal, the highest accuracy was obtained with a combination
Open-loop classification accuracies
Tables 3 and 4 show classification accuracies for both two-class and three-class classification of different cognitive and affective states (perceived difficulty, enjoyment, valence, arousal) using different feature sets. Both the two-class and three-class classification accuracies are comparable to those seen in single-user studies. Our highest reported accuracies based on only physiological responses were 81.1% for two-class and 59.5% for three-class classification; for comparison, our 2012
Conclusion
To our knowledge, this paper presents the first time that two participants’ physiological responses (including physiological linkage) were used to classify the participants’ cognitive and affective states in a competitive scenario and dynamically adapt the difficulty of that scenario based on the classified states. In the open-loop part of the paper, different combinations of physiological and performance data were used to classify four self-reported variables related to cognitive and affective
CRediT authorship contribution statement
Ali Darzi: Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing. Domen Novak: Conceptualization, Data curation, Funding acquisition, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgment
Research supported by the National Science Foundation under grants no. 1717705 and 2007908 as well as by the National Institute of General Medical Sciences of the National Institutes of Health under grant no. 2P20GM103432.
References (55)
- et al.
Emotions and respiratory patterns: review and critical analysis
Int. J. Psychophysiol.
(1994) - et al.
Physiological compliance for social gaming analysis: cooperative versus competitive play
Interact. Comput.
(2012) - et al.
Using smartwatches to facilitate a group dynamics-based statewide physical activity intervention
Int. J. Hum. Comput. Stud.
(2020) - et al.
Monitoring in collaborative learning: Co-occurrence of observed behavior and physiological synchrony explored
Comput. Hum. Behav.
(2018) - et al.
Inter-brain synchrony and cooperation context in interactive decision making
Biol. Psychol.
(2018) Autonomic nervous system activity in emotion: a review
Biol. Psychol.
(2010)- et al.
A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies
Int. J. Hum. Comput. Stud.
(2007) - et al.
Toward a system of customized video game mechanics based on player personality: relating the Big Five personality traits with difficulty adaptation in a first-person shooter game
Entertain. Comput.
(2016) - et al.
A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing
Interact. Comput.
(2012) - et al.
Development of a contextual decision-making game for improving students’ learning performance in a health education course
Comput. Edu.
(2015)
Teams on the same wavelength perform better: Inter-brain phase synchronization constitutes a neural substrate for social facilitation
Neuroimage
Anxiety and respiratory variability
Physiol. Behav.
Effects of task difficulty on pilot workload
Comput. Ind. Eng.
Affect prediction from physiological measures via visual stimuli
Int. J. Hum. Comput. Stud.
Cognitive collaboration found in cardiac physiology: Study in classroom environment
PLoS One
A framework of dynamic difficulty adjustment in competitive multiplayer video games
Trends in robot-assisted and virtual reality-assisted neuromuscular therapy: a systematic review of health-related multiplayer games
J. Neuroeng. Rehabil.
Design of a physiology-based adaptive virtual reality driving platform for individuals with ASD
ACM Transact. Access. Comput.
Electrodermal Activity
Grand challenge problem 2: adaptive awareness for social regulation of emotions in online collaborative learning environments
Grand Challenge Problems in Technology-Enhanced Learning II
Highlight detection in movie scenes through inter-users, physiological linkage
Soc. Media Retr.
Evenly matched competitive strategies: dynamic difficulty adaptation in a game-based learning system
Res. Pract. Technol. Enhanc. Learn.
Difficulty adaptation in a competitive arm rehabilitation game using real-time control of arm electromyogram and respiration
Using physiological linkage for patient state assessment in a competitive rehabilitation game
Classification of multiple psychological dimensions of computer game players using physiology, performance and personality characteristics
Front. Neurosci.
Interpersonal synchrony: a survey of evaluation methods across disciplines
IEEE Trans. Affective Comput.
Cited by (14)
Effects of adaptation accuracy and magnitude in affect-aware difficulty adaptation for the multi-attribute task battery
2024, International Journal of Human Computer StudiesFusing ECG signals and IRT models for task difficulty prediction in computerised educational systems
2023, Knowledge-Based SystemsDesign premise and diary study exploring felt senses as data for self-reflection
2024, Design JournalAutomated Classification of Dyadic Conversation Scenarios Using Autonomic Nervous System Responses
2023, IEEE Transactions on Affective ComputingChallengeDetect: Investigating the Potential of Detecting In-Game Challenge Experience from Physiological Measures
2023, Conference on Human Factors in Computing Systems - Proceedings