Personal informatics and negative emotions during commuter driving: Effects of data visualization on cardiovascular reactivity & mood
Introduction
Negative emotions, such as anxiety, anger and depression, are associated with inflammatory processes in the human body (Camacho, 2013; Steptoe et al., 2007). This process of inflammation is directly linked to those biological processes that underpin the development of cardiovascular disease, such as atherosclerosis (Hansson, 2005; Suls and Bunde, 2005). The relationship between disease and negative emotions, such as anxiety (Thurston et al., 2013) and anger (Suls, 2013), is moderated by the cumulative influence of inflammation over the life course of the individual (Juster et al., 2010). According to the model of preservative cognition (Brosschot et al., 2010), this association is pernicious because physiological changes underpinning inflammation, which coincide with the experience of negative emotional states, can be sustained and often occur outside the conscious awareness of the individual (Lovallo, 2005). Previous research has associated a link between changes in heart rate variability (HRV) and biological markers of inflammation and reduced emotional regulation (Henriques et al., 2011). Both Low Frequency (LF) and High Frequency (HF) measures of HRV are associated with blood-borne indicators of inflammation, such as proinflammatory cytokines (e.g. Interleukin-6) and C-Reactive Protein (CRP) (Cooper et al., 2015), i.e. reduced HF activity of HRV is associated with increased activation of the parasympathetic nervous system and reduced inflammation.
Driving represents a common activity undertaken by millions of people each day that is associated with negative emotions and inflammation. The average American driver spends 46 min driving per journey, with UK drivers averaging 22 min per car trip (Dunn, 2015; National Travel Survey 2014, 2015). Over a period of weeks and months, the cumulation of these journeys amounts to a significant proportion of time spent by individuals on the road. The average driver can expect to experience negative emotions, such as anger (e.g. actions of other drivers) and anxiety (e.g. journey schedules), on a daily basis. These negative emotions can have a cumulative and detrimental influence on long-term cardiovascular health. For example, there is a positive association between traffic congestion and elevated blood pressure due to journey impedance (Schaeffer et al., 1988; Stokols et al., 1978). The development of driving sensors (Welch et al., 2019) and lifelogging technology with wearable physiological sensors represents one way to raise awareness of unconscious physiological changes associated with emotion during real-world driving.
The development of wearable technology that utilises physiological sensors and is capable of delivering feedback on the process of inflammation in everyday life could be valuable for long-term health, because: (1) it would create explicit awareness of the physiological change associated with an episode of anger or anxiety, and (2) increased awareness of these physiological changes in everyday life could act as a driver for the development of coping strategies that are adaptive and ameliorate the influence of negative emotions/inflammation on long-term health (Ganzel et al., 2010). The Levels of Emotional Awareness (LEA) theory (Lane and Schwartz, 1987) made a distinction between implicit/preconscious awareness of an emotion (e.g. bodily sensation) and explicit awareness where one is not only aware of a particular emotion but capable of verbally expressing the emotion in question. Recent work demonstrated that explicit awareness of negative emotion was a necessary precondition for the development of adaptive coping strategies, such as reappraisal (Subic-Wrana et al., 2014).
It is proposed that mobile lifelogging technology, where data are gathered from multiple sensors during everyday life, can promote the requisite self-reflection to promote coping strategies and behaviour change. The effectiveness of lifelogging technology to support the development of adaptive coping strategies depends on the provision of feedback to the individual and the way in which the user interacts with their own data.
The process of measurement and inference is central to the development of a mobile lifelogging platform and provision of feedback during everyday life. Both miniaturization of sensors and increased computational power enables mobile devices to capture a variety of personal data, including physiological signals and environmental information. The availability of this hardware affords an opportunity to collect data “in the wild” within real-life situations and present these data to individual users. For example, Hänsel et al. (2016) utilised a smartwatch to log current emotional states, via self-reports, and collect sensing data, including location, heart rate, physical activity, ambient noise and wrist movements. A smartphone application was also developed to enable participants to review their own data. Singh et al. (2014) collected photoplethysmogram (PPG), galvanic skin response (GSR) and respiration data to detect driver stress. Using a cascade forward neural network (CASFNN), their system achieved an overall accuracy of 80% when drivers drove around three pre-planned driving setups. In other works, the GStress model (Vhaduri et al., 2014) used smartphone GPS traces to estimates driver's stress using a generalized linear mixed model (GLMM) and captured a positive correlation coefficient (0.72) between subjective stress and GPS. Muaremi et al. (2013) utilised a smartphone to collect audio, physical activity, location and communication data during the day and a chest belt to collect heart rate variability data during sleep. Results of their multinomial logistic regression model indicated a maximum accuracy of 61% for a multiclass (low, moderate, and high stress) model. Whilst Castaldo et al. (2016) attempted to detect changes in mental stress by collecting HRV features via a 3-lead electrocardiogram (ECG) from students during an oral exam (stress period) and under controlled resting conditions (calm period); they reported that the C4.5 tree algorithm obtained a maximum accuracy of 79% classification between exam and rest.
In other lifelogging works, Affective Diary (Lindstrom et al., 2006) is an affective diary system that records pulse, step count and acceleration, as well as mobile phone data, including texts, multimedia messages, photographs and Bluetooth. The data that is collected is subsequently transformed into ambiguous, abstract colourful shapes. The purpose of this system was to allow users to organise, reflect and alter their diaries. Similarly, AffectAura (McDuff et al., 2012) is a system that continuously tracks valence, arousal and engagement and then transforms these data into the AffectAura lifelog visualisation, enabling users to reflect on their emotional experiences over extended periods of time. The system captured facial expressions, posture, speech, electro-dermal activity (EDA) via a wearable wrist sensor, location via GPS, file activity of web URLs visited, documents opened, applications used and emails sent and received and calendar information. The interface was structured in a timeline, with affective states visualised as coloured bubbles. Results indicated that participants found the interface useful and could “reason forward and backward in time about their emotional experiences” (McDuff et al., 2012). Meanwhile, FEEL (Ayzenberg et al., 2012) is a lifelogging system that measures EDA, via a wrist-worn commercial sensor, and captures mobile phone data to determine the context of social interactions. The data is analysed in real-time to associate stress (via EDA) with events captured via the mobile phone (e.g. emails, phone calls, etc.). The data is then combined into a digital journal of calendar/list events where the user can “reflect on his/her physiological responses” (Ayzenberg et al., 2012). A limited evaluation of the system was undertaken by one user over 200 h, whereby it was reported that the system enabled him to better recall past stressful events.
While the availability of wearable sensors improves the ease of data collection, this type of ambulatory measurement is associated with significant challenges with respect to the collection of high-quality data (Rahman et al., 2014) and the mitigation of confounding factors (Fahrenberg et al., 2007). Extensive signal processing and data analysis is often necessary to remove confounds from these data in order to draw robust inferences about behaviour and emotional states in the real-world.
If a lifelogging system can collect physiological data in a way that is both sensitive and reliable for the detection of negative emotions in everyday life, the next challenge for the designer is to deliver effective feedback to the user about these data. Feedback from a mobile lifelogging platform can support a process of introspection by transforming implicit emotional experience into explicit self-awareness, and in doing so, creating a foundational level of knowledge for the development of reappraisal, acceptance and other adaptive forms of coping (Boden and Thompson, 2015).
Ubiquitous technologies, which are dedicated to personal informatics (Li et al., 2011) and the quantified self (Shin and Biocca, 2017) deliver feedback that is based upon objective data in order to enhance self-improvement (Gemmell et al., 2006) or support cognitive function (Dobbins et al., 2014) . It has been argued that personal informatics support behavioural change by allowing the user to reflect upon behaviour, integrate personal data with a sense of self and finally take action (Li et al., 2010). The ability of personal informatic technology to support this type of behaviour change is influenced by the fidelity of the user interface (e.g. how personal history is represented) and utility of feedback to forecast future actions and feelings (Hollis et al., 2017). However, naïve users often encounter a number of obstacles during interaction with their personal data. The sheer volume of the database combined with abstract visualisations (e.g. bar or line charts) can make it difficult to efficiently extract meaning, rendering the process of self-tracking burdensome and unrewarding (Rapp and Cena, 2016). In addition, the absence of context often prevents the identification of triggers in the environment that can promote desirable or undesirable behaviours (Choe et al., 2014). The presentation of data visualisation at the interface is an important design issue as real-time feedback of negative emotions can amplify those emotions and be counterproductive. The act of emotional regulation in situ can be cognitively demanding (Gross, 2002) hence feedback may be more constructive when delivered retrospectively at a time of the user's choosing when they have sufficient cognitive capacity to reflect upon their data.
Previous research on visualisation of emotions includes StressTree (Yu et al., 2017), which is a heart rate variability (HRV) biofeedback system that transforms HRV data into a representation of a tree that metaphorically represents the health of the user as the growth pattern is affected by the users’ stress state. For example, when the user experiences high stress for a long period of time, the appearance of the tree grows more fragile. However, during periods of healthy balance the appearance of the tree is healthier. HRV was measured via a PPG sensor attached to the index finger. Participants reported that the visualisation promoted self-reflection of their behaviours but the interface perhaps would have been more suitable for visualisation on a mobile device, like a smartphone. Similarly, Chill-Out (Parnandi et al., 2014) is an adaptive biofeedback game that assists the user in developing relaxation skills by monitoring breathing. HRV and respiration were measured using a Bioharness sensor, whilst electrodermal activity (EDA) was measured using FlexComp Infinity encoder. Results indicated that playing Chill-Out led to lower arousal, in contrast to a non-biofeedback game and traditional relaxation methods, when participants were exposed to a stressor task. Affective Health (Sanches et al., 2010) is another stress management biofeedback mobile system that utilises a triaxial accelerometer, skin conductance, and three-lead ECG sensors to capture physiological data. The system was designed to allow users to see their own bodily reactions in real time and help them to identify stressful patterns in their data. The interface for this system consisted of two designs: layers and spirals, with colour representing arousal. They argued that feedback to the users should be presented using an interface design that encompassed properties, such as: ambiguity, openness to interpretation, interactive history, fluency and aliveness. Cardiomorphologies (Muller et al., 2006) included measures of breathing (via a respiratory strain gauge) and heart rate (via electrocardiogram (ECG), which were transformed into an interactive abstract visualisation of coloured rings. Results indicated “that the relationship between the visuals and the participants’ mental and physical states leads them to a feeling of engagement” (Muller et al., 2006). A different approach was adopted by StressEraser (Moore, 2007), which is a hardware device that measured heart rate via pulse oximetry and visualised the heart rate signal as a simple wave, accompanied by an audible tone, which supported the user to develop deep breathing exercises to alleviate stress. Whilst MoodWings (MacLean et al., 2013) is a real-time biofeedback system that captures EDA and ECG data to detect stress during simulated driving. The system utilises a wearable butterfly that depicts the user's stress state through movement of the wings, which acts as an early warning system and physical interface. Results indicated that MoodWings resulted in safer driving but also significantly increased both self-perceived and biophysical stress.
The primary contribution of the current work was to create an interactive data visualisation in the lifelogging mode that provided users with feedback on “hidden” cardiovascular measures of inflammation during everyday life. The primary purpose of the data visualisation was to increase awareness of implications for long-term health that are associated with the experience of negative emotions during commuter driving. The secondary goal of the data visualisation was to function as an agent for behavioural change. Unlike most work in this area that uses real-time feedback, this visualisation of personal data delivers feedback retrospectively when participants are able to reflect on negative emotions and their precedents. Therefore, the novelty of the current work is twofold: (1) to allow participants to make an association between inflammation, health and emotional responses during commuter driving in order to gain insight into their emotional experiences, and (2) to evaluate whether this kind of interaction with personal data led to demonstrable change in subjective mood or cardiovascular physiology during subsequent episodes of commuter driving.
The work included in this paper had four main objectives. The first was to investigate whether there is an association between cardiovascular markers and the experience of negative emotion during commuter driving in the real-world. Secondly, we wished to establish whether variables captured during commuter driving, i.e. both cardiovascular physiology and vehicle parameters (e.g. speed of car), could successfully classify episodes of high and low anger. Both objectives were addressed through study one that consists of a data collection exercise of thirteen participants conducted over five consecutive days of commuter driving. The third goal of the paper was to develop an interactive data visualisation that integrated cardiovascular reactivity, driving parameters via geolocation in combination with still images. The purpose of this interactive visualisation was to provide users with sufficient insight to develop adaptive coping strategies with respect to negative emotion and cardiovascular reactivity during their daily commute. The specific parameters and measures that we adopted for the data visualisation were directed by the results of the first study, specifically the process of feature selection used for classification of low vs. high subjective anger. Given that the purpose of the interactive data visualisation was to promote adaptive coping strategies, the fourth objective was to evaluate the effects of data visualisation on subjective mood and cardiovascular psychophysiology. The second study in the paper describes a study where data were collected during two days of commuter driving (as in study one) as a pre-test period, then participants spent a period of time interacting with the data visualisation, before data were collected from two more days of commuter driving (post-test period). It was hypothesised that exposure to the data visualisation would enhance self-awareness and promote adaptive coping strategies, hence negative mood and cardiovascular reactivity will be reduced during the post-test phase.
Section snippets
Design of study
Study one consisted of a data collection exercise to collect raw data over a period of five working days using the developed lifelogging platform (described below). This was a longitudinal data collection exercise designed to be integrated with participants’ normal working routines. Prior to providing written consent, participants were presented with an Information Sheet giving full details of the study. Participants were instructed that physiology and other parameters would be monitored over a
Design of study
The second study was designed with three distinct phases (see Fig. 6): (1) a pre-test period when data were collected during commuter journeys both to and from work on two successive days, (2) a data visualization session where participants were invited to the laboratory to interact with a data visualization based on data collected during the pre-test period, and (3) a post-test period when data were collected during commuter journeys on two successive days following exposure to data
Discussion
The goals of the studies were to: (1) inform the development of the data visualisation by utilising physiological variables that were associated with anger during commuter driving, (2) generate an interactive data visualisation that represented cardiovascular reactivity in everyday life, which was easy to interpret and raised awareness of negative emotions and underlying physiological activity, and (3) investigate the impact of interaction with this data visualisation on subsequent emotion and
Conclusion
The paper describes research into the development of a lifelogging system designed to capture negative emotions and cardiovascular activity during commuter driving and present interactive feedback to users. The first study demonstrated that features derived from psychophysiology and driving behaviour could be used to classify instances of high vs. low anger based upon subjective self-assessment. This study led to the development of an interactive data visualization wherein cardiovascular data
Acknowledgements
This work has been supported by the UK Engineering and Physical Sciences Research Council (EPSRC) under Research Grant EP/M029484/1. The authors would like to thank all of the participants for agreeing to take part in these studies. Owing to ethical concerns/the sensitive nature of this research, the data underlying this publication cannot be made openly available. Further information, including conditions for metadata access, can be found at LJMU Data Repository http://opendata.ljmu.ac.uk/24/.
References (71)
- et al.
Cardiac vagal control as a marker of emotion regulation in healthy adults: a review
Biol. Psychol.
(2017) - et al.
Conscious and unconscious perseverative cognition: is a large part of prolonged physiological activity due to unconscious stress?
J. Psychosom. Res
(2010) Is anxious-depression an inflammatory state?
Med Hypotheses
(2013)- et al.
Heart rate variability predicts levels of inflammatory markers: evidence for the vagal anti-inflammatory pathway
Brain. Behav. Immun.
(2015) - et al.
Allostatic load biomarkers of chronic stress and impact on health and cognition
Neurosci. Biobehav. Rev.
(2010) - et al.
Non-contact heart rate and heart rate variability measurements: a review
Biomed. Signal Process. Control
(2014) Cardiovascular reactivity: mechanisms and pathways to cardiovascular disease
Int. J. Psychophysiol.
(2005)- et al.
A review of methods for non-invasive and continuous blood pressure monitoring: pulse transit time method is promising?
IRBM
(2014) Cardiac vagal tone: a physiological index of stress
Neuoscience Biobehav. Rev.
(1995)- et al.
Personal informatics for everyday life
Int. J. Hum.-Comput. Stud.
(2016)
Health experience model of personal informatics: the case of a quantified self
Comput. Human Behav.
The effects of acute psychological stress on circulating inflammatory factors in humans: a review and meta-analysis
Brain. Behav. Immun.
Anger and the Heart: perspectives on Cardiac Risk, Mechanisms and Interventions
Prog Cardiovasc Dis
An anxious heart: anxiety and the onset of cardiovascular diseases
Prog Cardiovasc Dis
FEEL: frequent EDA and Event Logging – a Mobile Social Interaction Stress Monitoring System
Can smartphones detect stress-related changes in the behaviour of individuals?
Heart rate variability: origins, methods, and interpretive caveats
Psychophysiology
Facets of emotional awareness and associations with emotional regulation and depression
Emotion
Daily Stress Recognition from Mobile Phone Data, Weather Conditions and Individual Traits
Detection of mental stress due to oral academic examination via ultra-short-term HRV analysis
SMOTE: synthetic Minority Over-sampling Technique
J. Artif. Int. Res
Understanding quantified-selfers’ practices in collecting and exploring personal data
Signal Processing of Multimodal Mobile Lifelogging Data towards Detecting Stress in Real-World Driving
IEEE Trans. Mob. Comput.
Capturing Human Digital Memories for Assisting Memory Recall
American Driving Survey: Methodology and Year 1 Results, May 2013- May 2014
Heart rate variability: standards of measurement, physiological interpretation, and clinical use
Eur. Heart J
A Behavioural-Scientific Challenge for Psychology
Eur. J. Psychol. Assess.
Effects of mood induction via music on cardiovascular measures of negative emotion during simulated driving
Physiol. Behav.
Allostasis and the human brain: integrating models of stress from the social and life sciences
Psychol. Rev.
MyLifeBits: a personal database for everything
Commun. ACM
Continuous blood pressure measurement by using the pulse transit time: comparison to a cuff-based method
Eur. J. Appl. Physiol.
Emotion regulation: affective, cognitive, and social consequences
Psychophysiology
Large Scale Mood and Stress Self-Assessments on a Smartwatch
Inflammation, Atherosclerosis, and Coronary Artery Disease
N. Engl. J. Med
Cited by (15)
A multi-modal driver emotion dataset and study: Including facial expressions and synchronized physiological signals
2024, Engineering Applications of Artificial IntelligenceSandplayAR: Evaluation of psychometric game for people with generalized anxiety disorder
2022, Arts in PsychotherapyCitation Excerpt :Biofeedback device (DG3): Biofeedback data visualization is common in wearable emotion regulation products. Some studies have shown that real-time physical sign data visualization helps participants understand their emotional states and make self-adaptive adjustments (Fairclough & Dobbins, 2020). Ensure the stability and security of the virtual environment (DG4): As mentioned in the interview, some cards will not be recognized out of focus due to distance or angle.
Would John Dewey Wear a Fitbit? A Pragmatist Analysis of Self-Tracking Technologies’ Impact on Habit Formation
2024, Philosophy and TechnologyThe Future of Cognitive Personal Informatics
2023, Proceedings of the 25th International Conference on Mobile Human-Computer Interaction, MobileHCI 2023 Companion