Personal informatics and negative emotions during commuter driving: Effects of data visualization on cardiovascular reactivity & mood

Highlights

• Classification of anger requires features derived from psychophysiology and driving.

• Data from driving provides context for psychophysiological assessment.

• Interactive data visualisation reduced heart rate during traffic congestion.

• Wearable sensors can provide transformative insight into real-world emotions.

Abstract

Mobile technology and wearable sensors can provide objective measures of psychological stress in everyday life. Data from sensors can be visualized and viewed by the user to increase self-awareness and promote adaptive coping strategies. A capacity to effectively self-regulate negative emotion can mitigate the biological process of inflammation, which has implications for long-term health. Two studies were undertaken utilizing a mobile lifelogging platform to collect cardiovascular data over a week of real-life commuter driving. The first was designed to establish a link between cardiovascular markers of inflammation and the experience of anger during commuter driving in the real world. Results indicated that an ensemble classification model provided an accuracy rate of 73.12% for the binary classification of episodes of high vs. low anger based upon a combination of features derived from driving (e.g. vehicle speed) and cardiovascular psychophysiology (heart rate, heart rate variability, pulse transit time). During the second study, participants interacted with an interactive, geolocated visualisation of vehicle parameters, photographs and cardiovascular psychophysiology collected over two days of commuter driving (pre-test). Data were subsequently collected over two days of driving following their interaction with the dynamic, data visualization (post-test). A comparison of pre- and post-test data revealed that heart rate significantly reduced during episodes of journey impedance after interaction with the data visualization. There was also evidence that heart rate variability increased during the post-test phase, suggesting greater vagal activation and adaptive coping. Subjective mood data were collected before and after each journey, but no statistically significant differences were observed between pre- and post-test periods. The implications of both studies for ambulatory monitoring, user interaction and the capacity of personal informatics to enhance long-term health are discussed.

Introduction

Negative emotions, such as anxiety, anger and depression, are associated with inflammatory processes in the human body (Camacho, 2013; Steptoe et al., 2007). This process of inflammation is directly linked to those biological processes that underpin the development of cardiovascular disease, such as atherosclerosis (Hansson, 2005; Suls and Bunde, 2005). The relationship between disease and negative emotions, such as anxiety (Thurston et al., 2013) and anger (Suls, 2013), is moderated by the cumulative influence of inflammation over the life course of the individual (Juster et al., 2010). According to the model of preservative cognition (Brosschot et al., 2010), this association is pernicious because physiological changes underpinning inflammation, which coincide with the experience of negative emotional states, can be sustained and often occur outside the conscious awareness of the individual (Lovallo, 2005). Previous research has associated a link between changes in heart rate variability (HRV) and biological markers of inflammation and reduced emotional regulation (Henriques et al., 2011). Both Low Frequency (LF) and High Frequency (HF) measures of HRV are associated with blood-borne indicators of inflammation, such as proinflammatory cytokines (e.g. Interleukin-6) and C-Reactive Protein (CRP) (Cooper et al., 2015), i.e. reduced HF activity of HRV is associated with increased activation of the parasympathetic nervous system and reduced inflammation.

Driving represents a common activity undertaken by millions of people each day that is associated with negative emotions and inflammation. The average American driver spends 46 min driving per journey, with UK drivers averaging 22 min per car trip (Dunn, 2015; National Travel Survey 2014, 2015). Over a period of weeks and months, the cumulation of these journeys amounts to a significant proportion of time spent by individuals on the road. The average driver can expect to experience negative emotions, such as anger (e.g. actions of other drivers) and anxiety (e.g. journey schedules), on a daily basis. These negative emotions can have a cumulative and detrimental influence on long-term cardiovascular health. For example, there is a positive association between traffic congestion and elevated blood pressure due to journey impedance (Schaeffer et al., 1988; Stokols et al., 1978). The development of driving sensors (Welch et al., 2019) and lifelogging technology with wearable physiological sensors represents one way to raise awareness of unconscious physiological changes associated with emotion during real-world driving.

The development of wearable technology that utilises physiological sensors and is capable of delivering feedback on the process of inflammation in everyday life could be valuable for long-term health, because: (1) it would create explicit awareness of the physiological change associated with an episode of anger or anxiety, and (2) increased awareness of these physiological changes in everyday life could act as a driver for the development of coping strategies that are adaptive and ameliorate the influence of negative emotions/inflammation on long-term health (Ganzel et al., 2010). The Levels of Emotional Awareness (LEA) theory (Lane and Schwartz, 1987) made a distinction between implicit/preconscious awareness of an emotion (e.g. bodily sensation) and explicit awareness where one is not only aware of a particular emotion but capable of verbally expressing the emotion in question. Recent work demonstrated that explicit awareness of negative emotion was a necessary precondition for the development of adaptive coping strategies, such as reappraisal (Subic-Wrana et al., 2014).

It is proposed that mobile lifelogging technology, where data are gathered from multiple sensors during everyday life, can promote the requisite self-reflection to promote coping strategies and behaviour change. The effectiveness of lifelogging technology to support the development of adaptive coping strategies depends on the provision of feedback to the individual and the way in which the user interacts with their own data.

The process of measurement and inference is central to the development of a mobile lifelogging platform and provision of feedback during everyday life. Both miniaturization of sensors and increased computational power enables mobile devices to capture a variety of personal data, including physiological signals and environmental information. The availability of this hardware affords an opportunity to collect data “in the wild” within real-life situations and present these data to individual users. For example, Hänsel et al. (2016) utilised a smartwatch to log current emotional states, via self-reports, and collect sensing data, including location, heart rate, physical activity, ambient noise and wrist movements. A smartphone application was also developed to enable participants to review their own data. Singh et al. (2014) collected photoplethysmogram (PPG), galvanic skin response (GSR) and respiration data to detect driver stress. Using a cascade forward neural network (CASFNN), their system achieved an overall accuracy of 80% when drivers drove around three pre-planned driving setups. In other works, the GStress model (Vhaduri et al., 2014) used smartphone GPS traces to estimates driver's stress using a generalized linear mixed model (GLMM) and captured a positive correlation coefficient (0.72) between subjective stress and GPS. Muaremi et al. (2013) utilised a smartphone to collect audio, physical activity, location and communication data during the day and a chest belt to collect heart rate variability data during sleep. Results of their multinomial logistic regression model indicated a maximum accuracy of 61% for a multiclass (low, moderate, and high stress) model. Whilst Castaldo et al. (2016) attempted to detect changes in mental stress by collecting HRV features via a 3-lead electrocardiogram (ECG) from students during an oral exam (stress period) and under controlled resting conditions (calm period); they reported that the C4.5 tree algorithm obtained a maximum accuracy of 79% classification between exam and rest.

In other lifelogging works, Affective Diary (Lindstrom et al., 2006) is an affective diary system that records pulse, step count and acceleration, as well as mobile phone data, including texts, multimedia messages, photographs and Bluetooth. The data that is collected is subsequently transformed into ambiguous, abstract colourful shapes. The purpose of this system was to allow users to organise, reflect and alter their diaries. Similarly, AffectAura (McDuff et al., 2012) is a system that continuously tracks valence, arousal and engagement and then transforms these data into the AffectAura lifelog visualisation, enabling users to reflect on their emotional experiences over extended periods of time. The system captured facial expressions, posture, speech, electro-dermal activity (EDA) via a wearable wrist sensor, location via GPS, file activity of web URLs visited, documents opened, applications used and emails sent and received and calendar information. The interface was structured in a timeline, with affective states visualised as coloured bubbles. Results indicated that participants found the interface useful and could “reason forward and backward in time about their emotional experiences” (McDuff et al., 2012). Meanwhile, FEEL (Ayzenberg et al., 2012) is a lifelogging system that measures EDA, via a wrist-worn commercial sensor, and captures mobile phone data to determine the context of social interactions. The data is analysed in real-time to associate stress (via EDA) with events captured via the mobile phone (e.g. emails, phone calls, etc.). The data is then combined into a digital journal of calendar/list events where the user can “reflect on his/her physiological responses” (Ayzenberg et al., 2012). A limited evaluation of the system was undertaken by one user over 200 h, whereby it was reported that the system enabled him to better recall past stressful events.

While the availability of wearable sensors improves the ease of data collection, this type of ambulatory measurement is associated with significant challenges with respect to the collection of high-quality data (Rahman et al., 2014) and the mitigation of confounding factors (Fahrenberg et al., 2007). Extensive signal processing and data analysis is often necessary to remove confounds from these data in order to draw robust inferences about behaviour and emotional states in the real-world.

If a lifelogging system can collect physiological data in a way that is both sensitive and reliable for the detection of negative emotions in everyday life, the next challenge for the designer is to deliver effective feedback to the user about these data. Feedback from a mobile lifelogging platform can support a process of introspection by transforming implicit emotional experience into explicit self-awareness, and in doing so, creating a foundational level of knowledge for the development of reappraisal, acceptance and other adaptive forms of coping (Boden and Thompson, 2015).

Ubiquitous technologies, which are dedicated to personal informatics (Li et al., 2011) and the quantified self (Shin and Biocca, 2017) deliver feedback that is based upon objective data in order to enhance self-improvement (Gemmell et al., 2006) or support cognitive function (Dobbins et al., 2014) . It has been argued that personal informatics support behavioural change by allowing the user to reflect upon behaviour, integrate personal data with a sense of self and finally take action (Li et al., 2010). The ability of personal informatic technology to support this type of behaviour change is influenced by the fidelity of the user interface (e.g. how personal history is represented) and utility of feedback to forecast future actions and feelings (Hollis et al., 2017). However, naïve users often encounter a number of obstacles during interaction with their personal data. The sheer volume of the database combined with abstract visualisations (e.g. bar or line charts) can make it difficult to efficiently extract meaning, rendering the process of self-tracking burdensome and unrewarding (Rapp and Cena, 2016). In addition, the absence of context often prevents the identification of triggers in the environment that can promote desirable or undesirable behaviours (Choe et al., 2014). The presentation of data visualisation at the interface is an important design issue as real-time feedback of negative emotions can amplify those emotions and be counterproductive. The act of emotional regulation in situ can be cognitively demanding (Gross, 2002) hence feedback may be more constructive when delivered retrospectively at a time of the user's choosing when they have sufficient cognitive capacity to reflect upon their data.

Previous research on visualisation of emotions includes StressTree (Yu et al., 2017), which is a heart rate variability (HRV) biofeedback system that transforms HRV data into a representation of a tree that metaphorically represents the health of the user as the growth pattern is affected by the users’ stress state. For example, when the user experiences high stress for a long period of time, the appearance of the tree grows more fragile. However, during periods of healthy balance the appearance of the tree is healthier. HRV was measured via a PPG sensor attached to the index finger. Participants reported that the visualisation promoted self-reflection of their behaviours but the interface perhaps would have been more suitable for visualisation on a mobile device, like a smartphone. Similarly, Chill-Out (Parnandi et al., 2014) is an adaptive biofeedback game that assists the user in developing relaxation skills by monitoring breathing. HRV and respiration were measured using a Bioharness sensor, whilst electrodermal activity (EDA) was measured using FlexComp Infinity encoder. Results indicated that playing Chill-Out led to lower arousal, in contrast to a non-biofeedback game and traditional relaxation methods, when participants were exposed to a stressor task. Affective Health (Sanches et al., 2010) is another stress management biofeedback mobile system that utilises a triaxial accelerometer, skin conductance, and three-lead ECG sensors to capture physiological data. The system was designed to allow users to see their own bodily reactions in real time and help them to identify stressful patterns in their data. The interface for this system consisted of two designs: layers and spirals, with colour representing arousal. They argued that feedback to the users should be presented using an interface design that encompassed properties, such as: ambiguity, openness to interpretation, interactive history, fluency and aliveness. Cardiomorphologies (Muller et al., 2006) included measures of breathing (via a respiratory strain gauge) and heart rate (via electrocardiogram (ECG), which were transformed into an interactive abstract visualisation of coloured rings. Results indicated “that the relationship between the visuals and the participants’ mental and physical states leads them to a feeling of engagement” (Muller et al., 2006). A different approach was adopted by StressEraser (Moore, 2007), which is a hardware device that measured heart rate via pulse oximetry and visualised the heart rate signal as a simple wave, accompanied by an audible tone, which supported the user to develop deep breathing exercises to alleviate stress. Whilst MoodWings (MacLean et al., 2013) is a real-time biofeedback system that captures EDA and ECG data to detect stress during simulated driving. The system utilises a wearable butterfly that depicts the user's stress state through movement of the wings, which acts as an early warning system and physical interface. Results indicated that MoodWings resulted in safer driving but also significantly increased both self-perceived and biophysical stress.

The primary contribution of the current work was to create an interactive data visualisation in the lifelogging mode that provided users with feedback on “hidden” cardiovascular measures of inflammation during everyday life. The primary purpose of the data visualisation was to increase awareness of implications for long-term health that are associated with the experience of negative emotions during commuter driving. The secondary goal of the data visualisation was to function as an agent for behavioural change. Unlike most work in this area that uses real-time feedback, this visualisation of personal data delivers feedback retrospectively when participants are able to reflect on negative emotions and their precedents. Therefore, the novelty of the current work is twofold: (1) to allow participants to make an association between inflammation, health and emotional responses during commuter driving in order to gain insight into their emotional experiences, and (2) to evaluate whether this kind of interaction with personal data led to demonstrable change in subjective mood or cardiovascular physiology during subsequent episodes of commuter driving.

The work included in this paper had four main objectives. The first was to investigate whether there is an association between cardiovascular markers and the experience of negative emotion during commuter driving in the real-world. Secondly, we wished to establish whether variables captured during commuter driving, i.e. both cardiovascular physiology and vehicle parameters (e.g. speed of car), could successfully classify episodes of high and low anger. Both objectives were addressed through study one that consists of a data collection exercise of thirteen participants conducted over five consecutive days of commuter driving. The third goal of the paper was to develop an interactive data visualisation that integrated cardiovascular reactivity, driving parameters via geolocation in combination with still images. The purpose of this interactive visualisation was to provide users with sufficient insight to develop adaptive coping strategies with respect to negative emotion and cardiovascular reactivity during their daily commute. The specific parameters and measures that we adopted for the data visualisation were directed by the results of the first study, specifically the process of feature selection used for classification of low vs. high subjective anger. Given that the purpose of the interactive data visualisation was to promote adaptive coping strategies, the fourth objective was to evaluate the effects of data visualisation on subjective mood and cardiovascular psychophysiology. The second study in the paper describes a study where data were collected during two days of commuter driving (as in study one) as a pre-test period, then participants spent a period of time interacting with the data visualisation, before data were collected from two more days of commuter driving (post-test period). It was hypothesised that exposure to the data visualisation would enhance self-awareness and promote adaptive coping strategies, hence negative mood and cardiovascular reactivity will be reduced during the post-test phase.