Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses☆
Graphical abstract
Introduction
Unipolar depression is a common psychological disorder and one of the leading causes of disease burden worldwide [56]; its symptoms are broad and pervasive, impacting affect, cognition, and behavior [2]. As a channel of both emotional expression and interpersonal communication, nonverbal behavior is central to how depression presents and is maintained [1], [51], [76]. Several theories of depression address nonverbal behavior directly and make predictions about what patterns should be indicative of the depressed state.
The Affective Dysregulation hypothesis [15] interprets depression in terms of valence, which captures the positivity or negativity (i.e., pleasantness or aversiveness) of an emotional state. This hypothesis argues that depression is marked by deficient positivity and excessive negativity. Deficient positivity is consistent with many neurobiological theories of depression [22], [42] and highlights the symptom of anhedonia. Excessive negativity is consistent with many cognitive theories of depression [5], [8] and highlights the symptom of rumination. In support of the Affective Dysregulation hypothesis, with few exceptions [61], observational studies have found that depression is marked by reduced positive expressions, such as smiling and laughter [7], [13], [26], [38], [39], [47], [49], [62], [64], [67], [75], [66], [78], [80], [83]. Several studies also found that depression is marked by increased negative expressions [9], [26], [61], [74]. However, other studies found the opposite effect: that depression is marked by reduced negative expressions [38], [47], [62]. These studies, combined with findings of reduced emotional reactivity using self-report and physiological measures (see [10] for a review), led to the development of an alternative hypothesis.
The Emotion Context Insensitivity hypothesis [63] interprets depression in terms of deficient appetitive motivation, which directs behavior towards exploration and the pursuit of potential rewards. This hypothesis argues that depression is marked by a reduction of both positive and negative emotion reactivity. Reduced reactivity is consistent with many evolutionary theories of depression [52], [34], [60] and highlights the symptoms of apathy and psychomotor retardation. In support of this hypothesis, some studies found that depression is marked by reductions in general facial expressiveness [26], [38], [47], [62], [69], [79] and head movement [30], [48]. However, it is unclear how much of this general reduction is accounted for by reduced positive expressions, and it is problematic for the hypothesis that negative expressions are sometimes increased in depression.
As both hypotheses predict reductions in positive expressions, their main distinguishing feature is their treatment of negative facial expressions. The lack of a clear result on negative expressions is thus vexing. Three limitations of previous studies likely contributed to these mixed results.
First, most studies compared the behavior of depressed participants with that of non-depressed controls. This type of comparison is problematic because depression is highly correlated with numerous personality traits that influence nonverbal behavior. For instance, people with psychological disorders in general are more likely to have high neuroticism (more anxious, moody, and jealous) and people with depression in particular are more likely to have low extraversion (less outgoing, talkative, and energetic) [54]. Thus, many previous studies confounded depression with the stable personality traits that are correlated with it.
Second, many studies used experimental contexts of low sociality, such as viewing emotional stimuli while alone. However, many nonverbal behaviors – especially those involved in social signaling – are typically rare in such contexts [36]. Thus, many previous studies may have introduced a “floor effect” by using contexts in which relevant expressions were unlikely to occur in any of the experimental groups.
Finally, most studies examined a limited range of nonverbal behavior. Many aggregated multiple facial expressions into single categories such as “positive expressions” and “negative expressions,” while others used single facial expressions to represent each category (e.g., smiles for positive affect and brow-lowering for negative affect). These approaches assume that all facial expressions in each category are equivalent and will be affected by depression in the same way. However, in addition to expressing a person's affective state, nonverbal behavior is also capable of regulating social interactions and communicating behavioral intentions [36], [37]. Perhaps individual nonverbal behaviors are increased or decreased in depression based on their social-communicative function. Thus, we propose an alternative hypothesis to explain the nonverbal behavior of depressed individuals, which we refer to as the Social Withdrawal hypothesis.
The Social Withdrawal hypothesis interprets depression in terms of affiliation: the motivation to cooperate, comfort, and request aid. This hypothesis argues that depression is marked by reduced affiliative behavior and increased non-affiliative behavior, which combine to maintain or increase interpersonal distance. This pattern is consistent with several evolutionary theories [1], [76] and highlights the symptom of social withdrawal. It also accounts for variability within the group of negative facial expressions, as each varies in its social-communicative function. Specifically, expressions that signal approachability should be decreased in depression and expressions that signal hostility should be increased. In support of this hypothesis, several ethological studies of depressed inpatients found that their nonverbal behavior was marked by hostility and a lack of social engagement [23], [33], [67], [79]. Findings of decreased smiling in depression also support this interpretation, as smiling is often an affiliative signal [43], [44], [53].
We address many of the limitations of previous work by: (1) following depressed participants over the course of treatment; (2) observing them during a clinical interview; and (3) measuring multiple, objectively-defined nonverbal behaviors. By comparing participants to themselves over time, we control for stable personality traits; by using a clinical interview, we provide a context that is both social and representative of typical client–patient interactions; and by examining head motion and multiple facial movements individually, we explore the possibility that depression may affect different aspects of nonverbal behavior in different ways.
We measure nonverbal behavior with both manual and automatic coding. In an effort to alleviate the substantial time burden of manual coding, interdisciplinary researchers have begun to develop automatic systems to supplement (and perhaps one day replace) human coders [84]. Initial applications to the study of depression are also beginning to appear [17], [48], [49], [59], [68], [69]. However, many of these applications are “black-box” procedures that are difficult to interpret, some require the use of invasive recording equipment, and most have not been validated against expert human coders.
We examine patterns of head motion and the occurrence of four facial expressions that have been implicated as affiliative or non-affiliative signals (Fig. 1). These expressions are defined by the Facial Action Coding System (FACS) [24] in terms of individual muscle movements called action units (AUs).
First, we examine the lip corner puller or smile expression (AU 12). As part of the prototypical expression of happiness, the smile is implicated in positive affect [25] and may signal affiliative intent [43], [44], [53]. Second, we examine the dimpler expression (AU 14). As part of the prototypical expression of contempt, the dimpler is implicated in negative affect [25] and may signal non-affiliative intent [43]. The dimpler can also counteract and obscure an underlying smile, potentially augmenting its affiliative signal and expressing embarrassment or ambivalence [26], [50]. Third, we examine the lip corner depressor expression (AU 15). As part of the prototypical expression of sadness, the lip corner depressor is implicated in negative affect [25] and may signal affiliative intent (especially the desire to show or receive empathy) [44]. Fourth, we examine the lip presser expression (AU 24). As part of the prototypical expression of anger, the lip presser is implicated in negative affect [25] and may signal non-affiliative intent [43], [44], [53]. Finally, we also examine the amplitude and velocity of head motion. [35] has argued that movement behavior should be seen as part of an effort to communicate. As such, head motion may signal affiliative intent.
We hypothesize that the depressed state will be marked by reduced amplitude and velocity of head motion, reduced activity of affiliative expressions (AUs 12 and 15), and increased activity of non-affiliative expressions (AUs 14 and 24). We also test the predictions of two established hypotheses. The Affective Dysregulation hypothesis is that depression will be marked by increased negativity (AUs 14, 15, and 24) and decreased positivity (AU 12), and the Emotion Context Insensitivity hypothesis is that the depressed state will be marked by reduced activity of all behaviors (AUs 12, 14, 15, and 24, and head motion).
Section snippets
Participants
The current study analyzed video of 33 adults from a clinical trial for treatment of depression [17]. At the time of study intake, all participants met DSM-IV [2] criteria for major depressive disorder [29]. In the clinical trial, participants were randomly assigned to receive either antidepressant medication (i.e., SSRI) or interpersonal psychotherapy (IPT); both treatments are empirically validated for use in depression [45].
Symptom severity was evaluated on up to four occasions at 1, 7, 13,
Results
Manual and automatic FACS coding demonstrated high frame-level and session-level reliability (Table 2) and revealed similar patterns between high and low severity interviews (Table 3). Manual analysis found that two AUs were reduced during high severity interviews (AUs 12 and 15) and one facial expression was increased (AU 14); automatic analysis replicated these results except that the difference with AU 12 did not reach statistical significance. Automatic head pose analysis also revealed
Discussion
Facial expression and head motion tracked changes in symptom severity. They systematically differed between times when participants were severely depressed (Hamilton score of 15 or greater) and those during which depression abated (Hamilton score of 7 or less). While some specific findings were consistent with all three alternative hypotheses, support was strongest for Social Withdrawal.
According to the Affective Dysregulation hypothesis, depression entails a deficit in the ability to
Acknowledgments
The authors wish to thank Nicole Siverling and Shawn Zuratovic for their generous assistance. This work was supported in part by the US National Institutes of Health grants MH61435, MH65376, and MH096951 to the University of Pittsburgh. Any opinions, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Institutes of Health.
References (85)
- et al.
A meta-analysis of emotional reactivity in major depressive disorder
Clin. Psychol. Rev.
(2008) An introduction to ROC analysis
Pattern Recogn. Lett.
(2006)- et al.
Reduced facial expression and social context in major depression: discrepancies between facial muscle activity and self-reported emotion
Psychiatry Res.
(2000) - et al.
Facial expression of emotions in borderline personality disorder and depression
J. Behav. Ther. Exp. Psychiatry
(2005) Social skills deficits associated with depression
Clin. Psychol. Rev.
(2000)- et al.
Subjective and expressive emotional responses in depression
J. Affect. Disord.
(1997) Evolved mechanisms in depression: the role and interaction of attachment and social rank in depression
J. Affect. Disord.
(2003)- et al.
Gender differences in depression: an ethological study of nonverbal behavior during interviews
J. Psychiatr. Res.
(1999) - et al.
The social risk hypothesis of depressed mood: evolutionary, psychosocial, and neurobiological perspectives
Psychol. Bull.
(2003) Diagnostic and Statistical Manual of Mental Disorders
(1994)
Mind Reading: The Interactive Guide to Emotion
Automatic recognition of facial actions in spontaneous expressions
J. Multimedia
Depression: Clinical, Experimental, and Theoretical Aspects
Laplacian eigenmaps for dimensionality reduction and data representation
Neural Comput.
Emotional experience and expression in schizophrenia and depression
J. Abnorm. Psychol.
Mood and memory
Am. Psychol.
Social functioning and facial emotional expression in neurological and psychiatric disorders
Appl. Neuropsychol.
Algorithms for manifold learning
LIBSVM: a library for support vector machines
ACM Trans. Intell. Syst. Technol.
Further evidence for the cultural norm hypothesis: positive emotion in depressed and control European American and Asian American women
Cult. Divers. Ethn. Minor. Psychol.
In the pursuit of effective affective computing: the relationship between features and registration
IEEE Trans. Syst. Man Cybern.
Tripartite model of anxiety and depression: psychometric evidence and taxonomic implications
J. Abnorm. Psychol.
A coefficient of agreement for nominal scales
Educ. Psychol. Meas.
Detecting depression from facial actions and vocal prosody, in: Affective Computing and Intelligent Interaction
Spontaneous facial expression in a small group can be automatically measured: an initial demonstration
Behav. Res. Methods
Active appearance models
IEEE Trans. Pattern Anal. Mach. Intell.
CSIRO Face Analysis SDK
Toward an interactional description of depression
Psychiatry: Journal for the Study of Interpersonal Processes
Neurobehavioral aspects of affective disorders
Annu. Rev. Psychol.
Ethological studies in animals and man, their use in psychiatry
Pharmacopsychiatry
Facial Action Coding System (FACS): A Technique for the Measurement of Facial Movement
Facial expression analysis
Nonverbal Communication in Depression
The effects of alcohol on the emotional displays of whites in interracial groups
Emotion
User's Guide for the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I, Version 2.0, October 1995 Final Version)
Analyzing nonverbal behavior in depression
J. Abnorm. Psychol.
Statistical Methods for Rates and Proportions
A survey of dimension reduction techniques
The ethological approach to the assessment of depressive disorders
J. Nerv. Ment. Dis.
A motivational view of psychopathology
The analysis of movement behavior during the clinical interview
Human Facial Expression: An Evolutionary View
Cited by (0)
- ☆
This paper has been recommended for acceptance by Stan Sclaroff.