Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses

https://doi.org/10.1016/j.imavis.2013.12.007Get rights and content

Highlights

  • We investigated the relation between nonverbal behavior and severity of depression.

  • When symptoms were severe, participants showed less AU 12 and AU 15 and more AU 14.

  • When symptoms were severe, participants' head motion was reduced in size and speed.

  • The pattern of findings was highly consistent for automated and manual measurements.

  • The findings support the hypothesis of nonverbal social withdrawal in depression.

Abstract

The relationship between nonverbal behavior and severity of depression was investigated by following depressed participants over the course of treatment and video recording a series of clinical interviews. Facial expressions and head pose were analyzed from video using manual and automatic systems. Both systems were highly consistent for FACS action units (AUs) and showed similar effects for change over time in depression severity. When symptom severity was high, participants made fewer affiliative facial expressions (AUs 12 and 15) and more non-affiliative facial expressions (AU 14). Participants also exhibited diminished head motion (i.e., amplitude and velocity) when symptom severity was high. These results are consistent with the Social Withdrawal hypothesis: that depressed individuals use nonverbal behavior to maintain or increase interpersonal distance. As individuals recover, they send more signals indicating a willingness to affiliate. The finding that automatic facial expression analysis was both consistent with manual coding and revealed the same pattern of findings suggests that automatic facial expression analysis may be ready to relieve the burden of manual coding in behavioral and clinical science.

Introduction

Unipolar depression is a common psychological disorder and one of the leading causes of disease burden worldwide [56]; its symptoms are broad and pervasive, impacting affect, cognition, and behavior [2]. As a channel of both emotional expression and interpersonal communication, nonverbal behavior is central to how depression presents and is maintained [1], [51], [76]. Several theories of depression address nonverbal behavior directly and make predictions about what patterns should be indicative of the depressed state.

The Affective Dysregulation hypothesis [15] interprets depression in terms of valence, which captures the positivity or negativity (i.e., pleasantness or aversiveness) of an emotional state. This hypothesis argues that depression is marked by deficient positivity and excessive negativity. Deficient positivity is consistent with many neurobiological theories of depression [22], [42] and highlights the symptom of anhedonia. Excessive negativity is consistent with many cognitive theories of depression [5], [8] and highlights the symptom of rumination. In support of the Affective Dysregulation hypothesis, with few exceptions [61], observational studies have found that depression is marked by reduced positive expressions, such as smiling and laughter [7], [13], [26], [38], [39], [47], [49], [62], [64], [67], [75], [66], [78], [80], [83]. Several studies also found that depression is marked by increased negative expressions [9], [26], [61], [74]. However, other studies found the opposite effect: that depression is marked by reduced negative expressions [38], [47], [62]. These studies, combined with findings of reduced emotional reactivity using self-report and physiological measures (see [10] for a review), led to the development of an alternative hypothesis.

The Emotion Context Insensitivity hypothesis [63] interprets depression in terms of deficient appetitive motivation, which directs behavior towards exploration and the pursuit of potential rewards. This hypothesis argues that depression is marked by a reduction of both positive and negative emotion reactivity. Reduced reactivity is consistent with many evolutionary theories of depression [52], [34], [60] and highlights the symptoms of apathy and psychomotor retardation. In support of this hypothesis, some studies found that depression is marked by reductions in general facial expressiveness [26], [38], [47], [62], [69], [79] and head movement [30], [48]. However, it is unclear how much of this general reduction is accounted for by reduced positive expressions, and it is problematic for the hypothesis that negative expressions are sometimes increased in depression.

As both hypotheses predict reductions in positive expressions, their main distinguishing feature is their treatment of negative facial expressions. The lack of a clear result on negative expressions is thus vexing. Three limitations of previous studies likely contributed to these mixed results.

First, most studies compared the behavior of depressed participants with that of non-depressed controls. This type of comparison is problematic because depression is highly correlated with numerous personality traits that influence nonverbal behavior. For instance, people with psychological disorders in general are more likely to have high neuroticism (more anxious, moody, and jealous) and people with depression in particular are more likely to have low extraversion (less outgoing, talkative, and energetic) [54]. Thus, many previous studies confounded depression with the stable personality traits that are correlated with it.

Second, many studies used experimental contexts of low sociality, such as viewing emotional stimuli while alone. However, many nonverbal behaviors – especially those involved in social signaling – are typically rare in such contexts [36]. Thus, many previous studies may have introduced a “floor effect” by using contexts in which relevant expressions were unlikely to occur in any of the experimental groups.

Finally, most studies examined a limited range of nonverbal behavior. Many aggregated multiple facial expressions into single categories such as “positive expressions” and “negative expressions,” while others used single facial expressions to represent each category (e.g., smiles for positive affect and brow-lowering for negative affect). These approaches assume that all facial expressions in each category are equivalent and will be affected by depression in the same way. However, in addition to expressing a person's affective state, nonverbal behavior is also capable of regulating social interactions and communicating behavioral intentions [36], [37]. Perhaps individual nonverbal behaviors are increased or decreased in depression based on their social-communicative function. Thus, we propose an alternative hypothesis to explain the nonverbal behavior of depressed individuals, which we refer to as the Social Withdrawal hypothesis.

The Social Withdrawal hypothesis interprets depression in terms of affiliation: the motivation to cooperate, comfort, and request aid. This hypothesis argues that depression is marked by reduced affiliative behavior and increased non-affiliative behavior, which combine to maintain or increase interpersonal distance. This pattern is consistent with several evolutionary theories [1], [76] and highlights the symptom of social withdrawal. It also accounts for variability within the group of negative facial expressions, as each varies in its social-communicative function. Specifically, expressions that signal approachability should be decreased in depression and expressions that signal hostility should be increased. In support of this hypothesis, several ethological studies of depressed inpatients found that their nonverbal behavior was marked by hostility and a lack of social engagement [23], [33], [67], [79]. Findings of decreased smiling in depression also support this interpretation, as smiling is often an affiliative signal [43], [44], [53].

We address many of the limitations of previous work by: (1) following depressed participants over the course of treatment; (2) observing them during a clinical interview; and (3) measuring multiple, objectively-defined nonverbal behaviors. By comparing participants to themselves over time, we control for stable personality traits; by using a clinical interview, we provide a context that is both social and representative of typical client–patient interactions; and by examining head motion and multiple facial movements individually, we explore the possibility that depression may affect different aspects of nonverbal behavior in different ways.

We measure nonverbal behavior with both manual and automatic coding. In an effort to alleviate the substantial time burden of manual coding, interdisciplinary researchers have begun to develop automatic systems to supplement (and perhaps one day replace) human coders [84]. Initial applications to the study of depression are also beginning to appear [17], [48], [49], [59], [68], [69]. However, many of these applications are “black-box” procedures that are difficult to interpret, some require the use of invasive recording equipment, and most have not been validated against expert human coders.

We examine patterns of head motion and the occurrence of four facial expressions that have been implicated as affiliative or non-affiliative signals (Fig. 1). These expressions are defined by the Facial Action Coding System (FACS) [24] in terms of individual muscle movements called action units (AUs).

First, we examine the lip corner puller or smile expression (AU 12). As part of the prototypical expression of happiness, the smile is implicated in positive affect [25] and may signal affiliative intent [43], [44], [53]. Second, we examine the dimpler expression (AU 14). As part of the prototypical expression of contempt, the dimpler is implicated in negative affect [25] and may signal non-affiliative intent [43]. The dimpler can also counteract and obscure an underlying smile, potentially augmenting its affiliative signal and expressing embarrassment or ambivalence [26], [50]. Third, we examine the lip corner depressor expression (AU 15). As part of the prototypical expression of sadness, the lip corner depressor is implicated in negative affect [25] and may signal affiliative intent (especially the desire to show or receive empathy) [44]. Fourth, we examine the lip presser expression (AU 24). As part of the prototypical expression of anger, the lip presser is implicated in negative affect [25] and may signal non-affiliative intent [43], [44], [53]. Finally, we also examine the amplitude and velocity of head motion. [35] has argued that movement behavior should be seen as part of an effort to communicate. As such, head motion may signal affiliative intent.

We hypothesize that the depressed state will be marked by reduced amplitude and velocity of head motion, reduced activity of affiliative expressions (AUs 12 and 15), and increased activity of non-affiliative expressions (AUs 14 and 24). We also test the predictions of two established hypotheses. The Affective Dysregulation hypothesis is that depression will be marked by increased negativity (AUs 14, 15, and 24) and decreased positivity (AU 12), and the Emotion Context Insensitivity hypothesis is that the depressed state will be marked by reduced activity of all behaviors (AUs 12, 14, 15, and 24, and head motion).

Section snippets

Participants

The current study analyzed video of 33 adults from a clinical trial for treatment of depression [17]. At the time of study intake, all participants met DSM-IV [2] criteria for major depressive disorder [29]. In the clinical trial, participants were randomly assigned to receive either antidepressant medication (i.e., SSRI) or interpersonal psychotherapy (IPT); both treatments are empirically validated for use in depression [45].

Symptom severity was evaluated on up to four occasions at 1, 7, 13,

Results

Manual and automatic FACS coding demonstrated high frame-level and session-level reliability (Table 2) and revealed similar patterns between high and low severity interviews (Table 3). Manual analysis found that two AUs were reduced during high severity interviews (AUs 12 and 15) and one facial expression was increased (AU 14); automatic analysis replicated these results except that the difference with AU 12 did not reach statistical significance. Automatic head pose analysis also revealed

Discussion

Facial expression and head motion tracked changes in symptom severity. They systematically differed between times when participants were severely depressed (Hamilton score of 15 or greater) and those during which depression abated (Hamilton score of 7 or less). While some specific findings were consistent with all three alternative hypotheses, support was strongest for Social Withdrawal.

According to the Affective Dysregulation hypothesis, depression entails a deficit in the ability to

Acknowledgments

The authors wish to thank Nicole Siverling and Shawn Zuratovic for their generous assistance. This work was supported in part by the US National Institutes of Health grants MH61435, MH65376, and MH096951 to the University of Pittsburgh. Any opinions, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Institutes of Health.

References (85)

  • S. Baron-Cohen

    Mind Reading: The Interactive Guide to Emotion

    (2003)
  • M.S. Bartlett et al.

    Automatic recognition of facial actions in spontaneous expressions

    J. Multimedia

    (2006)
  • A.T. Beck

    Depression: Clinical, Experimental, and Theoretical Aspects

    (1967)
  • M. Belkin et al.

    Laplacian eigenmaps for dimensionality reduction and data representation

    Neural Comput.

    (2003)
  • H. Berenbaum et al.

    Emotional experience and expression in schizophrenia and depression

    J. Abnorm. Psychol.

    (1992)
  • G.H. Bower

    Mood and memory

    Am. Psychol.

    (1981)
  • A.Z. Brozgold et al.

    Social functioning and facial emotional expression in neurological and psychiatric disorders

    Appl. Neuropsychol.

    (1998)
  • L. Cayton

    Algorithms for manifold learning

  • C.C. Chang et al.

    LIBSVM: a library for support vector machines

    ACM Trans. Intell. Syst. Technol.

    (2011)
  • Y.E. Chentsova-Dutton et al.

    Further evidence for the cultural norm hypothesis: positive emotion in depressed and control European American and Asian American women

    Cult. Divers. Ethn. Minor. Psychol.

    (2010)
  • S.W. Chew et al.

    In the pursuit of effective affective computing: the relationship between features and registration

    IEEE Trans. Syst. Man Cybern.

    (2012)
  • L.A. Clark et al.

    Tripartite model of anxiety and depression: psychometric evidence and taxonomic implications

    J. Abnorm. Psychol.

    (1991)
  • J. Cohen

    A coefficient of agreement for nominal scales

    Educ. Psychol. Meas.

    (1960)
  • J.F. Cohn et al.

    Detecting depression from facial actions and vocal prosody, in: Affective Computing and Intelligent Interaction

    (2009)
  • J.F. Cohn et al.

    Spontaneous facial expression in a small group can be automatically measured: an initial demonstration

    Behav. Res. Methods

    (2010)
  • T.F. Cootes et al.

    Active appearance models

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2001)
  • M. Cox et al.

    CSIRO Face Analysis SDK

    (2013)
  • J.C. Coyne

    Toward an interactional description of depression

    Psychiatry: Journal for the Study of Interpersonal Processes

    (1976)
  • R. Depue et al.

    Neurobehavioral aspects of affective disorders

    Annu. Rev. Psychol.

    (1989)
  • A.K. Dixon et al.

    Ethological studies in animals and man, their use in psychiatry

    Pharmacopsychiatry

    (1989)
  • P. Ekman et al.

    Facial Action Coding System (FACS): A Technique for the Measurement of Facial Movement

    (2002)
  • P. Ekman et al.

    Facial expression analysis

    (2008)
  • H. Ellgring

    Nonverbal Communication in Depression

    (1989)
  • C.E. Fairbairn et al.

    The effects of alcohol on the emotional displays of whites in interracial groups

    Emotion

    (2013)
  • M.B. First et al.

    User's Guide for the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I, Version 2.0, October 1995 Final Version)

    (1995)
  • H.U. Fisch et al.

    Analyzing nonverbal behavior in depression

    J. Abnorm. Psychol.

    (1983)
  • J.L. Fleiss

    Statistical Methods for Rates and Proportions

    (1981)
  • I.K. Fodor

    A survey of dimension reduction techniques

  • L. Fossi et al.

    The ethological approach to the assessment of depressive disorders

    J. Nerv. Ment. Dis.

    (1984)
  • D.C. Fowles

    A motivational view of psychopathology

  • N. Freedman

    The analysis of movement behavior during the clinical interview

  • A.J. Fridlund

    Human Facial Expression: An Evolutionary View

    (1994)
  • Cited by (0)

    This paper has been recommended for acceptance by Stan Sclaroff.

    View full text