Skip to main content
SpringerLink
Log in

The Role of Physiology and Voice in Emotion Perception During Social Stress

  • Original Paper
  • Published:
Journal of Nonverbal Behavior Aims and scope Submit manuscript

Abstract

Deciphering others’ affect is ubiquitous in daily life and is important for navigating social interactions and relationships. Research has found that behavioral components, such as facial expressions or body language, are critical channels by which people understand other people’s affect. In the current research, we examined how people’s perceptions of targets’ positive affect (PA) and negative affect (NA) are associated with targets’ physiological reactivity, and whether behavioral indices mediate these associations. A total of 94 participants (i.e., observers) watched videos of targets completing a social stress task during which targets’ physiological reactivity [i.e., changes in respiratory sinus arrhythmia (RSA), cardiac output (CO), and ventricular contractility (VC)] was assessed. We predicted (1) targets’ RSA reactivity would be negatively associated with observers’ perceptions of PA and NA (to a lesser magnitude than PA); (2) targets’ CO reactivity would be positively associated with observers’ perceptions of PA and unrelated to perceptions of NA; and (3) targets’ VC would be positively associated perceptions of PA or NA (VC was an exploratory hypothesis). Our hypotheses were largely supported. Mediational analyses revealed that vocal prosody was a significant mediator of the association between perceptions of targets’ affect and their physiological reactivity. The findings suggest that observers can reliably detect targets’ emotional experiences as they manifest at a physiological level and that voice is an especially useful marker of how people perceive others’ affective experience. The findings have implications for aspects of relationships involving emotion perception, including affect contagion and interpersonal emotion regulation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Additional measures included: Basic Empathy Scale (Jolliffe and Farrington 2006), Trait Meta Mood Scale (Salovey et al. 1995) expanded with a subset of items from the Toronto Alexithymia Scale (Bagby et al. 1994), Mood and Anxiety Symptoms Questionnaire (Watson et al. 1995), and Emotion Regulation Questionnaire (Gross and John 2003).

  2. Observers also rated additional emotions after the final minute of the TSST (i.e., Positive: proud, excited, cheerful, inspired, attentive, pleased, happy, determined, active; Negative: upset, guilty, scared, hostile, grouchy, discouraged, irritable, ashamed, afraid, sad). Observers did not rate all of these emotions after each minute because the task would have been too fatiguing. To keep the calculation of PA and NA consistent across the three points in the TSST, we calculated PA and NA using the same subset of emotions that was consistent across the points. At the first and final minute, observers also rated the targets on ten items related to their preparedness and performance on the task.

  3. Although, our question suggests the target physiology could be the outcome variable, the structure of the data does not allow for this. The observer ratings are at the lowest level of analysis (i.e., Level-1) in the nesting of our data, meaning if target physiology was used as the outcome, observer ratings would have to be collapsed across targets and time points in the TSST. This would result in valuable information being lost, which the cross-classified multilevel modeling approach maintains. Since this is a correlational analysis, the specification of the outcome and predictor in the model does not infer a causal relationship. Therefore, in our multilevel modeling analyses, we used the observer perceptions as the outcome, but in our mediation models (which were not implemented in a multilevel framework) we used the target state as the outcome.

References

  • Akinola, M., & Mendes, W. B. (2008). The dark side of creativity: Biological vulnerability and negative emotions lead to greater artistic creativity. Personality and Social Psychology Bulletin, 34, 1677–1686.

    PubMed  Google Scholar 

  • Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412.

    Google Scholar 

  • Bagby, R. M., Parker, J. D., & Taylor, G. J. (1994). The twenty-item Toronto Alexithymia Scale. I. Item selection and cross-validation of the factor structure. Journal of Psychosomatic Research, 38(1), 23–32.

    PubMed  Google Scholar 

  • Bänziger, T., Grandjean, D., & Scherer, K. R. (2009). Emotion recognition from expressions in face, voice, and body: The multimodal emotion recognition test (MERT). Emotion, 9(5), 691–704.

    PubMed  Google Scholar 

  • Barrett, L. F., & Kensinger, E. A. (2010). Context is routinely encoded during emotion perception. Psychological Science, 21(4), 595–599.

    PubMed  Google Scholar 

  • Barrett, L. F., & Russell, J. A. (1999). The structure of current affect: Controversies and emerging consensus. Current Directions in Psychological Science, 8, 10–14.

    Google Scholar 

  • Baucom, B. R., Atkins, D. C., Eldridge, K., McFarland, P., Sevier, M., & Christensen, A. (2011). The language of demand/withdraw: Verbal and vocal expression in dyadic interactions. Journal of Family Psychology, 25(4), 570–580.

    PubMed  PubMed Central  Google Scholar 

  • Bell, B. A., Ferron, J. M., & Kromrey, J. D. (2008). Cluster size in multilevel models: The impact of sparse data structures on point and interval estimates in two-level models. JSM Proceedings, Section on Survey Research Methods, pp 1122–1129.

  • Bell, B. A., Morgan, G. B., Schoeneberger, J. A., Kromrey, J. D., & Ferron, J. M. (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology, 10, 1–11.

    Google Scholar 

  • Berntson, G. G., Cacioppo, J. T., & Quigley, K. S. (1993). Respiratory sinus arrhythmia: Autonomic origins, physiological mechanisms, and psychophysiological implications. Psychophysiology, 30(2), 183–196.

    PubMed  Google Scholar 

  • Berntson, G. G., Thomas Bigger, J., Eckberg, D. L., Grossman, P., Kaufmann, P. G., Malik, M., et al. (1997). Heart rate variability: Origins, methods, and interpretive caveats. Psychophysiology, 34(6), 623–648.

    PubMed  Google Scholar 

  • Blascovich, J., & Mendes, W. B. (2010). Social psychophysiology and embodiment. In S. T. Fiske, D. T. Gilbert, & G. Lindzey (Eds.), The handbook of social psychology (5th ed., pp. 194–227). New York, NY: Wiley.

    Google Scholar 

  • Bliss-Moreau, E., Machado, C. J., & Amaral, D. G. (2013). Macaque cardiac physiology is sensitive to the valence of passively viewed sensory stimuli. PLoS ONE, 8(8), e71170.

    PubMed  PubMed Central  Google Scholar 

  • Boersma, P., & Weenink, D. (2017). Praat: doing phonetics by computer [Computer program]. Version 6.0.21, retrieved 25 July 2017 from http://www.praat.org/

  • Campbell, J., & Ehlert, U. (2012). Acute psychosocial stress: Does the emotional stress response correspond with physiological responses? Psychoneuroendocrinology, 37(8), 1111–1134.

    PubMed  Google Scholar 

  • Clore, G. L., Gasper, K., & Garvin, E. (2001). Affect as information. In J. P. Forgas (Ed.), Handbook of affect and social cognition (pp. 121–144). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Cordaro, D. T., Sun, R., Keltner, D., Kamble, S., Huddar, N., & McNeil, G. (2018). Universals and cultural variations in 22 emotional expressions across five cultures. Emotion, 18(1), 75–93.

    PubMed  Google Scholar 

  • Demaree, H. A., & Everhart, D. E. (2004). Healthy high-hostiles: Reduced parasympathetic activity and decreased sympathovagal flexibility during negative emotional processing. Personality and Individual Differences, 36(2), 457–469.

    Google Scholar 

  • Eckland, N. S., Leyro, T. M., Mendes, W. B., & Thompson, R. J. (2018). A multi-method investigation of the association between emotional clarity and empathy. Emotion, 18(5), 638–645.

    PubMed  Google Scholar 

  • Edwards, L. J., Muller, K. E., Wolfinger, R. D., Qaqish, B. F., & Schabenberger, O. (2008). An R2 statistic for fixed effects in the linear mixed model. Statistics in Medicine, 27(29), 6137–6157.

    PubMed  PubMed Central  Google Scholar 

  • Eisinga, R., Te Grotenhuis, M., & Pelzer, B. (2013). The reliability of a two-item scale: Pearson, cronbach, or spearman-brown? International Journal of Public Health, 58(4), 637–642.

    PubMed  Google Scholar 

  • Ekman, P., Freisen, W. V., & Ancoli, S. (1980). Facial signs of emotional experience. Journal of Personality and Social Psychology, 39(6), 1125–1134.

    Google Scholar 

  • Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system. Salt Lake City, UT: Network Information Research.

    Google Scholar 

  • Elfenbein, H. A., & Ambady, N. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128, 203–235.

    PubMed  Google Scholar 

  • Global Workplace Analytics & Flexjobs. (2017). The 2017 state of telecommuting in the U.S. employee workforce. Retrieved from https://www.flexjobs.com/2017-State-of-Telecommuting-US/

  • Gros, D. F., Morland, L. A., Greene, C. J., Acierno, R., Strachan, M., Egede, L. E., et al. (2013). Delivery of evidence-based psychotherapy via video telehealth. Journal of Psychopathology and Behavioral Assessment, 35(4), 506–521.

    Google Scholar 

  • Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85, 348–362.

    PubMed  Google Scholar 

  • Hagan, M. J., Bush, N., Mendes, W. B., Arenander, J., Cohodes, E., Epel, E. S., et al. (2017). Childhood adversity is related to daily coping strategies among those who are sensitive to context. Anxiety Stress and Coping, 30, 163–175.

    Google Scholar 

  • Hayes, A. F. (2013). An introduction to mediation, moderation, and conditional process analysis. New York: The Guilford Press.

    Google Scholar 

  • Herrald, M. M., & Tomaka, J. (2002). Patterns of emotion-specific appraisal, coping, and cardiovascular reactivity during an ongoing emotional episode. Journal of Personality and Social Psychology, 83(2), 434–450.

    PubMed  Google Scholar 

  • Hess, U. (2017). Body language. Encyclopedia of personality and individual differences. Berlin: Springer.

  • Hess, U., Kafetsios, K., Mauersberger, H., Blaison, C., & Kessler, C. L. (2016). Signal and noise in the perception of facial emotion expressions: From labs to life. Personality and Social Psychology Bulletin, 42(8), 1092–1110.

    PubMed  Google Scholar 

  • Human, L. J., & Mendes, W. B. (2018). Cardiac vagal flexibility and accurate personality impressions: Examining a physiological correlate of the good judge. Journal of Personality, 86, 1065–1077.

    PubMed  Google Scholar 

  • Jameson, J. P., & Blank, M. B. (2007). The role of clinical psychology in rural mental health services: Defining problems and developing solutions. Clinical Psychology: Science and Practice, 14(3), 283–298.

    Google Scholar 

  • Jefferson, A. L. (2010). Cardiac output as a potential risk factor for abnormal brain aging. Journal of Alzheimer’s Disease, 20(3), 813–821.

    PubMed  Google Scholar 

  • Jolliffe, D., & Farrington, D. P. (2006). Development and validation of the Basic Empathy Scale. Journal of Adolescence, 29(4), 589–611.

    PubMed  Google Scholar 

  • Juslin, P. N., & Scherer, K. R. (2005). Vocal expression of affect. In J. Harrigan, R. Rosenthal, & K. Scherer (Eds.), The new handbook of methods in nonverbal behaviour research (pp. 65–135). Oxford: New York, NY.

    Google Scholar 

  • Kappas, A., Hess, U., & Scherer, K. (1991). Voice and emotion. In R. S. Feldman & B. Rimé (Eds.), Fundamentals of nonverbal behavior (pp. 200–238). New York, NY: Cambridge University Press.

    Google Scholar 

  • Kassam, K. S., Koslov, K., & Mendes, W. B. (2009). Decisions under distress: Stress profiles influence anchoring and adjustment. Psychological Science, 20(11), 1394–1399.

    PubMed  Google Scholar 

  • Keltner, D., & Cordaro, D. T. (2017). Understanding multimodal emotional expressions: Recent advances in basic emotion theory. In J. Russell & J. M. Fernandez-Dols (Eds.), Facial expression. New York: Oxford University Press.

    Google Scholar 

  • Keltner, D., & Gross, J. J. (1999). Functional accounts of emotions. Cognition and Emotion, 13(5), 467–480.

    Google Scholar 

  • Keltner, D., Tracy, J., Sauter, D., Cordaro, D., & McNeil, G. (2016). Emotional expression. In L. E. Barrett, M. Lewis, & J. Haviland-Jones (Eds.), Handbook of emotions (4th ed.). New York, NY: Guilford Press.

    Google Scholar 

  • Kirschbaum, C., Pirke, K. M., & Hellhammer, D. H. (1993). The ‘Trier Social Stress Test’—A tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology, 28(1–2), 76–81.

    PubMed  Google Scholar 

  • Koslov, K., Mendes, W. B., Pajtas, P. E., & Pizzagalli, D. A. (2011). Asymmetry in resting intracortical activity as a buffer to social threat. Psychological Science, 22(5), 641–649.

    PubMed  Google Scholar 

  • Kubzansky, L. D., Mendes, W. B., Appleton, A. A., Block, J., & Adler, G. K. (2012). A heartfelt response: Oxytocin effects on response to social stress in men and women. Biological Psychology, 90(1), 1–9.

    PubMed  PubMed Central  Google Scholar 

  • Larsen, R. J., & Diener, E. (1992). Promises and problems with the circumplex model of emotion. In M. S. Clark (Ed.), Emotion: The review of personality and social psychology (Vol. 13, pp. 25–59). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Luo, W., & Kwok, O. M. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44(2), 182–212.

    PubMed  Google Scholar 

  • Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98, 224–253.

    Google Scholar 

  • Mauss, I. B., Cook, C. L., Cheng, J. Y., & Gross, J. J. (2007). Individual differences in cognitive reappraisal: Experiential and physiological responses to an anger provocation. International Journal of Psychophysiology, 66(2), 116–124.

    PubMed  Google Scholar 

  • Mendes, W. B. (2016). Emotion and the autonomic nervous system. In L. E. Barrett, M. Lewis, & J. Haviland-Jones (Eds.), Handbook of emotions (4th ed.). New York, NY: Guilford Press.

    Google Scholar 

  • Mendes, W. B., Major, B., McCoy, S., & Blascovich, J. (2008). How attributional ambiguity shapes physiological and emotional responses to social rejection and acceptance. Journal of Personality and Social Psychology, 94(2), 278–291.

    PubMed  PubMed Central  Google Scholar 

  • Meyers, J. L., & Beretvas, S. N. (2006). The impact of inappropriate modeling of cross-classified data structures. Multivariate Behavioral Research, 41(4), 473–497.

    PubMed  Google Scholar 

  • Morland, L. A., Hynes, A. K., Mackintosh, M. A., Resick, P. A., & Chard, K. M. (2011). Group cognitive processing therapy delivered to veterans via telehealth: A pilot cohort. Journal of Traumatic Stress, 24(4), 465–469.

    PubMed  Google Scholar 

  • Muhtadie, L., Koslov, K., Akinola, M., & Mendes, W. B. (2015). Vagal flexibility: A physiological predictor of social sensitivity. Journal of Personality and Social Psychology, 109(1), 106–120.

    PubMed  Google Scholar 

  • Obradović, J., Bush, N. R., Stamperdahl, J., Adler, N. E., & Boyce, W. T. (2010). Biological sensitivity to context: The interactive effects of stress reactivity and family adversity on socioemotional behavior and school readiness. Child Development, 81(1), 270–289.

    PubMed  PubMed Central  Google Scholar 

  • Porges, S. W. (2001). The polyvagal theory: Phylogenetic substrates of a social nervous system. International Journal of Psychophysiology, 42(2), 123–146.

    PubMed  Google Scholar 

  • Porges, S. W., Doussard-Roosevelt, J. A., & Maiti, A. K. (1994). Vagal tone and the physiological regulation of emotion. In N. A. Fox (Ed.), Monographs of the society for research in child development (Vol. 59, pp. 167–186). Boston: Blackwell Publishing.

    Google Scholar 

  • Porges, S. W., Doussard-Roosevelt, J. A., Portales, A. L., & Greenspan, S. I. (1996). Infant regulation of the vagal “brake” predicts child behavior problems: A psychobiological model of social behavior. Developmental Psychobiology, 29(8), 697–712.

    PubMed  Google Scholar 

  • Psychology Software Tools, Inc. [E-Prime 2.0]. (2012). Retrieved from http://www.pstnet.com.

  • Russell, J. A., Bachorowski, J. A., & Fernández-Dols, J. M. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54(1), 329–349.

    PubMed  Google Scholar 

  • Salovey, P., Mayer, J. D., Goldman, S. L., Turvey, C., & Palfai, T. P. (1995). Emotional attention, clarity, and repair: Exploring emotional intelligence using the Trait Meta-Mood Scale. In J. W. Pennebaker (Ed.), Emotion, disclosure and health (pp. 125–154). Washington, DC: American Psychological Association.

    Google Scholar 

  • Seery, M. D. (2013). The biopsychosocial model of challenge and threat: Using the heart to measure the mind. Social and Personality Psychology Compass, 7, 637–653.

    Google Scholar 

  • Smith, A. (2006). Cognitive empathy and emotional empathy in human behavior and evolution. The Psychological Record, 56(3), 3–21.

    Google Scholar 

  • Storbeck, J., & Clore, G. L. (2008). Affective arousal as information: How affective arousal influences judgments, learning, and memory. Social and Personality Psychology Compass, 2(5), 1824–1843.

    PubMed  PubMed Central  Google Scholar 

  • Thayer, J. F., Åhs, F., Fredrikson, M., Sollers, J. J., III, & Wager, T. D. (2012). A meta-analysis of heart rate variability and neuroimaging studies: Implications for heart rate variability as a marker of stress and health. Neuroscience and Biobehavioral Reviews, 36(2), 747–756.

    PubMed  Google Scholar 

  • Waters, S. F., West, T. V., & Mendes, W. B. (2014). Stress contagion: Physiological covariation between mothers and infants. Psychological Science, 25(4), 934–942.

    PubMed  Google Scholar 

  • Watson, D., Weber, K., Assenheimer, J. S., & Clark, L. A. (1995). Testing a tripartite model: I. Evaluating the convergent and discriminant validity of anxiety and depression scales. Journal of Abnormal Psychology, 104, 3–14.

    PubMed  Google Scholar 

  • Weisbuch, M., Seery, M. D., Ambady, N., & Blascovich, J. (2009). On the correspondence between physiological and nonverbal responses: Nonverbal behavior accompanying challenge and threat. Journal of Nonverbal Behavior, 33(2), 141–148.

    Google Scholar 

  • Weusthoff, S., Baucom, B. R., & Hahlweg, K. (2013). The siren song of vocal fundamental frequency for romantic relationships. Frontiers in Psychology, 4, 439.

    PubMed  PubMed Central  Google Scholar 

  • Zaki, J., Bolger, N., & Ochsner, K. (2008). It takes two the interpersonal nature of empathic accuracy. Psychological Science, 19(4), 399–404.

    PubMed  Google Scholar 

Download references

Acknowledgements

This research was supported by the Sarlo-Ekman endowment to Wendy Berry Mendes. We thank the research assistants who worked on this project through the Emotion, Health, and Psychophysiology Lab at UCSF for their help in the collection of experimental stimuli

Author information

Authors and Affiliations

  1. Department of Psychology, University of Illinois at Urbana-Champaign, 603 E. Daniel St, Champaign, IL, 61820, USA

    Nathaniel S. Eckland

  2. Department of Psychology, Rutgers University, The State University of New Jersey, New Brunswick, NJ, USA

    Teresa M. Leyro

  3. Department of Psychiatry, University of California, San Francisco, San Francisco, CA, USA

    Wendy Berry Mendes

  4. Department of Psychological and Brain Sciences, Washington University in St. Louis, St. Louis, MO, USA

    Renee J. Thompson

Authors
  1. Nathaniel S. Eckland
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Teresa M. Leyro
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wendy Berry Mendes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Renee J. Thompson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nathaniel S. Eckland.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflicts of interest.

Human and Animal Rights

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Eckland, N.S., Leyro, T.M., Mendes, W.B. et al. The Role of Physiology and Voice in Emotion Perception During Social Stress. J Nonverbal Behav 43, 493–511 (2019). https://doi.org/10.1007/s10919-019-00311-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10919-019-00311-4

Keywords

Search

Navigation