Skip to main content
SpringerLink
Log in

Training public speaking with virtual social interactions: effectiveness of real-time feedback and delayed feedback

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Social signal processing and virtual social interaction technologies have allowed the creation of social skills training applications, and initial studies have shown that such solutions can lead to positive training outcomes and could complement traditional teaching methods by providing cheap, accessible, safe tools for training social skills. However, these studies evaluated social skills training systems as a whole and it is unclear to what extent which components contributed to positive outcomes. In this paper, we describe an experimental study where we compared the relative efficacy of real-time interactive feedback and after-action feedback in the context of a public speaking training application. We observed that both components provide benefits to the overall training: the real-time interactive feedback made the experience more immersive and improved participants’ motivation in using the system, while the after-action feedback led to positive training outcomes when it contained personalized feedback elements. Taken in combination, these results confirm that both social signal processing technologies and virtual social interactions are both contributing to social skills training systems’ efficiency. Additionally, we observed that several individual factors, here the subjects’ initial level of public speaking anxiety, personality and tendency to immersion significantly influenced the training experience. This finding suggests that social skills training systems could benefit from being tailored to participants’ particular individual circumstances.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. https://www.ibm.com/watson/services/speech-to-text/.

References

  1. Ali MR, Crasta D, Jin L, Baretto A, Pachter J, Rogge RD, Hoque ME (2015) LISSA—live interactive social skill assistance. In: 2015 international conference on affective computing and intelligent interaction (ACII). IEEE, pp 173–179

  2. Anderson K, André E, Baur T, Bernardini S, Chollet M, Chryssafidou E, Damian I, Ennis C, Egges A, Gebhard P, et al (2013) The tardis framework: intelligent virtual agents for social coaching in job interviews. In: International conference on advances in computer entertainment technology. Springer, pp 476–491

  3. Baltrusaitis T, Zadeh A, Lim YC, Morency LP (2018) Openface 2.0: facial behavior analysis toolkit. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp 59–66. IEEE

  4. Bandura A (1997) Self-efficacy: The exercise of control. Macmillan

  5. Barmaki R, Hughes CE (2018) Embodiment analytics of practicing teachers in a virtual immersive environment. J Comput Assist Learn

  6. Batrinca L, Stratou G, Shapiro A, Morency LP, Scherer S (2013) Cicero—towards a multimodal virtual audience platform for public speaking training. In: Intelligent virtual agents, pp 116–128

  7. Bissonnette J, Dubé F, Provencher MD, Moreno Sala MT (2016) Evolution of music performance anxiety and quality of performance during virtual reality exposure training. Virtual Real 20(1):71–81. https://doi.org/10.1007/s10055-016-0283-y

    Article  Google Scholar 

  8. Blascovich J, Mendes W (2000) Challenge and threat appraisals: the role of affective cues. In: Forgas J (ed) Feeling and thinking: the role of affect in social cognition, pp 59–82

  9. Bower M, Cavanagh M, Moloney R, Dao M (2011) Developing communication competence using an online video reflection system: pre-service teachers’ experiences. Asia-Pac J Teach Educ 39(4):311–326

    Article  Google Scholar 

  10. Brown T, Morrissey L (2004) The effectiveness of verbal self-guidance as a transfer of training intervention: its impact on presentation performance, self efficacy and anxiety. Innov Educ Teach Int 41(3):255–271

    Article  Google Scholar 

  11. Casler K, Bickel L, Hackett E (2013) Separate but equal? a comparison of participants and data gathered via amazons mturk, social media, and face-to-face behavioral testing. Comput Hum Behav 29(6):2156–2160

    Article  Google Scholar 

  12. Chan V (2011) Teaching oral communication in undergraduate science: are we doing enough and doing it right? J Learn Des 4(3):71–79

    Google Scholar 

  13. Chollet M, Chandrashekhar N, Shapiro A, Morency LP, Scherer S (2016) Manipulating the perception of virtual audiences using crowdsourced behaviors. In: International conference on intelligent virtual agents. Springer, pp 164–174

  14. Chollet M, Prendinger H, Scherer S (2016) Native vs. non-native language fluency implications on multimodal interaction for interpersonal skills training. In: Proceedings of the 18th ACM international conference on multimodal interaction. ACM, pp 386–393

  15. Chollet M, Scherer S (2017) Assessing public speaking ability from thin slices of behavior. In: Proceedings of the IEEE international conference on face and gesture recognition (to appear)

  16. Chollet M, Scherer S (2017) Perception of virtual audiences. IEEE Comput Graph Appl 37(4):50–59

    Article  Google Scholar 

  17. Chollet M, Wörtwein T, Morency LP, Shapiro A, Scherer S (2015) Exploring feedback strategies to improve public speaking: an interactive virtual audience framework. In: Proceedings of UbiComp 2015. ACM, Osaka, Japan

  18. Cicchetti DV (1994) Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 6(4):284

    Article  Google Scholar 

  19. Damian I, Baur T, Lugrin B, Gebhard P, Mehlmann G, André E (2015) Games are better than books: in-situ comparison of an interactive job interview game with conventional training. In: International conference on artificial intelligence in education. Springer, pp 84–94

  20. Damian I, Tan CSS, Baur T, Schöning J, Luyten K, André E (2015) Augmenting social interactions: realtime behavioural feedback using social signal processing techniques. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ’15. ACM, New York, NY, USA, pp 565–574. https://doi.org/10.1145/2702123.2702314

  21. De Grez L, Valcke M, Roozen I (2009) The impact of goal orientation, self-reflection and personal characteristics on the acquisition of oral presentation skills. Eur J Psychol Educ 24(3):293

    Article  Google Scholar 

  22. Feher A, Vernon PA (2020) Looking beyond the big five: a selective review of alternatives to the big five model of personality. Personality and Individual Differences p 110002

  23. Flanagan JR, Johansson RS (2003) Action plans used in action observation. Nature 424(6950):769–771

    Article  Google Scholar 

  24. Fung M, Jin Y, Zhao R, Hoque ME (2015) Roc speak: semi-automated personalized feedback on nonverbal behavior from recorded videos. In: Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp 1167–1178

  25. Furmark T, Tillfors M, Everz PO, Marteinsdottir I, Gefvert O, Fredrikson M (1999) Social phobia in the general population: prevalence and sociodemographic profile. Soc Psychiatry Psychiatr Epidemiol 34(8):416–424

    Article  Google Scholar 

  26. van Ginkel S, Gulikers J, Biemans H, Mulder M (2015) Towards a set of design principles for developing oral presentation competence: a synthesis of research in higher education. Educ Res Rev 14:62–80

    Article  Google Scholar 

  27. van Ginkel S, Gulikers J, Biemans H, Noroozi O, Roozen M, Bos T, van Tilborg R, van Halteren M, Mulder M (2019) Fostering oral presentation competence through a virtual reality-based task for delivering feedback. Comput Educ 134:78–97

    Article  Google Scholar 

  28. Hallgren KA (2012) Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials Quant Methods Psychol 8(1):23

    Article  Google Scholar 

  29. Harris SR, Kemmerling RL, North MM (2002) Brief virtual reality therapy for public speaking anxiety. Cyberpsychol Behav 5:543–550

    Article  Google Scholar 

  30. Hassenzahl M, Burmester M, Koller F (2003) Attrakdiff: a questionnaire to measure perceived hedonic and pragmatic quality. In: Mensch & computer, pp 187–196

  31. Hofmann SG, DiBartolo PM (2000) An instrument to assess self-statements during public speaking: scale development and preliminary psychometric properties. Behav Therapy 31(3):499–515

    Article  Google Scholar 

  32. Hoque ME, Courgeon M, Martin JC, Mutlu B, Picard RW (2013) Mach: My automated conversation coach. In: Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, pp 697–706

  33. Hoque ME, Picard RW (2014) Rich nonverbal sensing technology for automated social skills training. Computer 47(4):28–35

    Article  Google Scholar 

  34. Jennett C, Cox AL, Cairns P, Dhoparee S, Epps A, Tijs T, Walton A (2008) Measuring and defining the experience of immersion in games. Int J Hum Comput Stud 66(9):641–661

    Article  Google Scholar 

  35. Kerby D, Romine J (2009) Develop oral presentation skills through accounting curriculum design and course-embedded assessment. J Educ Bus 85(3):172–179

    Article  Google Scholar 

  36. Liebowitz M (1987) Liebowitz social anxiety scale. Mod Probl Pharmacopsychiatry 22:141–173

    Article  Google Scholar 

  37. Ling Y, Nefs HT, Brinkman WP, Qu C, Heynderickx I (2013) The relationship between individual characteristics and experienced presence. Comput Hum Behav 29(4):1519–1530. https://doi.org/10.1016/j.chb.2012.12.010

    Article  Google Scholar 

  38. Lombard M, Ditton TB, Weinstein L (2009) Measuring presence: the temple presence inventory. In: Proceedings of the 12th annual international workshop on presence, pp. 1–15

  39. McCroskey JC (1978) Validity of the prca as an index of oral communication apprehension. Commun Monogr 45(3):192–203

    Article  Google Scholar 

  40. Mihoub A, Lefebvre G (2019) Wearables and social signal processing for smarter public presentations. ACM Trans Interact Intell Syst (TiiS) 9(2–3):9

    Google Scholar 

  41. Paul G (1966) Insight vs. desensitization in psychotherapy: an experiment in anxiety reduction. Stanford University Press

  42. Pertaub DP, Slater M, Barker C (2002) An experiment on public speaking anxiety in response to three different types of virtual audience. Presence Teleoperators Virtual Environ 11(1):68–78. https://doi.org/10.1162/105474602317343668

    Article  Google Scholar 

  43. Petukhova V, Mayer T, Malchanau A, Bunt H (2017) Virtual debate coach design: assessing multimodal argumentation performance. In: Proceedings of the 19th ACM international conference on multimodal interaction, ICMI 2017. ACM, New York, NY, USA, pp 41–50. https://doi.org/10.1145/3136755.3136775

  44. Potdevin D, Sabouret N, Clavel C (2020) Intimacy perception: does the artificial or human nature of the interlocutor matter? Int J Hum Comput Stud 142:102464. https://doi.org/10.1016/j.ijhcs.2020.102464

    Article  Google Scholar 

  45. Price M, Anderson PL (2012) Outcome expectancy as a predictor of treatment response in cognitive behavioral therapy for public speaking fears within social anxiety disorder. Psychotherapy 49(2):173

    Article  Google Scholar 

  46. Ramanarayanan V, Leong CW, Chen L, Feng G, Suendermann-Oeft D (2015)Evaluating speech, face, emotion and body movement time-series features for automated multimodal presentation scoring. In: Proceedings of the ACM international conference on multimodal interaction, ICMI ’15. ACM, New York, NY, USA, pp 23–30. https://doi.org/10.1145/2818346.2820765

  47. Rammstedt B, John O (2007) Measuring personality in one minute or less: a 10-item short version of the big five inventory in English and German. J Res Person 41(1):203–212

    Article  Google Scholar 

  48. Safir MP, Wallach HS, Bar-Zvi M (2012) Virtual reality cognitive-behavior therapy for public speaking anxiety: one-year follow-up. Behav Modif 36(2):235–246

    Article  Google Scholar 

  49. Scherer S, Marsella S, Stratou G, Xu Y, Morbini F, Egan A, Morency LP et al (2012)Perception markup language: Towards a standardized representation of perceived nonverbal behaviors. In: International conference on intelligent virtual agents. Springer, pp 455–463

  50. Schneider J, Börner D, van Rosmalen P, Specht M (2015) Presentation trainer, your public speaking multimodal coach. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, ICMI ’15. ACM, New York, NY, USA, pp 539–546. https://doi.org/10.1145/2818346.2830603

  51. Slater M (1999) Measuring presence: a response to the Witmer and singer presence questionnaire. Presence 8(5):560–565

    Article  Google Scholar 

  52. Tanveer MI, Lin E, Hoque ME (2015) Rhema: a real-time in-situ intelligent interface to help people with public speaking. In: Proceedings of the 20th international conference on intelligent user interfaces. ACM, pp 286–295

  53. Tanveer MI, Zhao R, Chen K, Tiet Z, Hoque ME (2016) Automanner: An automated interface for making public speakers aware of their mannerisms. In: Proceedings of the 21st International Conference on Intelligent User Interfaces, pp. 385–396. ACM

  54. Vinciarelli A, Pantic M, Bourlard H (2009) Social signal processing: survey of an emerging domain. Image Vis Comput 27(12):1743–1759

    Article  Google Scholar 

  55. Wagner J, Lingenfelser F, Baur T, Damian I, Kistler F, André E (2013) The social signal interpretation (ssi) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM international conference on Multimedia. ACM, pp 831–834

  56. Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence 7(3):225–240

    Article  Google Scholar 

  57. Wörtwein T, Chollet M, Schauerte B, Morency LP, Stiefelhagen R, Scherer S (2015) Multimodal public speaking performance assessment. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 43–50

  58. Zhao R, Li V, Barbosa H, Ghoshal G, Hoque ME (2017) Semi-automated 8 collaborative online training module for improving communication skills. In: Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, vol 1(2), p 32

Download references

Author information

Authors and Affiliations

  1. IMT Atlantique, LS2N, UMR 6004, 4 rue Alfred Kastler, 44307, Nantes, France

    Mathieu Chollet

  2. University of Neuroscience and Psychology, University of Glasgow, 62 Hillhead St, Glasgow, G12 8AD, UK

    Stacy Marsella

  3. Institute for Creative Technologies, University of Southern California, 12015 E Waterfront Dr, Playa Vista, CA, 90094, USA

    Stefan Scherer

Authors
  1. Mathieu Chollet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stacy Marsella
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Scherer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mathieu Chollet.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This material is based upon work supported by the U.S. Army Research Laboratory under contract number W911NF-14-D-0005. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Government, and no official endorsement should be inferred.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chollet, M., Marsella, S. & Scherer, S. Training public speaking with virtual social interactions: effectiveness of real-time feedback and delayed feedback. J Multimodal User Interfaces 16, 17–29 (2022). https://doi.org/10.1007/s12193-021-00371-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-021-00371-1

Keywords

Search

Navigation