Skip to main content
SpringerLink
Log in

Effects of adapting to user pitch on rapport perception, behavior, and state with a social robotic learning companion

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Social robots such as learning companions, therapeutic assistants, and tour guides are dependent on the challenging task of establishing a rapport with their users. People rarely communicate with just words alone; facial expressions, gaze, gesture, and prosodic cues like tone of voice and speaking rate combine to help individuals express their words and convey emotion. One way that individuals communicate a sense of connection with one another is entrainment, where interaction partners adapt their way of speaking, facial expressions, or gestures to each other; entrainment has been linked to trust, liking, and task success and is thought to be a vital phenomenon in how people build rapport. In this work, we introduce a social robot that combines multiple channels of rapport-building behavior, including forms of social dialog and prosodic entrainment. We explore how social dialog and entrainment contribute to both self-reported and behavioral rapport responses. We find prosodic adaptation enhances perceptions of social dialog, and that social dialog and entrainment combined build rapport. Individual differences indicated by gender mediate these social responses; an individual’s underlying rapport state, as indicated by their verbal rapport behavior, is exhibited and triggered differently depending on gender. These results have important repercussions for assessing and modeling a user’s social responses and designing adaptive social agents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • ALICE: A.L.I.C.E AI Foundation (2002). http://www.alicebot.org/. Accessed 21 Apr 2015

  • Aneja, D., Hoegen, R., McDuff, D., Czerwinski, M.: Designing style matching conversational agents. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM (2019)

  • Arroyo, I., Burleson, W., Tai, M., Muldner, K., Woolf, B.P.: Gender differences in the use and benefit of advanced learning technologies for mathematics. J. Educ. Psychol. 105(4), 957–969 (2013). https://doi.org/10.1037/a0032748

    Article  Google Scholar 

  • Babel, M.: Dialect divergence and convergence in New Zealand English. Lang. Soc. 39, 437–456 (2010)

    Article  Google Scholar 

  • Babel, M.: Evidence for phonetic and social selectivity in spontaneous phonetic imitation. J. Phon. 40, 177–189 (2011)

    Article  Google Scholar 

  • Babel, M., Bulatov, D.: The role of fundamental frequency in phonetic accommodation. Lang. Speech 55(2), 231–248 (2012)

    Article  Google Scholar 

  • Bales, R.F.: A set of categories for the analysis of small group interaction. Am. Sociol. Rev. 15(2), 257–263 (1950)

    Article  Google Scholar 

  • Beal, C., Mitra, S., Cohen, P.: Modeling learning patterns of students with a tutoring system using Hidden Markov Model. In: Luckin, R., et al. (eds.) Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED), pp. 238–245. Marina del Rey (2007)

  • Bechade, L., Dubuisson Duplessis, G., Sehili, M., Devillers, L.: Behavioral and emotional spoken cues related to mental states in human–robot social interaction. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, November 2015, pp. 347–350. ACM (2015)

  • Bell, L., Gustafson, J., Heldner, M.: Prosodic adaptation in human–computer interaction. In: Proceedings of ICPHS, vol. 3 (2003)

  • Bell, D., Arnold, H., Haddock, R.: Linguistic politeness and peer tutoring. Learn. Assist. Rev. 14(1), 37–54 (2009)

    Google Scholar 

  • Bengio, Y., Frasconi, P.: An input output HMM architecture. In: Advances in Neural Information Processing Systems, pp. 427–434 (1995)

  • Beňuš, Š., Levitan, R., Hirschberg, J., Gravano, A., Darjaa, S.: Entrainment in Slovak collaborative dialogues. In: 2014 5th IEEE Conference on Cognitive Infocommunications (CogInfoCom), November 2014, pp. 309–313. IEEE (2014)

  • Beňuš, Š., Levitan, R., Hirschberg, J.: Entrainment in spontaneous speech: the case of filled pauses in Supreme Court hearings. In: 3rd IEEE Conference on Cognitive Infocommunications, Kosice, Slovakia (2012)

  • Beňuš, Š.: Are we in sync’: turn-taking in collaborative dialogues. In: Tenth Annual Conference of the International Speech Communication Association (2009)

  • Bergner, Y., Walker, E., Ogan, A.: Dynamic Bayesian network models for peer tutoring interactions. In: Innovative Assessment of Collaboration, pp. 249–268. Springer, Cham (2017)

  • Bickmore, T.W.: Relational Agents: Effecting Change Through Human-Computer Relationships (Doctoral dissertation, Massachusetts Institute of Technology) (2003)

  • Bickmore, T., Cassell, J.: Relational agents: a model and implementation of building user trust. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 396–403 (2001)

  • Bickmore, T.W., Picard, R.W.: Establishing and maintaining long-term human-computer relationships. ACM Trans. Comput.-Human. Interact. (TOCHI) 12(2), 293–327 (2005)

    Article  Google Scholar 

  • Bickmore, T.W., Vardoulakis, L.M.P., Schulman, D.: Tinker: a relational agent museum guide. Auton. Agent. Multi-Agent Syst. 27(2), 254–276 (2013)

    Article  Google Scholar 

  • Biocca, F., Harms, C.: Defining and measuring social presence: contribution to the networked minds theory and measure. In: Proceedings of PRESENCE, 2002, pp. 1–36 (2002)

  • Boersma, P.: Praat: doing phonetics by computer. http://www.praat.org/ (2006). Accessed 31 Mar 2014

  • Bone, D., Lee, C.C., Chaspari, T., Black, M.P., Williams, M.E., Lee, S., Levitt, P., Narayanan, S.: Acoustic-prosodic, turn-taking, and language cues in child–psychologist interactions for varying social demand. In: INTERSPEECH-2013, pp. 2400–2404 (2013)

  • Bonin, F., De Looze, C., Ghosh, S., Gilmartin, E., Vogel, C., Polychroniou, A., Salamin, H., Vinciarelli, A., Campbell, N.: Investigating fine temporal dynamics of prosodic and lexical accommodation. In: INTERSPEECH-2013, pp. 539–543 (2013)

  • Borrie, S.A., Lubold, N., Pon-Barry, H.: Disordered speech disrupts conversational entrainment: a study of acoustic-prosodic entrainment and communicative success in populations with communication challenges. Front. Psychol. 6, 1187 (2015)

    Article  Google Scholar 

  • Boyer, K.E., Phillips, R., Ingram, A., Ha, E.Y., Wallis, M., Vouk, M., Lester, J.: Characterizing the effectiveness of tutorial dialogue with hidden markov models. In: International Conference on Intelligent Tutoring Systems, June 2010, pp. 55–64. Springer, Berlin, Heidelberg (2010)

  • Breazeal, C.: Emotion and sociable humanoid robots. Int. J. Human-Comput. Stud. 59(1–2), 119–155 (2003)

    Article  Google Scholar 

  • Brown, P.: How and why are women more polite: some evidence from a Mayan community. In: McConnell-Ginet, S., Borker, R., Furman, N. (eds.) Women and Language in Literature and Society, pp. 111–136. Praeger, New York (1980)

    Google Scholar 

  • Brown, P., Levinson, S.: Politeness. Some Universals in Language Usage. CUP, Cambridge (1987). Originally published as Universals in language usage: politeness phenomenon. In: Goody, E. (ed.) Questions and Politeness: Strategies in Social Interaction. CUP, Cambridge (1978)

    Chapter  Google Scholar 

  • Burleson, W., Picard, R.W.: Gender-specific approaches to developing emotionally intelligent learning companions. IEEE Intell. Syst. 22(4), 62–69 (2007). https://doi.org/10.1109/MIS.2007.69

    Article  Google Scholar 

  • Cassell, J., Bickmore, T.: Negotiated collusion: modeling social language and its relationship effects in intelligent agents. User Model. User-Adap. Inter. 13(1–2), 89–132 (2003)

    Article  Google Scholar 

  • Chalupnik, M., Christie, C., Mullany, L.: (Im)politeness and gender. In: Culpeper, J., Haugh, M., Kádár, D. (eds.) The Palgrave Handbook of Linguistic (Im)politeness, pp. 517–537. Palgrave Macmillan, London (2017)

    Chapter  Google Scholar 

  • Chidambaram, V., Chiang, Y.-H., Mutlu, B.: Designing persuasive robots: how robots might persuade people using vocal and nonverbal cues. In: 7th Annual ACM/IEEE International Conference on Human–Robot Interaction (HRI’12), pp. 293–300 (2012). http://doi.org/10.1145/2157689.2157798

  • Csapo, A., Gilmartin, E., Grizou, J., Han, J., Meena, R., Anastasiou, D., Jokinen, K., Wilcock, G.: Multimodal conversational interaction with a humanoid robot. In: 2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom), pp. 667–672. IEEE (2012)

  • Coates, J.: Women, Men and Language: A Sociolinguistic Account of Gender Differences in Language. Routledge, Abingdon (2015)

    Book  Google Scholar 

  • Darves, C., Oviatt, S.: Adaptation of users’ spoken dialogue patterns in a conversational interface. In: Seventh International Conference on Spoken Language Processing (2002)

  • De Carolis, B., Ferilli, S., Palestra, G., Carofiglio, V.: Modeling and simulating empathic behavior in social assistive robots. In: Proceedings of the 11th Biannual Conference on Italian SIGCHI Chapter, September 2015, pp. 110–117. ACM (2015)

  • Drolet, A.L., Morris, M.W.: Rapport in conflict resolution: accounting for how face-to-face contact fosters mutual cooperation in mixed-motive conflicts. J. Exp. Soc. Psychol. 36(1), 26–50 (2000)

    Article  Google Scholar 

  • Foster, M.E., Gaschler, A., Giuliani, M.: How can I help you’: comparing engagement classification strategies for a robot bartender. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, December 2013, pp. 255–262. ACM (2013)

  • Giles, H., Smith, P.: Accommodation theory: optimal levels of convergence. In: Howard, G., Robert, N. (eds.) Language and Social Psychology, pp. 45–65. St Clair (1979)

  • Gordon, G., Spaulding, S., Westlund, J.K., Lee, J.J., Plummer, L., Martinez, M., Breazeal, C., et al.: Affective personalization of a social robot tutor for children’s second language skills. In: Thirtieth AAAI Conference on Artificial Intelligence, March 2016 (2016)

  • Gratch, J., Wang, N., Gerten, J., Fast, E., Duffy, R.: Creating rapport with virtual agents. In: International Workshop on Intelligent Virtual Agents, September 2007, pp. 125–138. Springer, Berlin, Heidelberg (2007)

  • Gravano, A., Benus, S., Levitan, R., Hirschberg, J.: Three ToBI-based measures of prosodic entrainment and their correlations with speaker engagement. In: 2014 IEEE Spoken Language Technology Workshop (SLT), December 2014, pp. 578–583. IEEE (2014)

  • Gregory, S.W., Dagan, K., Webster, S.: Evaluating the relation of vocal accommodation in conversation partners’ fundamental frequencies to perceptions of communication quality. J. Nonverbal Behav. 21(1), 23–43 (1997)

    Article  Google Scholar 

  • Gulz, A., Haake, M., Silvervarg, A.: Extending a teachable agent with a social conversation module—effects on student experiences and learning. In: International Conference on Artificial Intelligence in Education, June 2011, pp. 106–114. Springer, Berlin, Heidelberg (2011)

  • Gweon, G., Jain, M., McDonough, J., Raj, B., Rosé, C.P.: Measuring prevalence of other-oriented transactive contributions using an automated measure of speech style accommodation. Int. J. Comput. Support. Collab. Learn. 8(2), 245–265 (2013)

    Article  Google Scholar 

  • Hess, U., Blairy, S.: Facial mimicry and emotional contagion to dynamic emotional facial expressions and their influence on decoding accuracy. Int. J. Psychophysiol. 40(2), 129–141 (2001)

    Article  Google Scholar 

  • Holmes, J.: Women, Men and Politeness. Longman, London (1995)

    Google Scholar 

  • Hong, J.-C., Hwang, M.-Y.: Gender differences in help-seeking and supportive dialogue during on-line game. Procedia Soc. Behav. Sci. 64(2007), 342–351 (2012). https://doi.org/10.1016/j.sbspro.2012.11.041

    Article  Google Scholar 

  • Huang, L., Morency, L.P., Gratch, J.: Virtual rapport 2.0. In: International Workshop on Intelligent Virtual Agents, September 2011, pp. 68–79. Springer, Berlin, Heidelberg (2011)

  • Jacq, A., Lemaignan, S., Garcia, F., Dillenbourg, P., Paiva, A.: Building successful long child–robot interactions in a learning context. In: 2016 11th ACM/IEEE International Conference on Human–Robot Interaction (HRI), March 2016, pp. 239–246. IEEE (2016)

  • Jokinen, K., Hurtig, T.: User expectations and real experience on a multimodal interactive system. In: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, January 2006, vol. 2, pp. 1049–1052 (2006)

  • Kanda, T., Hirano, T., Eaton, D., Ishiguro, H.: Interactive robots as social partners and peer tutors for children: a field trial. Hum. Comput. Interact. 19(1), 61–84 (2004)

    Article  Google Scholar 

  • Kasap, Z., Magnenat-Thalmann, N.: Building long-term relationships with virtual and robotic characters: the role of remembering. Vis. Comput. 28(1), 87–97 (2012)

    Article  Google Scholar 

  • Kasap, Z., Magnenat-Thalmann, N.: Towards episodic memory-based long-term affective interaction with a human-like robot. In: 19th International Symposium in Robot and Human Interactive Communication, September 2010, pp. 452–457. IEEE (2010)

  • Kory-Westlund, J.M., Breazeal, C.: Exploring the effects of a social robot’s speech entrainment and backstory on young children’s emotion, rapport, relationship, and learning. Front. Robot. AI 6, 54 (2019)

    Article  Google Scholar 

  • Kose-Bagci, H., Ferrari, E., Dautenhahn, K., Syrdal, D.S., Nehaniv, C.L.: Effects of embodiment and gestures on social interaction in drumming games with a humanoid robot. Adv. Robot. 23(14), 1951 (2009). https://doi.org/10.1163/016918609X12518783330360

    Article  Google Scholar 

  • Kousidis, S., Dorran, D.: Monitoring convergence of temporal features in spontaneous dialogue speech. In: First Young Researchers Workshop on Speech Technology, Dublin, Ireland, January 2009 (2009)

  • Krämer, N.C., Karacora, B., Lucas, G., Dehghani, M., Rüther, G., Gratch, J.: Closing the gender gap in STEM with friendly male instructors? On the effects of rapport behavior and gender of a virtual agent in an instructional interaction. Comput. Educ. 99, 1–13 (2016)

    Article  Google Scholar 

  • Kumar, R., Ai, H., Beuth, J.L., Rosé, C.P.: Socially capable conversational tutors can be effective in collaborative learning situations. In: Aleven, V., Kay, J., Mostow, J. (eds.) Intelligent Tutoring Systems, pp. 156–164. Springer, Berlin (2010)

    Chapter  Google Scholar 

  • Lakin, J.L., Chartrand, T.L.: Using nonconscious behavioral mimicry to create affiliation and rapport. Psychol. Sci. 14(4), 334–339 (2003)

    Article  Google Scholar 

  • Lakens, D., Stel, M.: If they move in sync, they must feel in sync: Movement synchrony leads to attributions of rapport and entitativity. Soc. Cognit. 29(1), 1–14 (2011)

    Article  Google Scholar 

  • Lee, C.C., Katsamanis, A., Black, M.P., Baucom, B.R., Georgiou, P.G., Narayanan, S.S.: An analysis of PCA-based vocal entrainment measures in married couples’ affective spoken interactions. In: Proceedings of Interspeech, Florence, Italy (2011)

  • Lee, M.K., Forlizzi, J., Kiesler, S., Rybski, P., Antanitis, J., Savetsila, S.: Personalization in HRI: a longitudinal field experiment. In: 2012 7th ACM/IEEE International Conference on Human–Robot Interaction (HRI), March 2012, pp. 319–326. IEEE (2012)

  • Lee, N., Shin, H., Sundar, S.S.: Utilitarian vs. hedonic robots: role of parasocial tendency and anthropomorphism in shaping user attitudes. In: Proceedings of the 6th International Conference on Human–Robot Interaction, March 2011, pp. 183–184 (2011)

  • Leelawong, K., Biswas, G.: Designing learning by teaching agents: the Betty’s Brain system. Int. J. Artif. Intell. Educ. 18(3), 181–208 (2008)

    Google Scholar 

  • Levitan, R., Hirschberg, J.: Measuring acoustic-prosodic entrainment with respect to multiple levels and dimensions. In: Interspeech 2011 (2011)

  • Levitan, R., Benus, S., Galvez, R.H., Gravano, A., Savoretti, F., Trnka, M., Hirschberg, J.: Implementing acoustic-prosodic entrainment in a conversational avatar. In: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 08–12 September 2016, pp. 1166–1170 (2016). http://doi.org/10.21437/Interspeech.2016-985

  • Levitan, R., Beňuš, Š., Gravano, A., Hirschberg, J.: Acoustic-prosodic entrainment in Slovak, Spanish, English and Chinese: a cross-linguistic comparison. In: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 325–334 (2015)

  • Levitan, R., Gravano, A., Willson, L., Benus, S., Hirschberg, J., Nenkova, A.: Acoustic-prosodic entrainment and social behavior. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2012, pp. 11–19. Association for Computational Linguistics (2012)

  • Longster, J.: Concatenative speech synthesis: a framework for reducing perceived distortion when using the TD-PSOLA algorithm. Dissertation, Bournemouth University (2003)

  • Looije, R., van der Zalm, A., Neerincx, M.A., Beun, R.J.: Help, I need somebody: the effect of embodiment on playful learning, pp. 718–724. IEEE (2012). http://dx.doi.org/10.1109/ROMAN.2012.6343836

  • Lubold, N., Pon-Barry, H., Walker, E.: Naturalness and rapport in a pitch adaptive learning companion. In: 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), December 2015, pp. 103–110. IEEE (2015)

  • Lubold, N., Pon-Barry, H.: Acoustic-prosodic entrainment and rapport in collaborative learning dialogues. In: Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge, November 2014, pp. 5–12 (2014)

  • Lubold, N., Walker, E., Pon-Barry, H., Ogan, A.: Automated pitch convergence improves learning in a social, teachable robot for middle school mathematics. In: International Conference on Artificial Intelligence in Education, pp. 282–296. Springer, Cham (2018)

  • Lubold, N.: Producing acoustic-prosodic entrainment in a robotic learning companion to build learner rapport. Doctoral dissertation, Arizona State University (2018)

  • Lutfi, S., Fernández-Martínez, F., Lorenzo-Trueba, J., Barra-Chicote, R., Montero, J.: I feel you: the design and evaluation of a domotic affect-sensitive spoken conversational agent. Sensors 13(8), 10519–10538 (2013)

    Article  Google Scholar 

  • Meena, R., Jokinen, K., Wilcock, G.: Integration of gestures and speech in human–robot interaction. In: 2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom), December 2012, pp. 673–678. IEEE (2012)

  • Mehrabian, A.: Nonverbal Communication. Transaction Publishers, Piscataway (1972)

    Google Scholar 

  • Murphy, K.: The Bayes net toolbox for Matlab. Comput. Sci. Stat. 33(2), 1024–1034 (2001)

    Google Scholar 

  • Mushin, I., Stirling, L., Fletcher, J., Wales, R.: Discourse structure, grounding, and prosody in task-oriented dialogue. Discourse Process. 35(1), 1–31 (2003)

    Article  Google Scholar 

  • Natale, M.: Convergence of mean vocal intensity in dyadic communication as a function of social desirability. J. Pers. Soc. Psychol. 32(5), 790 (1975)

    Article  Google Scholar 

  • Nenkova, A., Gravano, A., Hirschberg, J.: High frequency word entrainment in spoken dialogue. In: Proceedings of ACL-08: HLT, Short Papers, pp. 169–172 (2008)

  • Novick, D., Gris, I.: Building rapport between human and ECA: a pilot study. In: International Conference on Human–Computer Interaction, June 2014, pp. 472–480. Springer, Cham (2014)

  • Nwe, T.L., Foo, S.W., De Silva, L.C.: Speech emotion recognition using hidden Markov models. Speech Commun. 41(4), 603–623 (2003)

    Article  Google Scholar 

  • Ogan, A., Finkelstein, S., Walker, E., Carlson, R., Cassell, J.: Rudeness and rapport: insults and learning gains in peer tutoring. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) Intelligent Tutoring Systems, pp. 11–21. Springer, Berlin (2012)

    Chapter  Google Scholar 

  • Pantic, M., Pentland, A., Nijholt, A., Huang, T.S.: Human computing and machine understanding of human behavior: a survey. In: Artificial Intelligence for Human Computing, pp. 47–71. Springer, Berlin, Heidelberg (2007)

  • Powers, A., Kiesler, S., Fussell, S., Torrey, C.: Comparing a computer agent with a humanoid robot. In: Proceedings of the ACM/IEEE International Conference on Human–Robot Interaction, March 2007, pp. 145–152 (2007)

  • Sadoughi, N., Pereira, A., Jain, R., Leite, I., Lehman, J.F.: Creating prosodic synchrony for a robot co-player in a speech-controlled game for children, pp. 91–99 (2017). https://doi.org/10.1145/2909824.3020244

  • Saerbeck, M., Schut, T., Bartneck, C., Janse, M.D.: Expressive robots in education: varying the degree of social supportive behavior of a robotic tutor. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1613–1622. ACM (2010)

  • Saint-Aimé, S., Le-Pevedic, B., Duhaut, D., Shibata, T.: EmotiRob: companion robot project. In: RO-MAN 2007-The 16th IEEE International Symposium on Robot and Human Interactive Communication, pp. 919–924. IEEE (2007)

  • Schermerhorn, P., Scheutz, M., Crowell, C.R.: Robot social presence and gender: do females view robots differently than males?. In: Proceedings of the 3rd ACM/IEEE International Conference on Human–Robot Interaction, March 2008, pp. 263–270 (2008)

  • Schweitzer, A., Lewandowski, N.: Convergence of articulation rate in spontaneous speech. In: INTERSPEECH, August 2013, pp. 525–529 (2013)

  • Scissors, L.E., Gill, A.J., Geraghty, K., Gergle, D.: In CMC we trust: the role of similarity. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, April 2009, pp. 527–536. ACM (2009)

  • Seo, S.H., Geiskkovitch, D., Nakane, M., King, C., Young, J.E.: Poor thing! Would you feel sorry for a simulated robot?. In: Proceedings of International Conference on Human–Robot Interaction—HRI’15, pp. 125–132. ACM (2015). https://doi.org/10.1145/2696454.2696471

  • Shawar, B.A., Atwell, E.: A comparison between Alice and Elizabeth chatbot systems. University of Leeds, School of Computing research report 2002.19 (2002)

  • Shawar, B.A., Atwell, E.: Chatbots: are they really useful?. In: Ldv Forum, January 2007, vol. 22, no. 1, pp. 29–49 (2007)

  • Sidaras, S.K.: Hearing what you expect to hear: the interaction of social and cognitive mechanisms underlying vocal accommodation. Doctoral dissertation, Emory University (2011)

  • Sinha, T., Cassell, J.: We click, we align, we learn: impact of influence and convergence processes on student learning and rapport building. In: Proceedings of the 1st Workshop on Modeling INTERPERsonal SynchrONy And infLuence, pp. 13–20 (2015). http://doi.org/10.1145/2823513.2823516

  • Spencer-Oatey, H.: (Im)Politeness, face and perceptions of rapport: unpackaging their bases and interrelationships. Polit. Res. 1(1), 95–119 (2005). https://doi.org/10.1515/jplr.2005.1.1.95

    Article  Google Scholar 

  • Stewart, M., Brown, J.B., Boon, H., Galajda, J., Meredith, L., Sangster, M.: Evidence on patient–doctor communication. Cancer 25(1999), 30 (1999)

    Google Scholar 

  • Strait, M., Briggs, P., Scheutz, M.: Gender, more so than age, modulates positive perceptions of language-based human–robot interactions. In: 4th International Symposium on New Frontiers in Human Robot Interaction, April 2015, pp. 21–22 (2015)

  • Tanaka, F., Matsuzoe, S.: Children teach a care-receiving robot to promote their learning: field experiments in a classroom for vocabulary learning. J. Hum. Robot Interact. 1(1), 78–95 (2012). https://doi.org/10.5898/JHRI.1.1.Tanaka

    Article  Google Scholar 

  • Tannen, D.: Gender and Discourse. Oxford University Press, Oxford (1994)

    Google Scholar 

  • Thomason, J., Nguyen, H.V., Litman, D.: Prosodic entrainment and tutoring dialogue success. In: International Conference on Artificial Intelligence in Education, July 2013, pp. 750–753. Springer, Berlin, Heidelberg (2013)

  • Tickle-Degnen, L., Rosenthal, R.: The nature of rapport and its nonverbal correlates. Psychol. Inq. 1(4), 285–293 (1990)

    Article  Google Scholar 

  • Vail, A.K., Boyer, K.E., Wiebe, E.N., Lester, J.C.: The mars and venus effect: the influence of user gender on the effectiveness of adaptive task support. In: International Conference on User Modeling, Adaptation, and Personalization, June 2015, pp. 265–276. Springer, Cham (2015)

  • Vaughan, B.: Prosodic synchrony in co-operative task-based dialogues: a measure of agreement and disagreement. In: Twelfth Annual Conference of the International Speech Communication Association (2011)

  • Walker, E., Girotto, V., Kim, Y., Muldner, K.: The effects of physical form and embodied action in a teachable robot for geometry learning. In: 2016 IEEE 16th International Conference on Advanced Learning Technologies (ICALT), July 2016, pp. 381–385. IEEE (2016)

  • Wallace, R.: The elements of AIML style. Alice AI Foundation (2003)

  • Wang, N., Gratch, J.: Rapport and facial expression. In: 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, September 2009, pp. 1–6. IEEE (2009)

  • Web Speech API: https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API. Accessed 26 Oct 2018

  • Weizenbaum, J.: ELIZA—a computer program for the study of natural language communication between man and machine. Commun. ACM 9(1), 36–45 (1966)

    Article  Google Scholar 

  • Westlund, J.K., Lee, J.J., Plummer, L., Faridi, F., Gray, J., Berlin, M., Dos Santos, K., et al.: Tega: a social robot. In: The Eleventh ACM/IEEE International Conference on Human Robot Interaction, March 2016, pp. 561–561. IEEE Press (2016)

  • Wheldall, K., Mettem, P.: Behavioural peer tutoring: training 16-year-old tutors to employ the ‘pause, prompt and praise’method with 12-year-old remedial readers. Educ. Psychol. 5(1), 27–44 (1985)

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Robotics Initiative and the National Science Foundation, Grant # CISE-IIS-1637809.

Author information

Authors and Affiliations

  1. School of Electrical, Computer, and Energy Engineering, Arizona State University, Tempe, AZ, USA

    Nichola Lubold

  2. Human Centered Systems Group, Honeywell Labs, Deer Valley, AZ, USA

    Nichola Lubold

  3. School of Computing and Information, University of Pittsburg, Pittsburg, PA, USA

    Erin Walker

  4. Department of Computer Science, Mount Holyoke College, South Hadley, MA, USA

    Heather Pon-Barry

Authors
  1. Nichola Lubold
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erin Walker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heather Pon-Barry
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nichola Lubold.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This paper is void of plagiarism or self-plagiarism as defined by the Committee on Publication Ethics and Springer Guidelines.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Rapport Measures: Participants responded on a Likert scale from 1 to 5

I felt I had a connection with Quinn

I felt I was able to engage Quinn

I think that Quinn and I understood each other

I felt that Quinn was interested in what I had to say

I felt that Quinn was warm and caring

I felt that Quinn was intensely involved in the interaction

I felt that Quinn seemed to find the interaction stimulating

I felt that Quinn was respectful to me

I felt that Quinn showed enthusiasm while talking to me

Social Presence Measures: Participants responded on a Likert scale from 1 to 7

Quinn was easily distracted

I was easily distracted

Quinn tended to ignore me

I tended to ignore Quinn

I sometimes pretend to pay attention to Quinn

Quinn sometimes pretended to pay attention to me

Quinn paid close attention to me

Coding Scheme:

Politeness: “P” is polite to Quinn, follows conversational niceties (like saying hello)

  • Ex 1: Thank you, Quinn

  • Ex 2: ah step four please

Complimenting or praising: “P” praises Quinn

  • Ex 1: good job Quinn

  • Ex 2: great! Now I factor out the two

  • Ex 3: nice!

Name usage: “P” uses Quinn’s name

  • Ex 1: Nice job Quinn (this would contain checks in both the praise column and the name column)

Inclusive: “P” includes Quinn, for example by using ‘inclusive’ language such as “us,” “we,”, “together”, “let’s”

  • Ex 1: Let’s do problem one!

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lubold, N., Walker, E. & Pon-Barry, H. Effects of adapting to user pitch on rapport perception, behavior, and state with a social robotic learning companion. User Model User-Adap Inter 31, 35–73 (2021). https://doi.org/10.1007/s11257-020-09267-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-020-09267-3

Keywords

Search

Navigation