Skip to main content
SpringerLink
Log in

Multimodal modeling of collaborative problem-solving facets in triads

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://zoom.us.

  2. https://www.ibm.com/watson/services/speech-to-text/.

  3. We utilized the same cross-validation procedure described in Sect. 5.3 for these models. We specifically tuned the following hyperparameters for each classifier: Naïve Bayes—alpha, class prior distributions; Logistic Regression—regularization norm, regularization strength; Support Vector Machines—kernel, regularization strength; Adaboost—number of estimators, learning rate.

References

  • Alterman, R., Harsch, K.: A more reflective form of joint problem solving. Int. J. Comput. Support. Collab. Learn. 12(1), 9–33 (2017). https://doi.org/10.1007/s11412-017-9250-1

    Article  Google Scholar 

  • Amon, M.J., Vrzakova, H., D’Mello, S.K.: Beyond dyadic coordination: multimodal behavioral irregularity in triads predicts facets of collaborative problem solving. Cogn. Sci. 43(10), e12787 (2019). https://doi.org/10.1111/cogs.12787

    Article  Google Scholar 

  • Andrews-Todd, J., Forsyth, C.M.: Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task. Comput. Hum. Behav. (2018). https://doi.org/10.1016/j.chb.2018.10.025

    Article  Google Scholar 

  • Aran, O., Gatica-Perez, D.: Fusing audio-visual nonverbal cues to detect dominant people in group conversations. In: 2010 20th International Conference on Pattern Recognition, pp. 3687–3690 (2010). https://doi.org/10.1109/ICPR.2010.898

  • Beyan, C., Capozzi, F., Becchio, C., Murino, V.: Identification of emergent leaders in a meeting scenario using multiple kernel learning. In: Proceedings of the 2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction, pp. 3–10 (2016a)

  • Beyan, C., Carissimi, N., Capozzi, F., Vascon, S., Bustreo, M., Pierro, A., Becchio, C., Murino, V.: Detecting emergent leader in a meeting environment using nonverbal visual features only. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 317–324 (2016b). https://doi.org/10.1145/2993148.2993175

  • Bird, S., Loper, E.: NLTK: The natural language toolkit. In: Proceedings of the Association for Computational Linguistics 2004 on Interactive Poster and Demonstration Sessions, 31-es (2004). https://doi.org/10.3115/1219044.1219075

  • Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)

    MATH  Google Scholar 

  • Calacci, D., Lederman, O., Shrier, D., Pentland, A.S.: Breakout: an open measurement and intervention tool for distributed peer learning groups (2016). CoRR, abs/1607.0. http://arxiv.org/abs/1607.01443

  • Camara, W., O’Connor, R., Mattern, K., Hanson, M.A.: Beyond academics: a holistic framework for enhancing education and workplace success. ACT Research Report Series. 2015(4). ACT, Inc. (2015)

  • Chopade, P, Edwards, D., Khan, S.M., Andrade, A., Pu, S.: CPSX: using AI-machine learning for mapping human–human interaction and measurement of CPS teamwork skills. In: 2019 IEEE International Symposium on Technologies for Homeland Security (HST), pp. 1–6 (2019). https://doi.org/10.1109/HST47167.2019.9032906

  • Chopade, P., Stoeffler, K.M., Khan, S., Rosen, Y., Swartz, S., von Davier, A.: Human-Agent Assessment: Interaction And Sub-Skills Scoring For Collaborative Problem Solving. In: Martínez-Maldonado, R., Hoppe, H.U., Luckin, R., Mavrikis, M., Porayska-Pomsta, K., McLaren, B., du Boulay, B. (eds.) C Penstein Rosé, pp. 52–57. Artificial Intelligence in Education. Springer International Publishing, Berlin (2018)

    Google Scholar 

  • Church, K.W., Hanks, P.: Word association norms, mutual information, and lexicography. Comput. Linguist. 16(1), 22–29 (1990)

    Google Scholar 

  • Code Studio. (n.d.). Retrieved April 1, 2018, from https://studio.code.org/s/mc/stage/1/puzzle/1

  • Cukurova, M., Luckin, R., Millán, E., Mavrikis, M.: The NISPI framework: analysing collaborative problem-solving from students’ physical interactions. Comput. Educ. 116, 93–109 (2018). https://doi.org/10.1016/j.compedu.2017.08.007

    Article  Google Scholar 

  • Cukurova, M., Zhou, Q., Spikol, D., Landolfi, L.: Modelling collaborative problem-solving competence with transparent learning analytics: is video data enough? In: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pp. 270–275 (2020). https://doi.org/10.1145/3375462.3375484

  • de Kok, I., Heylen, D.: Multimodal end-of-turn prediction in multi-party meetings. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, pp. 91–98 (2009). https://doi.org/10.1145/1647314.1647332

  • Delaherche, E., Chetouani, M., Mahdhaoui, A., Saint-Georges, C., Viaux, S., Cohen, D.: Interpersonal synchrony: a survey of evaluation methods across disciplines. IEEE Trans. Affect. Comput. 3(3), 349–365 (2012). https://doi.org/10.1109/T-AFFC.2012.12

    Article  Google Scholar 

  • Dielmann, A., Garau, G., Bourlard, H.: Floor holder detection and end of speaker turn prediction in meetings. In: Proceedings of the International Conference on Speech and Language Processing, Interspeech (2010)

  • Dozat, T.: Incorporating nesterov momentum into adam. In: Proceedings of the International Conference on Learning Representations (2016)

  • Duran, N.D., Fusaroli, R.: Conversing with a devil’s advocate: interpersonal coordination in deception and disagreement. PLoS ONE 12(6), e0178140 (2017). https://doi.org/10.1371/journal.pone.0178140

    Article  Google Scholar 

  • Dyke, G., Adamson, D., Howley, I., Penstein Rosé, C.: Towards academically productive talk supported by conversational agents. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) Intelligent Tutoring Systems, pp. 531–540. Springer, Berlin (2012)

    Chapter  Google Scholar 

  • Ekman, R.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, Oxford (1997)

    Google Scholar 

  • Evans, A. C., Wobbrock, J. O., Davis, K.: Modeling collaboration patterns on an interactive tabletop in a classroom setting. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, pp. 860–871 (2016). https://doi.org/10.1145/2818048.2819972

  • Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia (MM’13), pp. 835–838 (2013). https://doi.org/10.1145/2502081.2502224

  • Fan, B., Wang, L., Soong, F.K., Xie, L. Photo-real talking head with deep bidirectional LSTM. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4884–4888 (2015). https://doi.org/10.1109/ICASSP.2015.7178899

  • Faucett, H.A., Lee, M.L., Carter, S.: I should listen more: real-time sensing and feedback of non-verbal communication in video telehealth. In: Proceedings of the ACM on Human–Computer Interaction 1(CSCW), pp. 44:1–44:19 (2017). https://doi.org/10.1145/3134679

  • Fiore, S.M., Graesser, A., Greiff, S.: Collaborative problem-solving education for the twenty-first-century workforce. Nat. Hum. Behav. 2(6), 367–369 (2018). https://doi.org/10.1038/s41562-018-0363-y

    Article  Google Scholar 

  • Flor, M., Yoon, S.-Y., Hao, J., Liu, L., von Davier, A.: Automated classification of collaborative problem solving interactions in simulated science tasks. In: Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 31–41 (2016)

  • Fraser, N. (2015). Ten things we’ve learned from Blockly. In: Proceedings of the 2015 IEEE Blocks and Beyond Workshop, pp. 49–50. https://doi.org/10.1109/BLOCKS.2015.7369000

  • Fusaroli, R., Rkaczaszek-Leonardi, J., Tylén, K.: Dialog as interpersonal synergy. New Ideas Psychol. 32, 147–157 (2014)

    Article  Google Scholar 

  • Gosling, S.D., Rentfrow, P.J., Swann, W.B.: A very brief measure of the Big-Five personality domains. J. Res. Pers. 37(6), 504–528 (2003). https://doi.org/10.1016/S0092-6566(03)00046-1

    Article  Google Scholar 

  • Graesser, A.C., Fiore, S.M., Greiff, S., Andrews-Todd, J., Foltz, P.W., Hesse, F.W.: Advancing the science of collaborative problem solving. Psychol. Sci. Public Interest 19(2), 59–92 (2018). https://doi.org/10.1177/1529100618808244

    Article  Google Scholar 

  • Grafsgaard, J., Duran, N., Randall, A., Tao, C., D’Mello, S.: Generative multimodal models of nonverbal synchrony in close relationships. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 195–202 (2018). https://doi.org/10.1109/FG.2018.00037

  • Griffin, P., Care, E., McGaw, B.: The changing role of education and schools. In: Griffin, P., McGaw, B., Care, E. (eds.) Assessment and Teaching of 21st Century Skills, pp. 1–15. Springer, Dordrecht (2012). https://doi.org/10.1007/978-94-007-2324-5_1

    Chapter  Google Scholar 

  • Grover, S., Bienkowski, M., Tamrakar, A., Siddiquie, B., Salter, D., Divakaran, A.: Multimodal analytics to study collaborative problem solving in pair programming. In: Proceedings of the Sixth International Conference on Learning Analytics and Knowledge, pp. 516–517 (2016). https://doi.org/10.1145/2883851.2883877

  • Gutwin, C., Bateman, S., Arora, G., Coveney, A.: Looking away and catching up: dealing with brief attentional disconnection in synchronous groupware. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 2221–2235 (2017). https://doi.org/10.1145/2998181.2998226

  • Gwet, K.L.: Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. Advanced Analytics, LLC, Gaithersburg (2014)

    Google Scholar 

  • Hall, J.A., Coats, E.J., LeBeau, L.S.: Nonverbal behavior and the vertical dimension of social relations: a meta-analysis. Psychol. Bull. 131(6), 898–924 (2005). https://doi.org/10.1037/0033-2909.131.6.898

    Article  Google Scholar 

  • Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1), 29–36 (1982). https://doi.org/10.1148/radiology.143.1.7063747

    Article  Google Scholar 

  • Hao, J., Chen, L., Flor, M., Liu, L., von Davier, A.A.: CPS-Rater: automated sequential annotation for conversations in collaborative problem-solving activities. ETS Res. Rep. Ser. 2017(1), 1–9 (2017). https://doi.org/10.1002/ets2.12184

    Article  Google Scholar 

  • Hesse, F., Care, E., Buder, J., Sassenberg, K., Griffin, P.: A framework for teachable collaborative problem solving skills. In: Griffin, P., Care, E. (eds.) Assessment and Teaching of 21st Century Skills: Methods and Approach, pp. 37–56. Springer, Dordrecht (2015). https://doi.org/10.1007/978-94-017-9395-7_2

    Chapter  Google Scholar 

  • Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  • Hung, H., Gatica-Perez, D.: Estimating cohesion in small groups using audio–visual nonverbal behavior. IEEE Trans. Multimedia 12(6), 563–575 (2010)

    Article  Google Scholar 

  • Hunt, M.J.: Figures of merit for assessing connected-word recognisers. Speech Commun. 9(4), 329–336 (1990). https://doi.org/10.1016/0167-6393(90)90008-W

    Article  Google Scholar 

  • Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning, pp. 448–456 (2015). PMLR

  • Ishii, R., Otsuka, K., Kumano, S., Higashinaka, R., Tomita, J.: Analyzing Gaze behavior and dialogue act during turn-taking for estimating empathy skill level. In: Proceedings of the 2018 on International Conference on Multimodal Interaction, pp. 31–39 (2018). https://doi.org/10.1145/3242969.3242978

  • Jermann, P., Sharma, K.: Gaze as a proxy for cognition and communication. In: 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), pp. 152–154 (2018)

  • Jokinen, K., Furukawa, H., Nishida, M., Yamamoto, S.: Gaze and turn-taking behavior in casual conversational interactions. ACM Trans. Interact. Intell. Syst. 3(2), 12:1-12:30 (2013). https://doi.org/10.1145/2499474.2499481

    Article  Google Scholar 

  • Kelly, J.R., Barsade, S.G.: Mood and emotions in small groups and work teams. Organ. Behav. Hum. Decis. Process. 86(1), 99–130 (2001)

    Article  Google Scholar 

  • Kendon, A.: Some functions of gaze-direction in social interaction. Acta Physiol. 26, 22–63 (1967). https://doi.org/10.1016/0001-6918(67)90005-4

    Article  Google Scholar 

  • Kenton, M. C., Kristina, L., Devlin, J.: BERT paper (2017). http://arxiv.org/abs/1810.04805 [Cs]

  • Keras. (n.d.). Retrieved May 2, 2018, from https://github.com/keras-team/keras

  • Kerr, N.L., Tindale, R.S.: Group performance and decision making. Annu. Rev. Psychol. 55(1), 623–655 (2004). https://doi.org/10.1146/annurev.psych.55.090902.142009

    Article  Google Scholar 

  • Krafft, P.M., Baker, C.L., Tenenbaum, J.B., et al.: Modeling human ad hoc coordination. In: Thirtieth AAAI Conference on Artificial Intelligence (2016)

  • Lai, E., DiCerbo, K., Foltz, P.: Skills for today: what we know about teaching and assessing collaboration. Pearson (2017)

  • Latif, N., Barbosa, A.V., Vatiokiotis-Bateson, E., Castelhano, M.S., Munhall, K.G.: Movement coordination during conversation. PLoS ONE 9(8), 1–10 (2014). https://doi.org/10.1371/journal.pone.0105036

    Article  Google Scholar 

  • Lemaitre, G., Nogueira, F., Aridas, C.K.: Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. J. Mach. Learn. Res. 18(1), 559–563 (2017)

    Google Scholar 

  • Levitan, R., Gravano, A., Willson, L., Benus, S., Hirschberg, J., Nenkova, A. Acoustic–prosodic entrainment and social behavior. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 11–19 (2012)

  • Lin, D.: Extracting collocations from text corpora. In: First Workshop on Computational Terminology, pp. 57–63 (1998)

  • Lipton, Z.C., Kale, D.C., Wetzel, R.: Directly modeling missing data in sequences with RNNs: improved classification of clinical time series. In: Doshi-Velez, F., Fackler, J., Kale, D., Wallace, B., Wiens, J. (eds.) Proceedigns of Machine Learning Research, pp. 253–270. PMLR (2016)

  • Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (CERT). In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, FG 2011, pp. 298–305 (2011). https://doi.org/10.1109/FG.2011.5771414

  • Lo, R.T.-W., He, B., Ounis, I.: Automatically building a stopword list for an information retrieval system. In: Journal on Digital Information Management: Special Issue on the 5th Dutch-Belgian Information Retrieval Workshop (DIR), 5, 17–24 (2005)

  • Lu, J., Chiu, M.M., Law, N.W.: Collaborative argumentation and justifications: a statistical discourse analysis of online discussions. Comput. Hum. Behav. 27(2), 946–955 (2011). https://doi.org/10.1016/j.chb.2010.11.021

    Article  Google Scholar 

  • Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the International Conference on Machine Learning (2013)

  • Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A.: Deep captioning with multimodal recurrent neural networks (M-RNN). In: Proceedings of the 2015 International Conference on Learning Representations (2015)

  • Müller, P., Huang, M.X., Bulling, A.: Detecting low rapport during natural interactions in small groups from non-verbal behaviour. In: 23rd International Conference on Intelligent User Interfaces, pp. 153–164 (2018)

  • Murray, G., Oertel, C.: Predicting group performance in task-based interaction. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 14–20 (2018). https://doi.org/10.1145/3242969.3243027

  • Nelson, L.M.: Collaborative problem solving. Instr. Des. Theories Models New Paradigm Instr. Theory 2, 241–267 (1999)

    Google Scholar 

  • Newell, A., Simon, H.A., et al.: Human Problem Solving, vol. 104, Issue 9. Prentice-Hall, Englewood Cliffs (1972)

  • Ng, A.Y.: Feature selection, L1 vs. L2 regularization, and rotational invariance. In: Proceedings of the Twenty-First International Conference on Machine Learning, p. 78 (2004). https://doi.org/10.1145/1015330.1015435

  • Nihei, F., Nakano, Y.I., Hayashi, Y., Hung, H.-H., Okada, S.: Predicting influential statements in group discussions using speech and head motion information. In: Proceedings of the 16th International Conference on Multimodal Interaction, pp. 136–143 (2014). https://doi.org/10.1145/2663204.2663248

  • OECD.: PISA 2015 Results (Volume I): excellence and equity in education, PISA, OECD Publishing, Paris. (2016). https://doi.org/10.1787/9789264266490-en

  • Otsuka, K., Kasuga, K., Köhler, M.: Estimating visual focus of attention in multiparty meetings using deep convolutional neural networks. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 191–199 (2018). https://doi.org/10.1145/3242969.3242973

  • Oviatt, S., Cohen, A. (2013). Written and multimodal representations as predictors of expertise and problem-solving success in mathematics. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 599–606. https://doi.org/10.1145/2522848.2533793

  • Palumbo, R.V., Marraccini, M.E., Weyandt, L.L., Wilder-Smith, O., McGee, H.A., Liu, S., Goodwin, M.S.: Interpersonal autonomic physiology: a systematic review of the literature. Personal. Soc. Psychol. Rev. 21(2), 99–141 (2017). https://doi.org/10.1177/1088868316628405

    Article  Google Scholar 

  • Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

  • Pham, H.X., Cheung, S., Pavlovic, V.: Speech-driven 3D facial animation with implicit emotional awareness: a deep learning approach. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops 2017, 2328–2336 (2017). https://doi.org/10.1109/CVPRW.2017.287

    Article  Google Scholar 

  • Polyak, S.T., von Davier, A.A., Peterschmidt, K.: Computational psychometrics for the measurement of collaborative problem solving skills. Front. Psychol. 8, 2029 (2017). https://doi.org/10.3389/fpsyg.2017.02029

    Article  Google Scholar 

  • Porter, M.F.: Snowball: A Language for Stemming Algorithms (2001). https://api.semanticscholar.org/CorpusID:59634627

  • Prata, D.N., Baker, R.S.J., Costa, E.B., Rosé, C.P., Cui, Y., De Carvalho, A.M.J.B.: Detecting and understanding the impact of cognitive and interpersonal conflict in computer supported collaborative learning environments. In: International Working Group on Educational Data Mining (2009)

  • Richardson, D.C., Dale, R., Kirkham, N.Z.: The art of conversation is coordination. Psychol. Sci. 18(5), 407–413 (2007). https://doi.org/10.1111/j.1467-9280.2007.01914.x

    Article  Google Scholar 

  • Roschelle, J., Teasley, S.D.: The construction of shared knowledge in collaborative problem solving. In: O’Malley, C. (ed.) Computer Supported Collaborative Learning, pp. 69–97. Springer, Berlin (1995)

    Chapter  Google Scholar 

  • Rosé, C., Wang, Y.-C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F.: Analyzing collaborative learning processes automatically: exploiting the advances of computational linguistics in computer-supported collaborative learning. Int. J. Comput. Support. Collab. Learn. 3(3), 237–271 (2008). https://doi.org/10.1007/s11412-007-9034-0

    Article  Google Scholar 

  • Rosen, Y.: Computer-based assessment of collaborative problem solving: exploring the feasibility of human-to-agent approach. Int. J. Artif. Intell. Educ. 25(3), 380–406 (2015). https://doi.org/10.1007/s40593-015-0042-3

    Article  Google Scholar 

  • Salas, E., Sims, D.E., Burke, C.S.: Is there a “Big Five” in teamwork? Small Group Res. 36(5), 555–599 (2005). https://doi.org/10.1177/1046496405277134

    Article  Google Scholar 

  • Samrose, S., Zhao, R., White, J., Li, V., Nova, L., Lu, Y., Ali, M.R., Hoque, M.E.: CoCo: collaboration coach for understanding team dynamics during video conferencing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(4), 160:1-160:24 (2018)

    Article  Google Scholar 

  • Sanchez-Cortes, D., Aran, O., Mast, M.S., Gatica-Perez, D.: Identifying emergent leadership in small groups using nonverbal communicative cues. Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact. 39(1–39), 4 (2010). https://doi.org/10.1145/1891903.1891953

    Article  Google Scholar 

  • Schlösser, C., Harrer, A., Kienle, A.: Supporting dyadic chat communication with eye tracking based reading awareness. In: 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), pp. 149–151 (2018). https://doi.org/10.1109/ICALT.2018.00042

  • Schulze, J., Krumm, S.: The “virtual team player”: a review and initial model of knowledge, skills, abilities, and other characteristics for virtual collaboration. Organ. Psychol. Rev. 7(1), 66–95 (2017). https://doi.org/10.1177/2041386616675522

    Article  Google Scholar 

  • Sinha, T., Cassell, J.: We click, we align, we learn: Impact of influence and convergence processes on student learning and rapport building. In: Proceedings of the 1st Workshop on Modeling INTERPERsonal SynchrONy And InfLuence, pp. 13–20 (2015)

  • Spikol, D., Ruffaldi, E., Dabisias, G., Cukurova, M.: Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. J. Comput. Assist. Learn. 34(4), 366–377 (2018). https://doi.org/10.1111/jcal.12263

    Article  Google Scholar 

  • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  • Stewart, A.E.B., D’Mello, S.K.: Connecting the dots towards collaborative AIED: linking group makeup to process to learning. In: International Conference on Artificial Intelligence in Education, pp. 545–556 (2018)

  • Stewart, A.E.B., Keirn, Z.A., D’Mello, S.K.: Multimodal modeling of coordination and coregulation patterns in speech rate during triadic collaborative problem solving. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 21–30 (2018). https://doi.org/10.1145/3242969.3242989

  • Stewart, A.E.B., Vrzakova, H., Sun, C., Yonehiro, J., Stone, C.A., Duran, N.D., Shute, V., D’Mello, S.K.: I say, you say, we say: using spoken language to model socio-cognitive processes during computer-supported collaborative problem solving. Proc. ACM Hum. Comput. Interact. 3, 19 (2019). https://doi.org/10.1145/3359296

    Article  Google Scholar 

  • Stoeffler, K., Rosen, Y., Bolsinova, M., von Davier, A.: Gamified assessment of collaborative skills with chatbots, pp. 343–347 (2018). https://doi.org/10.1007/978-3-319-93846-2_64

  • Subburaj, S.K., Stewart, A.E.B., Rao, A.R., D’Mello, S.K.: Multimodal, multiparty modeling of collaborative problem solving performance. In: Proceedings of the 2020 Conference on Multimodal Interaction (2020)

  • Sun, C., Shute, V.J., Stewart, A.E.B., Yonehiro, J., Duran, N., D’Mello, S.K., D’Mello, S., D’Mello, S.K.: Towards a generalized competency model of collaborative problem solving. Comput. Educ. 143, 103672 (2020). https://doi.org/10.1016/j.compedu.2019.103672

    Article  Google Scholar 

  • Tausczik, Y.R., Pennebaker, J.W.: The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29(1), 24–54 (2010). https://doi.org/10.1177/0261927X09351676

    Article  Google Scholar 

  • Tegos, S., Demetriadis, S., Karakostas, A.: Promoting academically productive talk with conversational agent interventions in collaborative learning settings. Comput. Educ. 87, 309–325 (2015). https://doi.org/10.1016/j.compedu.2015.07.014

    Article  Google Scholar 

  • Tegos, S., Demetriadis, S., Papadopoulos, P.M., Weinberger, A.: Conversational agents for academically productive talk: a comparison of directed and undirected agent interventions. Int. J. Comput. Support. Collab. Learn. 11(4), 417–440 (2016). https://doi.org/10.1007/s11412-016-9246-2

    Article  Google Scholar 

  • Vrzakova, H., Amon, M.J., Stewart, A., Duran, N.D., D’Mello, S.K.: Focused or stuck together: multimodal patterns reveal triads’ performance in collaborative problem solving. In: Proceedings of the Tenth International Conference on Learning Analytics and Knowledge, pp. 295–304 (2020). https://doi.org/10.1145/3375462.3375467

  • Webb, M., Gibson, D.: Technology enhanced assessment in complex collaborative settings. Educ. Inf. Technol. 20(4), 675–695 (2015). https://doi.org/10.1007/s10639-015-9413-5

    Article  Google Scholar 

  • Weintrop, D.: Minding the gap between blocks-based and text-based programming (Abstract Only). In: Proceedings of the 46th ACM Technical Symposium on Computer Science Education, p. 720 (2015). https://doi.org/10.1145/2676723.2693622

  • Weintrop, D., Wilensky, U.: Bringing blocks-based programming into high school computer science classrooms. In: Annual Meeting of the American Educational Research Association (AERA). Washington DC, USA (2016)

  • Westlund, J.K., D’Mello, S.K., Olney, A.M.: Motion Tracker: camera-based monitoring of bodily movements using motion silhouettes. PLoS ONE 10(6), e0130293 (2015). https://doi.org/10.1371/journal.pone.0130293

    Article  Google Scholar 

  • Yoo, J., Kim, J.: Can online discussion participation predict group project performance? Investigating the roles of linguistic features and participation patterns. Int. J. Artif. Intell. Educ. 24(1), 8–32 (2014). https://doi.org/10.1007/s40593-013-0010-8

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Colorado Boulder, Boulder, USA

    Angela E. B. Stewart, Zachary Keirn & Sidney K. D’Mello

Authors
  1. Angela E. B. Stewart
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zachary Keirn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sidney K. D’Mello
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Angela E. B. Stewart.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stewart, A.E.B., Keirn, Z. & D’Mello, S.K. Multimodal modeling of collaborative problem-solving facets in triads. User Model User-Adap Inter 31, 713–751 (2021). https://doi.org/10.1007/s11257-021-09290-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-021-09290-y

Keywords

Search

Navigation