Abstract

Objective: We explore state of the art machine learning based tools for automatic facial and linguistic affect analysis to allow easier, faster, and more precise quantification and annotation of children’s verbal and non-verbal affective expressions in psychodynamic child psychotherapy. Method: The sample included 53 Turkish children: 41 with internalizing, externalizing and comorbid problems; 12 in the non-clinical range. We collected audio and video recordings of 148 sessions, which were manually transcribed. Independent raters coded children’s expressions of pleasure, anger, sadness and anxiety using the Children’s Play Therapy Instrument (CPTI). Automatic facial and linguistic affect analysis modalities were adapted, developed, and combined in a system that predicts affect. Statistical regression methods (linear and polynomial regression) and machine learning techniques (deep learning, support vector regression and extreme learning machine) were used for predicting CPTI affect dimensions. Results: Experimental results show significant associations between automated affect predictions and CPTI affect dimensions with small to medium effect sizes. Fusion of facial and linguistic features work best for pleasure predictions; however, for other affect predictions linguistic analyses outperform facial analyses. External validity analyses partially support anger and pleasure predictions. Discussion: The system enables retrieving affective expressions of children, but needs improvement for precision.

摘要

目的：我们探索基于机器学习的先进工具，用于自动进行面部和语言情感分析，以在心理动力儿童心理治疗中更轻松，更快，更准确地量化和注释儿童的言语和非言语情感表达。方法：样本包括53名土耳其儿童：41名有内在，外在和共病的问题；在非临床范围内为12。我们收集了148个会话的音频和视频记录，这些记录是手动转录的。独立评估者使用儿童游戏治疗仪（CPTI）对儿童的愉悦，愤怒，悲伤和焦虑的表情进行编码。自动的面部和语言情感分析模式已进行调整，开发并组合到一个可以预测情感的系统中。统计回归方法（线性和多项式回归）和机器学习技术（深度学习，支持向量回归和极限学习机）用于预测CPTI影响维度。结果：实验结果表明，自动影响预测与CPTI影响大小之间具有显着关联，影响大小为中到小。面部和语言特征的融合最能预测快感；但是，对于其他情感预测而言，语言分析的效果要优于面部分析。外部有效性分析部分支持了愤怒和愉悦的预测。讨论：该系统可以检索儿童的情感表达，但需要提高准确性。