Facial expression is one of the most direct and effective ways to recognize emotions, widely used in human‐computer interaction, affective computing, and other research fields. Expression recognition can be divided into discrete expression classification and continuous dimensional emotion recognition. Most of the existing multi‐dimensional emotional estimation only considers the data under laboratory conditions. In this paper, facial emotion estimation is performed based on real‐world images and combined with the advantages of multi‐task learning and attention mechanism. We improve the multi‐task attention network (MTAN) from two aspects: task and feature. At the aspect of the task, the multi‐task collaborative attention network (MTCAN), which is based on task correlation, is proposed to solve task deviation in multi‐task learning. At the aspect of the feature, based on MTCAN, we came up with MTACN, which used the self‐attention mechanism to measure the importance of each attention module for each specific task. Then, we can capture the local‐to‐global connection in one step and fully exploit the feature within different levels of each task. Experimental results on the AffectNet dataset show that the performance of the model is significantly better than the original network, and the Root‐mean‐square error and consistency correlation coefficient results are superior to other existing models. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

中文翻译：

---

多任务和注意力协作网络，用于面部情绪识别

面部表情是识别情绪的最直接，最有效的方法之一，广泛用于人机交互，情感计算和其他研究领域。表情识别可以分为离散表情分类和连续维度情感识别。现有的大多数多维情感估计都只考虑实验室条件下的数据。本文基于真实世界的图像进行面部情感估计，并结合多任务学习和注意力机制的优势。我们从任务和功能两个方面改进了多任务注意力网络（MTAN）。在任务方面，提出了一种基于任务相关性的多任务协作注意力网络（MTCAN），以解决多任务学习中的任务偏差问题。在功能方面，我们基于MTCAN提出了MTACN，它使用了自我注意机制来衡量每个注意模块对每个特定任务的重要性。然后，我们可以一步一步捕获本地到全局的连接，并在每个任务的不同级别充分利用该功能。在AffectNet数据集上的实验结果表明，该模型的性能明显优于原始网络，并且均方根误差和一致性相关系数结果优于其他现有模型。©2021日本电气工程师学会。由Wiley Periodicals LLC发布。我们可以一步一步捕获本地到全局的连接，并充分利用每个任务不同级别的功能。在AffectNet数据集上的实验结果表明，该模型的性能明显优于原始网络，并且均方根误差和一致性相关系数结果优于其他现有模型。©2021日本电气工程师学会。由Wiley Periodicals LLC发布。我们可以一步一步捕获本地到全局的连接，并充分利用每个任务不同级别的功能。在AffectNet数据集上的实验结果表明，该模型的性能明显优于原始网络，并且均方根误差和一致性相关系数结果优于其他现有模型。©2021日本电气工程师学会。由Wiley Periodicals LLC发布。©2021日本电气工程师学会。由Wiley Periodicals LLC发布。©2021日本电气工程师学会。由Wiley Periodicals LLC发布。