Facial landmark localization and expression recognition are two important and highly relevant topics in facial analysis. However, few works focus on using the complementary information between the two tasks to improve the performance. In this paper, we propose a residual multi-task learning framework to predict the two tasks simultaneously. Different from previous multi-task learning methods which directly train a deep multi-task network with additional branches and losses, we propose a novel residual learning module to further strengthen the linkages between the two tasks. Benefit from the proposed residual learning module, one task can learn complementary information from the other task, leading to the performance promotion. Another problem for the multi-task learning is the lack of training data with multi-task labels. For example, there is no landmark localization annotation for the two widely-used FER dataset (AffectNet and RAF), vice versa. To solve this problem, we propose an association learning method to further enhance the connection between the two tasks. Based on this connection, the dataset with single-task labels can be used in the multi-task learning. Extensive experiments are conducted on four popular datasets (i.e. 300-W, AFLW for landmark localization and AffectNet, RAF for expression recognition), demonstrating the effectiveness of the proposed algorithm.

面部界标定位和表情识别是面部分析中两个重要且高度相关的主题。但是，很少有工作专注于使用两个任务之间的补充信息来提高性能。在本文中，我们提出了一个残差的多任务学习框架来同时预测两个任务。与以前的直接训练具有更多分支和损失的深层多任务网络的多任务学习方法不同，我们提出了一种新颖的残差学习模块来进一步加强两个任务之间的联系。受益于建议的残差学习模块，一个任务可以从另一任务中学习补充信息，从而提高性能。多任务学习的另一个问题是缺乏带有多任务标签的训练数据。例如，对于两个广泛使用的FER数据集（AffectNet和RAF），没有地标本地化注释，反之亦然。为了解决这个问题，我们提出了一种关联学习方法，以进一步增强两个任务之间的联系。基于此连接，可以将具有单任务标签的数据集用于多任务学习。在四个流行的数据集上进行了广泛的实验（例如300 W，AFLW用于地标定位，AffectNet，RAF用于表情识别），证明了该算法的有效性。