In this paper, we propose a deep learning based approach for facial action unit (AU) detection by enhancing and cropping regions of interest of face images. The approach is implemented by adding two novel nets (a.k.a. layers): the enhancing layers and the cropping layers, to a pretrained convolutional neural network (CNN) model. For the enhancing layers (noted as E-Net ), we have designed an attention map based on facial landmark features and apply it to a pretrained neural network to conduct enhanced learning. For the cropping layers (noted as C-Net ), we crop facial regions around the detected landmarks and design individual convolutional layers to learn deeper features for each facial region. We then combine the E-Net and the C-Net to construct a so-called Enhancing and Cropping Net ( EAC-Net ), which can learn both features enhancing and region cropping functions effectively. The EAC-Net integrates three important elements, i.e., learning transfer, attention coding, and regions of interest processing, making our AU detection approach more efficient and more robust to facial position and orientation changes. Our approach shows a significant performance improvement over the state-of-the-art methods when tested on the BP4D and DISFA AU datasets. The EAC-Net with a slight modification also shows its potentials in estimating accurate AU intensities. We have also studied the performance of the proposed EAC-Net under two very challenging conditions: (1) faces with partial occlusion and (2) faces with large head pose variations. Experimental results show that (1) the EAC-Net learns facial AUs correlation effectively and predicts AUs reliably even with only half of a face being visible, especially for the lower half; (2) Our EAC-Net model also works well under very large head poses, which outperforms significantly a compared baseline approach. It further shows that the EAC-Net works much better without a face frontalization than with face frontalization through image warping as pre-processing, in terms of computational efficiency and AU detection accuracy.

中文翻译：

---

EAC-Net：具有增强和裁剪功能的深层网络，用于面部动作单元检测

在本文中，我们提出了一种基于深度学习的面部动作单元（AU）检测方法，该方法通过增强和裁剪面部图像的关注区域来实现。该方法是通过在预训练卷积神经网络（CNN）模型中添加两​​个新的网络（又称为层）：增强层和裁剪层来实现的。对于增强层（称为 网络 ），我们基于面部标志性特征设计了一个注意力图，并将其应用于预先训练的神经网络以进行增强的学习。对于裁剪层（标记为 网络 ），我们在检测到的地标周围裁剪面部区域，并设计单独的卷积层以了解每个面部区域的更深特征。然后，我们将E-Net和C-Net结合起来，构建了一个所谓的增强和裁剪网络（ EAC网络 ），可以有效地学习特征增强和区域裁剪功能。EAC-Net集成了三个重要元素，即学习转移，注意力编码和兴趣区域处理，从而使我们的AU检测方法对面部位置和方向变化更有效且更可靠。当在BP4D和DISFA AU数据集上进行测试时，我们的方法显示出与最新方法相比显着的性能改进。稍作修改的EAC-Net也显示了其在估计准确AU强度方面的潜力。我们还研究了提出的EAC-Net在两个非常具有挑战性的条件下的性能：（1）局部遮挡的面部和（2）头部姿势变化较大的面部。实验结果表明：（1）EAC-Net能够有效地学习面部AU的相关性，并且即使只有一半的面部可见，也能可靠地预测AU，尤其是下半部分。（2）我们的EAC-Net模型在非常大的头部姿势下也能很好地工作，这大大优于比较基准方法。它进一步表明，在计算效率和AU检测精度方面，EAC-Net在没有人脸正面化的情况下比通过图像扭曲作为预处理进行人脸正面化要好得多。