Face sketch synthesis, as a key technique for solving face sketch recognition, has made considerable progress in recent years. Due to the difference of modality between face photo and face sketch, traditional exemplar-based methods often lead to missed texture details and deformation while synthesizing sketches. And limited to the local receptive field, Convolutional Neural Networks-based methods cannot deal with the interdependence between features well, which makes the constraint of facial features insufficient; as such, it cannot retain some details in the synthetic image. Moreover, the deeper the network layer is, the more obvious the problems of gradient disappearance and explosion will be, which will lead to instability in the training process. Therefore, in this paper, we propose a multi-scale gradients self-attention residual learning framework for face photo-sketch transformation that embeds a self-attention mechanism in the residual block, making full use of the relationship between features to selectively enhance the characteristics of specific information through self-attention distribution. Simultaneously, residual learning can keep the characteristics of the original features from being destroyed. In addition, the problem of instability in GAN training is alleviated by allowing discriminator to become a function of multi-scale outputs of the generator in the training process. Based on cycle framework, the matching between the target domain image and the source domain image can be constrained while the mapping relationship between the two domains is established so that the tasks of face photo-to-sketch synthesis (FP2S) and face sketch-to-photo synthesis (FS2P) can be achieved simultaneously. Both Image Quality Assessment (IQA) and experiments related to face recognition show that our method can achieve state-of-the-art performance on the public benchmarks, whether using FP2S or FS2P.

中文翻译：

---

用于人脸照片素描变换的多尺度梯度自注意力残差学习


人脸草图合成作为解决人脸草图识别的关键技术，近年来取得了长足的进展。由于人脸照片和人脸草图的模态差异，传统的基于样本的方法在合成草图时经常会导致纹理细节丢失和变形。并且受限于局部感受野，基于卷积神经网络的方法不能很好地处理特征之间的相互依赖关系，这使得面部特征的约束不足；因此，它无法保留合成图像中的一些细节。而且网络层数越深，梯度消失和爆炸的问题就越明显，从而导致训练过程的不稳定。因此，在本文中，我们提出了一种用于人脸照片-素描变换的多尺度梯度自注意力残差学习框架，该框架在残差块中嵌入自注意力机制，充分利用特征之间的关系来选择性地增强特征通过自注意力分布来获取特定信息。同时，残差学习可以保持原始特征的特性不被破坏。此外，通过让判别器在训练过程中成为生成器多尺度输出的函数，缓解了 GAN 训练的不稳定问题。基于循环框架，可以在建立两个域之间的映射关系的同时，约束目标域图像和源域图像之间的匹配，从而实现人脸照片到草图合成（FP2S）和人脸草图到草图的任务-可以同时实现照片合成（FS2P）。 图像质量评估（IQA）和与人脸识别相关的实验都表明，无论使用 FP2S 还是 FS2P，我们的方法都可以在公共基准上实现最先进的性能。