Despite recent breakthroughs in deep learning methods for image lighting enhancement, they are inferior when applied to portraits because 3D facial information is ignored in their models. To address this, we present a novel deep learning framework for portrait lighting enhancement based on 3D facial guidance. Our framework consists of two stages. In the first stage, corrected lighting parameters are predicted by a network from the input bad lighting image, with the assistance of a 3D morphable model and a differentiable renderer. Given the predicted lighting parameter, the differentiable renderer renders a face image with corrected shading and texture, which serves as the 3D guidance for learning image lighting enhancement in the second stage. To better exploit the long-range correlations between the input and the guidance, in the second stage, we design an image-to-image translation network with a novel transformer architecture, which automatically produces a lighting-enhanced result. Experimental results on the FFHQ dataset and in-the-wild images show that the proposed method outperforms state-of-the-art methods in terms of both quantitative metrics and visual quality.

中文翻译：

---

具有 3D 引导的深度人像照明增强

尽管最近在图像照明增强的深度学习方法方面取得了突破，但它们在应用于人像时表现不佳，因为它们的模型中忽略了 3D 面部信息。为了解决这个问题，我们提出了一种新颖的深度学习框架，用于基于 3D 面部引导的人像照明增强。我们的框架由两个阶段组成。在第一阶段，在 3D 可变形模型和可微渲染器的帮助下，网络从输入的不良照明图像中预测校正后的照明参数。给定预测的光照参数，可微渲染器渲染具有校正阴影和纹理的人脸图像，作为第二阶段学习图像光照增强的 3D 指导。为了更好地利用输入和指导之间的长期相关性，在第二阶段，我们设计了一个具有新颖变压器架构的图像到图像转换网络，该网络会自动产生光照增强的结果。在 FFHQ 数据集和野外图像上的实验结果表明，所提出的方法在定量指标和视觉质量方面均优于最先进的方法。