Image relighting has emerged as a problem of significant research interest inspired by augmented reality applications. Physics-based traditional methods, as well as black box deep learning models, have been developed. The existing deep networks have exploited training to achieve a new state of the art; however, they may perform poorly when training is limited or does not represent problem phenomenology, such as the addition or removal of dense shadows. We propose a model which enriches neural networks with physical insight. More precisely, our method generates the relighted image with new illumination settings via two different strategies and subsequently fuses them using a weight map (w). In the first strategy, our model predicts the material reflectance parameters (albedo) and illumination/geometry parameters of the scene (shading) for the relit image (we refer to this strategy as intrinsic image decomposition (IID)). The second strategy is solely based on the black box approach, where the model optimizes its weights based on the ground-truth images and the loss terms in the training stage and generates the relit output directly (we refer to this strategy as direct). While our proposed method applies to both one-to-one and any-to-any relighting problems, for each case we introduce problem-specific components that enrich the model performance: 1) For one-to-one relighting we incorporate normal vectors of the surfaces in the scene to adjust gloss and shadows accordingly in the image. 2) For any-to-any relighting, we propose an additional multiscale block to the architecture to enhance feature extraction. Experimental results on the VIDIT 2020 and the VIDIT 2021 dataset (used in the NTIRE 2021 relighting challenge) reveals that our proposal can outperform many state-of-the-art methods in terms of well-known fidelity metrics and perceptual loss.

中文翻译：

---

灵感来自于物理的密集融合网络

在增强现实应用的启发下，图像重新照明已成为一个具有重大研究兴趣的问题。已经开发了基于物理的传统方法以及黑盒深度学习模型。现有的深层网络已利用培训来达到新的水平。但是，当训练受限或不代表问题现象时（例如添加或删除密集阴影），它们的性能可能会很差。我们提出了一个通过物理洞察丰富神经网络的模型。更准确地说，我们的方法通过两种不同的策略使用新的照明设置生成重新照明的图像，然后使用权重图（w）对其进行融合。在第一个策略中 我们的模型预测了已照明图像的材质反射率参数（反照率）和场景的照明/几何参数（阴影）（我们将此策略称为固有图像分解（IID））。第二种策略仅基于黑匣子方法，该模型在训练阶段基于地面真实图像和损耗项优化权重，并直接生成照明输出（我们将此策略称为直接）。虽然我们提出的方法适用于一对一和任意一对重新照明问题，但是对于每种情况，我们都引入了特定于问题的组件，它们丰富了模型的性能：1）对于一对一重新照明，我们结合了法线向量场景中的表面以相应地调整图像中的光泽和阴影。2）对于任何点到点的重新照明，我们为架构提出了一个额外的多尺度模块，以增强特征提取。在VIDIT 2020和VIDIT 2021数据集（用于NTIRE 2021照明挑战）上的实验结果表明，就众所周知的保真度指标和感知损失而言，我们的建议可以胜过许多最新方法。