The representation of consistent mixed reality (XR) environments requires adequate real and virtual illumination composition in real-time. Estimating the lighting of a real scenario is still a challenge. Due to the ill-posed nature of the problem, classical inverse-rendering techniques tackle the problem for simple lighting setups. However, those assumptions do not satisfy the current state-of-art in computer graphics and XR applications. While many recent works solve the problem using machine learning techniques to estimate the environment light and scene’s materials, most of them are limited to geometry or previous knowledge. This paper presents a CNN-based model to estimate complex lighting for mixed reality environments with no previous information about the scene. We model the environment illumination using a set of spherical harmonics (SH) environment lighting, capable of efficiently represent area lighting. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. Unlike previous CNN-based lighting estimation methods, we propose using a highly optimized deep neural network architecture, with a reduced number of parameters, that can learn high complex lighting scenarios from real-world high-dynamic-range (HDR) environment images. We show in the experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of 7.85$\times$ 10⁻⁴ when comparing SH lighting coefficients. We validate our model in a variety of mixed reality scenarios. Furthermore, we present qualitative results comparing relights of real-world scenes.

一致混合现实 (XR) 环境的表示需要足够的实时和虚拟照明组合。估计真实场景的照明仍然是一个挑战。由于问题的不适定性质，经典的逆向渲染技术解决了简单照明设置的问题。然而，这些假设并不满足当前计算机图形和 XR 应用程序的最新技术水平。虽然最近的许多工作使用机器学习技术来估计环境光和场景材料来解决这个问题，但其中大部分都仅限于几何或先前的知识。本文提出了一种基于 CNN 的模型，用于在没有场景先前信息的混合现实环境中估计复杂照明。我们使用一组球谐函数 (SH) 环境照明对环境照明进行建模，能够有效地表示区域照明。我们提出了一种新的 CNN 架构，可以输入 RGB 图像并实时识别环境照明。与之前基于 CNN 的照明估计方法不同，我们建议使用高度优化的深度神经网络架构，减少参数数量，可以从现实世界的高动态范围 (HDR) 环境图像中学习高度复杂的照明场景。我们在实验中表明，CNN 架构可以以 7.85 的平均均方误差 (MSE) 预测环境照明 与之前基于 CNN 的照明估计方法不同，我们建议使用高度优化的深度神经网络架构，减少参数数量，可以从现实世界的高动态范围 (HDR) 环境图像中学习高度复杂的照明场景。我们在实验中表明，CNN 架构可以以 7.85 的平均均方误差 (MSE) 预测环境照明 与之前基于 CNN 的照明估计方法不同，我们建议使用高度优化的深度神经网络架构，减少参数数量，可以从现实世界的高动态范围 (HDR) 环境图像中学习高度复杂的照明场景。我们在实验中表明，CNN 架构可以以 7.85 的平均均方误差 (MSE) 预测环境照明$\times$10 ^-4比较 SH 照明系数时。我们在各种混合现实场景中验证了我们的模型。此外，我们提出了比较真实世界场景的定性结果。