Corresponding lighting and reflectance between real and virtual objects is important for spatial presence in augmented and mixed reality (AR and MR) applications. We present a method to reconstruct real-world environmental lighting, encoded as a reflection map (RM), from a conventional photograph. To achieve this, we propose a stacked convolutional neural network (SCNN) that predicts high dynamic range (HDR) 360 ${}^\circ$ RMs with varying roughness from a limited field of view, low dynamic range photograph. The SCNN is progressively trained from high to low roughness to predict RMs at varying roughness levels, where each roughness level corresponds to a virtual object’s roughness (from diffuse to glossy) for rendering. The predicted RM provides high-fidelity rendering of virtual objects to match with the background photograph. We illustrate the use of our method with indoor and outdoor scenes trained on separate indoor/outdoor SCNNs showing plausible rendering and composition of virtual objects in AR/MR. We show that our method has improved quality over previous methods with a comparative user study and error metrics.

中文翻译：

---

使用 Stacked-CNN 重建反射图以进行混合现实渲染

真实和虚拟对象之间的相应照明和反射对于增强和混合现实（AR 和 MR）应用中的空间存在很重要。我们提出了一种从传统照片重建现实世界环境照明的方法，编码为反射图 (RM)。为了实现这一点，我们提出了一个堆叠卷积神经网络 (SCNN)，它可以预测高动态范围 (HDR) 360 ${}^\circ$从有限的视场、低动态范围照片来看，具有不同粗糙度的 RM。SCNN 从高粗糙度到低粗糙度逐步训练，以预测不同粗糙度级别的 RM，其中每个粗糙度级别对应于用于渲染的虚拟对象的粗糙度（从漫反射到光泽）。预测的 RM 提供虚拟对象的高保真渲染以匹配背景照片。我们说明了我们的方法与在单独的室内/室外 SCNN 上训练的室内和室外场景的使用，显示了 AR/MR 中虚拟对象的合理渲染和组合。我们通过比较用户研究和错误度量表明我们的方法比以前的方法提高了质量。