We present neural compositing, a deep-learning based method for augmented reality rendering, which uses convolutional neural networks to composite rendered layers of a virtual object with a real photograph to emulate shadow and reflection effects. The method starts from estimating the lighting and roughness information from the photograph using neural networks, renders the virtual object with a virtual floor into color, shadow and reflection layers by applying the estimated lighting, and finally refines the reflection and shadow layers using neural networks and blends them with the color layer and input image to yield the output image. We assume low-frequency lighting environments and adopt PRT (precomputed radiance transfer) for layer rendering, which makes the whole pipeline differentiable and enables fast end-to-end network training with synthetic scenes. Working on a single photograph, our method can produce realistic reflections in a real scene with spatially-varying material and cast shadows on background objects with unknown geometry and material at real-time frame rates.

我们介绍了神经合成，这是一种基于深度学习的增强现实渲染方法，该方法使用卷积神经网络将虚拟对象的渲染层与真实照片合成，以模拟阴影和反射效果。该方法从使用神经网络估计照片中的照明和粗糙度信息开始，通过应用估计的照明将具有虚拟地板的虚拟对象渲染为颜色，阴影和反射层，最后使用神经网络和图像优化反射和阴影层。将它们与颜色层和输入图像混合以产生输出图像。我们假设使用低频照明环境，并采用PRT（预计算辐射传递）进行图层渲染，这样可以使整个管道具有差异性，并可以使用合成场景进行快速的端到端网络训练。通过处理单张照片，我们的方法可以使用空间变化的材质在真实场景中产生逼真的反射，并以实时帧速率在具有未知几何形状和材质的背景对象上投射阴影。