Omnidirectional lighting provides the foundation for achieving spatially-variant photorealistic 3D rendering, a desirable property for mobile augmented reality applications. However, in practice, estimating omnidirectional lighting can be challenging due to limitations such as partial panoramas of the rendering positions, and the inherent environment lighting and mobile user dynamics. A new opportunity arises recently with the advancements in mobile 3D vision, including built-in high-accuracy depth sensors and deep learning-powered algorithms, which provide the means to better sense and understand the physical surroundings. Centering the key idea of 3D vision, in this work, we design an edge-assisted framework called Xihe to provide mobile AR applications the ability to obtain accurate omnidirectional lighting estimation in real time. Specifically, we develop a novel sampling technique that efficiently compresses the raw point cloud input generated at the mobile device. This technique is derived based on our empirical analysis of a recent 3D indoor dataset and plays a key role in our 3D vision-based lighting estimator pipeline design. To achieve the real-time goal, we develop a tailored GPU pipeline for on-device point cloud processing and use an encoding technique that reduces network transmitted bytes. Finally, we present an adaptive triggering strategy that allows Xihe to skip unnecessary lighting estimations and a practical way to provide temporal coherent rendering integration with the mobile AR ecosystem. We evaluate both the lighting estimation accuracy and time of Xihe using a reference mobile application developed with Xihe's APIs. Our results show that Xihe takes as fast as 20.67ms per lighting estimation and achieves 9.4% better estimation accuracy than a state-of-the-art neural network.

中文翻译：

---

Xihe：基于 3D 视觉的移动增强现实照明估计框架

全向照明为实现空间变化的逼真 3D 渲染奠定了基础，这是移动增强现实应用程序的理想特性。然而，在实践中，由于渲染位置的局部全景、固有的环境照明和移动用户动态等限制，估计全向照明可能具有挑战性。随着移动 3D 视觉的进步，最近出现了一个新的机会，包括内置的高精度深度传感器和深度学习驱动的算法，它们提供了更好地感知和理解物理环境的方法。围绕 3D 视觉的关键思想，在这项工作中，我们设计了一个名为 Xihe 的边缘辅助框架，为移动 AR 应用程序提供实时获取准确的全方位照明估计的能力。具体来说，我们开发了一种新颖的采样技术，可以有效地压缩在移动设备上生成的原始点云输入。该技术是基于我们对近期 3D 室内数据集的实证分析得出的，在我们基于 3D 视觉的照明估计器管道设计中起着关键作用。为了实现实时目标，我们为设备上的点云处理开发了量身定制的 GPU 管道，并使用了一种减少网络传输字节的编码技术。最后，我们提出了一种自适应触发策略，允许 Xihe 跳过不必要的照明估计，以及一种提供与移动 AR 生态系统时间相干渲染集成的实用方法。我们使用使用 Xihe 的 API 开发的参考移动应用程序评估 Xihe 的照明估计精度和时间。我们的结果表明，Xihe 的速度最快可达 20。