Semantic understanding of 3D environments is critical for both the unmanned system and the human involved virtual/augmented reality (VR/AR) immersive experience. Spatially-sparse convolution, taking advantage of the intrinsic sparsity of 3D point cloud data, makes high resolution 3D convolutional neural networks tractable with state-of-the-art results on 3D semantic segmentation problems. However, the exhaustive computations limits the practical usage of semantic 3D perception for VR/AR applications in portable devices. In this paper, we identify that the efficiency bottleneck lies in the unorganized memory access of the sparse convolution steps, i.e., the points are stored independently based on a predefined dictionary, which is inefficient due to the limited memory bandwidth of parallel computing devices (GPU). With the insight that points are continuous as 2D surfaces in 3D space, a chunk-based sparse convolution scheme is proposed to reuse the neighboring points within each spatially organized chunk. An efficient multi-layer adaptive fusion module is further proposed for employing the spatial consistency cue of 3D data to further reduce the computational burden. Quantitative experiments on public datasets demonstrate that our approach works 11° faster than previous approaches with competitive accuracy. By implementing both semantic and geometric 3D reconstruction simultaneously on a portable tablet device, we demo a foundation platform for immersive AR applications.

中文翻译：

---

用于沉浸式增强现实的实时语义3D感知。

对3D环境的语义理解对于无人系统和人类参与的虚拟/增强现实（VR / AR）沉浸式体验都是至关重要的。空间稀疏卷积利用3D点云数据的固有稀疏性，使得高分辨率3D卷积神经网络易于处理，并具有有关3D语义分割问题的最新结果。然而，详尽的计算限制了语义3D感知在便携式设备中的VR / AR应用中的实际使用。在本文中，我们确定效率瓶颈在于稀疏卷积步骤的无组织存储访问，即，基于预定义的字典独立存储点，由于并行计算设备（GPU）的有限存储带宽，效率低下）。了解到点在3D空间中作为2D曲面是连续的，提出了基于块的稀疏卷积方案，以重用每个空间组织的块中的相邻点。进一步提出了一种有效的多层自适应融合模块，用于采用3D数据的空间一致性提示，以进一步减轻计算负担。在公共数据集上进行的定量实验表明，我们的方法比以前的方法具有竞争性的精度快11°。通过在便携式平板设备上同时实现语义和几何3D重构，我们演示了沉浸式AR应用程序的基础平台。进一步提出了一种有效的多层自适应融合模块，用于采用3D数据的空间一致性提示，以进一步减轻计算负担。在公共数据集上进行的定量实验表明，我们的方法比以前的方法具有竞争性的精度快11°。通过在便携式平板设备上同时实现语义和几何3D重构，我们演示了沉浸式AR应用程序的基础平台。进一步提出了一种有效的多层自适应融合模块，用于采用3D数据的空间一致性提示，以进一步减轻计算负担。在公共数据集上进行的定量实验表明，我们的方法比以前的方法具有竞争性的准确性快11°。通过在便携式平板设备上同时实现语义和几何3D重构，我们演示了沉浸式AR应用程序的基础平台。