Online semantic 3D segmentation in company with real-time RGB-D reconstruction poses special challenges such as how to perform 3D convolution directly over the progressively fused 3D geometric data, and how to smartly fuse information from frame to frame. We propose a novel fusion-aware 3D point convolution which operates directly on the geometric surface being reconstructed and exploits effectively the inter-frame correlation for high quality 3D feature learning. This is enabled by a dedicated dynamic data structure which organizes the online acquired point cloud with global-local trees. Globally, we compile the online reconstructed 3D points into an incrementally growing coordinate interval tree, enabling fast point insertion and neighborhood query. Locally, we maintain the neighborhood information for each point using an octree whose construction benefits from the fast query of the global tree.Both levels of trees update dynamically and help the 3D convolution effectively exploits the temporal coherence for effective information fusion across RGB-D frames.

中文翻译：

---

用于在线语义 3D 场景分割的融合感知点卷积

结合实时 RGB-D 重建的在线语义 3D 分割提出了特殊的挑战，例如如何直接在渐进融合的 3D 几何数据上执行 3D 卷积，以及如何巧妙地将信息逐帧融合。我们提出了一种新颖的融合感知 3D 点卷积，它直接在被重建的几何表面上运行，并有效地利用帧间相关性进行高质量 3D 特征学习。这是通过专用的动态数据结构实现的，该结构使用全局-本地树组织在线获取的点云。在全球范围内，我们将在线重建的 3D 点编译成增量增长的坐标区间树，从而实现快速点插入和邻域查询。在当地，