Abstract This paper proposes a new deep learning-based mobile AR for intelligent task assistance by conducting 3D spatial mapping without pre-registration using AR markers, which can match virtual AR objects to their corresponding physical objects automatically and accurately using single snapshot-based RGB-D data. Firstly, the proposed approach applies a deep learning-based instance segmentation method to the snapshot-based RGB-D data to detect real object instances and to segment their surrounding regions in 3D point cloud data. Then, an iterative closest point (ICP) algorithm is used to perform a 3D spatial mapping between the segmented point cloud of the real object and its corresponding virtual model. Therefore, the virtual information can be seamlessly and automatically synchronized with its corresponding real object. To prove the effectiveness of the proposed method, we performed comparative experiments quantitatively and qualitatively, which evaluated the accuracy, basic task performance, and usability. Experimental results verify that the proposed deep learning-based 3D spatial mapping approach is more accurate and more suitable for mobile AR-based visualization and interaction than previous studies. We have also implemented several applications in actual working situations, which verifies the applicability and extensibility of the proposed approach.

中文翻译：

---

基于深度学习的移动增强现实使用 3D 空间映射和基于快照的 RGB-D 数据进行任务辅助

摘要 本文提出了一种新的基于深度学习的移动 AR 智能任务辅助，通过使用 AR 标记进行 3D 空间映射而无需预注册，它可以使用基于单个快照的 RGB 自动准确地将虚拟 AR 对象与其对应的物理对象匹配。 D 数据。首先，所提出的方法将基于深度学习的实例分割方法应用于基于快照的 RGB-D 数据，以检测真实对象实例并在 3D 点云数据中分割其周围区域。然后，使用迭代最近点 (ICP) 算法在真实物体的分割点云与其对应的虚拟模型之间执行 3D 空间映射。因此，虚拟信息可以与其对应的真实对象无缝自动同步。为了证明所提出方法的有效性，我们进行了定量和定性对比实验，评估了准确性、基本任务性能和可用性。实验结果验证了所提出的基于深度学习的 3D 空间映射方法比以前的研究更准确，更适合基于移动 AR 的可视化和交互。我们还在实际工作场景中实现了多个应用，验证了所提出方法的适用性和可扩展性。实验结果验证了所提出的基于深度学习的 3D 空间映射方法比以前的研究更准确，更适合基于移动 AR 的可视化和交互。我们还在实际工作场景中实现了多个应用，验证了所提出方法的适用性和可扩展性。实验结果验证了所提出的基于深度学习的 3D 空间映射方法比以前的研究更准确，更适合基于移动 AR 的可视化和交互。我们还在实际工作场景中实现了多个应用，验证了所提出方法的适用性和可扩展性。