The automatic semantic segmentation of the huge amount of acquired remote sensing data has become an important task in the last decade. Images and Point Clouds (PCs) are fundamental data representations, particularly in urban mapping applications. Textured 3D meshes integrate both data representations geometrically by wiring the PC and texturing the surface elements with available imagery. We present a mesh-centered holistic geometry-driven methodology that explicitly integrates entities of imagery, PC and mesh. Due to its integrative character, we choose the mesh as the core representation that also helps to solve the visibility problem for points in imagery. Utilizing the proposed multi-modal fusion as the backbone and considering the established entity relationships, we enable the sharing of information across the modalities imagery, PC and mesh in a twofold manner: (i) feature transfer and (ii) label transfer. By these means, we achieve to enrich feature vectors to multi-modal feature vectors for each representation. Concurrently, we achieve to label all representations consistently while reducing the manual label effort to a single representation. Consequently, we facilitate to train machine learning algorithms and to semantically segment any of these data representations – both in a multi-modal and single-modal sense. The paper presents the association mechanism and the subsequent information transfer, which we believe are cornerstones for multi-modal scene analysis. Furthermore, we discuss the preconditions and limitations of the presented approach in detail. We demonstrate the effectiveness of our methodology on the ISPRS 3D semantic labeling contest (Vaihingen 3D) and a proprietary data set (Hessigheim 3D).

在过去的十年中，对大量获取的遥感数据进行自动语义分割已成为一项重要任务。图像和点云（PC）是基本的数据表示形式，尤其是在城市制图应用程序中。带纹理的3D网格通过连接PC并用可用图像对表面元素进行纹理化，将两种数据表示形式几何地整合在一起。我们提出了一种以网格为中心的整体几何驱动方法，该方法明确地集成了图像，PC和网格的实体。由于其综合性，我们选择网格作为核心表示形式，这也有助于解决图像中点的可见性问题。利用拟议的多模式融合作为骨干并考虑已建立的实体关系，我们可以跨模式图像共享信息，PC和网格有两种方式：（i）特征转移和（ii）标签转移。通过这些手段，我们实现了将特征向量丰富为每种表示形式的多峰特征向量。同时，我们实现了一致地标记所有表示，同时将手动标记工作减少到单个表示。因此，我们有利于训练机器学习算法，并在多模式和单模式意义上对这些数据表示进行语义上的细分。本文介绍了关联机制和随后的信息传递，我们认为它们是多模式场景分析的基石。此外，我们详细讨论了所提出方法的前提条件和局限性。