This paper addresses the problem of semi-supervised transfer learning with limited cross-modality data in remote sensing. A large amount of multi-modal earth observation images, such as multispectral imagery (MSI) or synthetic aperture radar (SAR) data, are openly available on a global scale, enabling parsing global urban scenes through remote sensing imagery. However, their ability in identifying materials (pixel-wise classification) remains limited, due to the noisy collection environment and poor discriminative information as well as limited number of well-annotated training images. To this end, we propose a novel cross-modal deep-learning framework, called X-ModalNet, with three well-designed modules: self-adversarial module, interactive learning module, and label propagation module, by learning to transfer more discriminative information from a small-scale hyperspectral image (HSI) into the classification task using a large-scale MSI or SAR data. Significantly, X-ModalNet generalizes well, owing to propagating labels on an updatable graph constructed by high-level features on the top of the network, yielding semi-supervised cross-modality learning. We evaluate X-ModalNet on two multi-modal remote sensing datasets (HSI-MSI and HSI-SAR) and achieve a significant improvement in comparison with several state-of-the-art methods.

本文解决了遥感中有限跨模态数据的半监督迁移学习问题。大量多模态地球观测图像，例如多光谱图像（MSI）或合成孔径雷达（SAR）数据，在全球范围内公开提供，使得能够通过遥感图像解析全球城市场景。然而，由于嘈杂的收集环境和较差的判别信息以及有限数量的注释良好的训练图像，它们识别材料（按像素分类）的能力仍然有限。为此，我们提出了一种新颖的跨模态深度学习框架，称为 X-ModalNet，具有三个精心设计的模块：自对抗模块、交互式学习模块和标签传播模块，通过学习从使用大规模 MSI 或 SAR 数据将小规模高光谱图像 (HSI) 纳入分类任务中。值得注意的是，X-ModalNet 具有良好的泛化能力，因为它在由网络顶部的高级特征构建的可更新图上传播标签，从而产生半监督的跨模态学习。我们在两个多模态遥感数据集（HSI-MSI 和 HSI-SAR）上评估 X-ModalNet，与几种最先进的方法相比，取得了显着的改进。