With the extremely rapid advances in remote sensing (RS) technology, a great quantity of Earth observation (EO) data featuring considerable and complicated heterogeneity are readily available nowadays, which renders researchers an opportunity to tackle current geoscience applications in a fresh way. With the joint utilization of EO data, much research on multimodal RS data fusion has made tremendous progress in recent years, yet these developed traditional algorithms inevitably meet the performance bottleneck due to the lack of the ability to comprehensively analyze and interpret strongly heterogeneous data. Hence, this non-negligible limitation further arouses an intense demand for an alternative tool with powerful processing competence. Deep learning (DL), as a cutting-edge technology, has witnessed remarkable breakthroughs in numerous computer vision tasks owing to its impressive ability in data representation and reconstruction. Naturally, it has been successfully applied to the field of multimodal RS data fusion, yielding great improvement compared with traditional methods. This survey aims to present a systematic overview in DL-based multimodal RS data fusion. More specifically, some essential knowledge about this topic is first given. Subsequently, a literature survey is conducted to analyze the trends of this field. Some prevalent sub-fields in the multimodal RS data fusion are then reviewed in terms of the to-be-fused data modalities, i.e., spatiospectral, spatiotemporal, light detection and ranging-optical, synthetic aperture radar-optical, and RS-Geospatial Big Data fusion. Furthermore, We collect and summarize some valuable resources for the sake of the development in multimodal RS data fusion. Finally, the remaining challenges and potential future directions are highlighted.

随着遥感（RS）技术的飞速发展，如今大量具有相当复杂的异质性的地球观测（EO）数据变得容易获得，这为研究人员提供了以全新方式应对当前地球科学应用的机会。近年来，随着EO数据的联合利用，多模态RS数据融合的研究取得了长足的进步，但这些发展起来的传统算法由于缺乏对强异构数据的综合分析和解释能力，不可避免地遇到了性能瓶颈。因此，这种不可忽略的限制进一步引起了对具有强大处理能力的替代工具的强烈需求。深度学习（DL）作为一项前沿技术，由于其令人印象深刻的数据表示和重建能力，在众多计算机视觉任务中取得了显着突破。自然，它已经成功地应用于多模态RS数据融合领域，与传统方法相比有了很大的改进。本调查旨在对基于 DL 的多模态 RS 数据融合进行系统概述。更具体地说，首先给出了有关该主题的一些基本知识。随后，进行了文献调查，分析了该领域的发展趋势。然后根据待融合的数据模态，即空间光谱、时空、光探测和测距光学、合成孔径雷达光学和 RS-Geospatial Big数据融合。此外，为了多模态RS数据融合的发展，我们收集和总结了一些有价值的资源。最后，强调了剩余的挑战和潜在的未来方向。