Although deep learning has revolutionized remote sensing (RS) image scene classification, current deep learning-based approaches highly depend on the massive supervision of predetermined scene categories and have disappointingly poor performance on new categories that go beyond predetermined scene categories. In reality, the classification task often has to be extended along with the emergence of new applications that inevitably involve new categories of RS image scenes, so how to make the deep learning model own the inference ability to recognize the RS image scenes from unseen categories, which do not overlap the predetermined scene categories in the training stage, becomes incredibly important. By fully exploiting the RS domain characteristics, this paper constructs a new remote sensing knowledge graph (RSKG) from scratch to support the inference recognition of unseen RS image scenes. To improve the semantic representation ability of RS-oriented scene categories, this paper proposes to generate a Semantic Representation of scene categories by representation learning of RSKG (SR-RSKG). To pursue robust cross-modal matching between visual features and semantic representations, this paper proposes a novel deep alignment network (DAN) with a series of well-designed optimization constraints, which can simultaneously address zero-shot and generalized zero-shot RS image scene classification. Extensive experiments on one merged RS image scene dataset, which is the integration of multiple publicly open datasets, show that the recommended SR-RSKG obviously outperforms the traditional knowledge types (e.g., natural language processing models and manually annotated attribute vectors), and our proposed DAN shows better performance compared with the state-of-the-art methods under both the zero-shot and generalized zero-shot RS image scene classification settings. The constructed RSKG will be made publicly available along with this paper (https://github.com/kdy2021/SR-RSKG).

尽管深度学习已经彻底改变了遥感 (RS) 图像场景分类，但当前基于深度学习的方法高度依赖于对预定场景类别的大规模监督，并且在超出预定场景类别的新类别上的表现令人失望。现实中，分类任务往往要随着新应用的出现而扩展，不可避免地涉及到RS图像场景的新类别，那么如何让深度学习模型拥有从看不见的类别中识别RS图像场景的推理能力，在训练阶段不重叠预先确定的场景类别变得非常重要。通过充分利用 RS 域的特性，本文从头构建了一个新的遥感知识图谱（RSKG），以支持对未见过的 RS 图像场景的推理识别。为了提高面向RS的场景类别的语义表示能力，本文提出通过RSKG的表示学习（SR-RSKG）生成场景类别的语义表示。为了追求视觉特征和语义表示之间的鲁棒跨模态匹配，本文提出了一种新颖的深度对齐网络（DAN），它具有一系列精心设计的优化约束，可以同时解决零镜头和广义零镜头 RS 图像场景分类。在一个合并的 RS 图像场景数据集上进行了大量实验，这是多个公开开放的数据集的集成，表明推荐的 SR-RSKG 明显优于传统知识类型（例如，自然语言处理模型和手动注释的属性向量），并且我们提出的 DAN 在零镜头和广义零镜头 RS 图像场景分类设置。构建的 RSKG 将与本文（https://github.com/kdy2021/SR-RSKG）一起公开。