Multispectral and hyperspectral image fusion (MS/HS fusion) aims to fuse a high-resolution multispectral (HrMS) and a low-resolution hyperspectral (LrHS) images to generate a high-resolution hyperspectral (HrHS) image, which has become one of the most commonly addressed problems for hyperspectral image processing. In this paper, we specifically designed a network architecture for the MS/HS fusion task, called MHF-net, which not only contains clear interpretability, but also reasonably embeds the well studied linear mapping that links the HrHS image to HrMS and LrHS images. In particular, we first construct an MS/HS fusion model which merges the generalization models of low-resolution images and the low-rankness prior knowledge of HrHS image into a concise formulation, and then we build the proposed network by unfolding the proximal gradient algorithm for solving the proposed model. As a result of the careful design for the model and algorithm, all the fundamental modules in MHF-net have clear physical meanings and are thus easily interpretable. This not only greatly facilitates an easy intuitive observation and analysis on what happens inside the network, but also leads to its good generalization capability. Based on the architecture of MHF-net, we further design two deep learning regimes for two general cases in practice: consistent MHF-net and blind MHF-net. The former is suitable in the case that spectral and spatial responses of training and testing data are consistent, just as considered in most of the pervious general supervised MS/HS fusion researches. The latter ensures a good generalization in mismatch cases of spectral and spatial responses in training and testing data, and even across different sensors, which is generally considered to be a challenging issue for general supervised MS/HS fusion methods. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively as compared with state-of-the-art methods along this line of research.

中文翻译：

---

MHF-Net：用于多光谱和高光谱图像融合的可解释深度网络


多光谱和高光谱图像融合（MS/HS fusion）旨在融合高分辨率多光谱（HrMS）和低分辨率高光谱（LrHS）图像以生成高分辨率高光谱（HrHS）图像，该图像已成为多光谱和高光谱图像融合技术之一。高光谱图像处理最常解决的问题。在本文中，我们专门为MS/HS融合任务设计了一种网络架构，称为MHF-net，它不仅具有清晰的可解释性，而且合理地嵌入了经过充分研究的将HrHS图像链接到HrMS和LrHS图像的线性映射。特别是，我们首先构建了一个 MS/HS 融合模型，将低分辨率图像的泛化模型和 HrHS 图像的低秩先验知识合并为一个简洁的公式，然后我们通过展开近端梯度算法来构建所提出的网络用于解决所提出的模型。由于模型和算法的精心设计，MHF-net中的所有基本模块都具有明确的物理意义，因此易于解释。这不仅极大地方便了人们对网络内部发生的情况进行简单直观的观察和分析，而且具有良好的泛化能力。基于MHF-net的架构，我们进一步针对实践中的两种常见情况设计了两种深度学习机制：一致性MHF-net和盲目MHF-net。前者适用于训练数据和测试数据的光谱和空间响应一致的情况，正如大多数以往的一般监督MS/HS融合研究所考虑的那样。 后者确保了训练和测试数据中光谱和空间响应不匹配情况下的良好泛化，甚至跨不同传感器，这通常被认为是一般监督 MS/HS 融合方法的一个具有挑战性的问题。模拟和真实数据的实验结果在视觉和定量上证实了我们的方法与该研究领域最先进的方法相比的优越性。