The reconstruction of a high resolution image given a low resolution observation is an ill-posed inverse problem in imaging. Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a high-resolution output. Unlike existing deep multimodal models that do not incorporate domain knowledge about the problem, we propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information; therefore, the proposed neural network is interpretable by design. The deep unfolding architecture is used as a core component of a multimodal framework for guided image super-resolution. An alternative multimodal design is investigated by employing residual learning to improve the training efficiency. The presented multimodal approach is applied to super-resolution of near-infrared and multi-spectral images as well as depth upsampling using RGB images as side information. Experimental results show that our model outperforms state-of-the-art methods.

中文翻译：

---

用于引导图像超分辨率的多模态深度展开。


在低分辨率观察的情况下重建高分辨率图像是成像中的不适定逆问题。深度学习方法依靠训练数据来学习从低分辨率输入到高分辨率输出的端到端映射。与不结合有关问题的领域知识的现有深度多模态模型不同，我们提出了一种多模态深度学习设计，它结合了稀疏先验，并允许将来自另一种图像模态的信息有效地集成到网络架构中。我们的解决方案依赖于一种新颖的深度展开算子，执行类似于带有辅助信息的卷积稀疏编码的迭代算法的步骤；因此，所提出的神经网络在设计上是可解释的。深度展开架构被用作引导图像超分辨率多模态框架的核心组件。通过采用残差学习来研究替代的多模态设计以提高训练效率。所提出的多模态方法适用于近红外和多光谱图像的超分辨率以及使用 RGB 图像作为辅助信息的深度上采样。实验结果表明，我们的模型优于最先进的方法。