Image demosaicing and super-resolution are two important tasks in color imaging pipeline. So far they have been mostly independently studied in the open literature of deep learning; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem. In this paper, we propose an end-to-end optimization solution to the JDSR problem and demonstrate its practical significance in computational imaging. Our technical contributions are mainly two-fold. On network design, we have developed a Residual-Dense Squeeze-and-Excitation Networks (RDSEN) supported by a pre-demosaicing network (PDNet) as the pre-processing step. We address the issue of spatio-spectral attention for color-filter-array (CFA) data and discuss how to achieve better information flow by concatenating Residue-Dense Squeeze-and-Excitation Blocks (RDSEBs) for JDSR. Experimental results have shown that significant PSNR/SSIM gain can be achieved by RDSEN over previous network architectures including state-of-the-art RCAN. On perceptual optimization, we propose to leverage the latest ideas including relativistic discriminator and pre-excitation perceptual loss function to further improve the visual quality of textured regions in reconstructed images. Our extensive experiment results have shown that Texture-enhanced Relativistic average Generative Adversarial Network (TRaGAN) can produce both subjectively more pleasant images and objectively lower perceptual distortion scores than standard GAN for JDSR. Finally, we have verified the benefit of JDSR to high-quality image reconstruction from real-world Bayer pattern data collected by NASA Mars Curiosity.

中文翻译：

---

联合去马赛克和超分辨率（JDSR）：网络设计和感知优化

图像去马赛克和超分辨率是彩色成像管道中的两个重要任务。到目前为止，它们在深度学习的公开文献中大多是独立研究的；关于制定联合去马赛克和超分辨率 (JDSR) 问题的潜在好处知之甚少。在本文中，我们提出了 JDSR 问题的端到端优化解决方案，并证明了其在计算成像中的实际意义。我们的技术贡献主要有两个方面。在网络设计方面，我们开发了一个由预去马赛克网络 (PDNet) 支持的残差密集挤压和激励网络 (RDSEN) 作为预处理步骤。我们解决了彩色滤光阵列 (CFA) 数据的空间光谱注意问题，并讨论了如何通过为 JDSR 连接残留密集挤压和激发块 (RDSEB) 来实现更好的信息流。实验结果表明，与包括最先进 RCAN 在内的先前网络架构相比，RDSEN 可以实现显着的 PSNR/SSIM 增益。在感知优化方面，我们建议利用包括相对论鉴别器和预激感知损失函数在内的最新思想来进一步提高重建图像中纹理区域的视觉质量。我们广泛的实验结果表明，与用于 JDSR 的标准 GAN 相比，纹理增强相对论平均生成对抗网络 (TRaGAN) 可以产生主观上更令人愉悦的图像和客观上更低的感知失真分数。最后，我们验证了 JDSR 对从 NASA Mars Curiosity 收集的真实世界拜耳模式数据进行高质量图像重建的好处。