The depth maps obtained by the consumer-level sensors are always noisy in the low-resolution (LR) domain. Existing methods for the guided depth super-resolution, which are based on the pre-defined local and global models, perform well in general cases (e.g., joint bilateral filter and Markov random field). However, such model-based methods may fail to describe the potential relationship between RGB-D image pairs. To solve this problem, this paper proposes a data-driven approach based on the deep convolutional neural network with global and local residual learning. It progressively upsamples the LR depth map guided by the high-resolution intensity image in multiple scales. A global residual learning is adopted to learn the difference between the ground truth and the coarsely upsampled depth map, and the local residual learning is introduced in each scale-dependent reconstruction sub-network. This scheme can restore the depth structure from coarse to fine via multi-scale frequency synthesis. In addition, batch normalization layers are used to improve the performance of depth map denoising. Our method is evaluated in noise-free and noisy cases. A comprehensive comparison against 17 state-of-the-art methods is carried out. The experimental results show that the proposed method has faster convergence speed as well as improved performances based on the qualitative and quantitative evaluations.

中文翻译：

---

基于深度残差网络的引导深度图超分辨率多尺度频率重建

消费级传感器获得的深度图在低分辨率 (LR) 域中始终存在噪声。现有的基于预定义局部和全局模型的引导深度超分辨率方法在一般情况下（例如，联合双边滤波器和马尔可夫随机场）表现良好。然而，这种基于模型的方法可能无法描述 RGB-D 图像对之间的潜在关系。为了解决这个问题，本文提出了一种基于深度卷积神经网络的数据驱动方法，具有全局和局部残差学习。它在多个尺度上逐步对高分辨率强度图像引导的 LR 深度图进行上采样。采用全局残差学习来学习地面实况和粗略上采样的深度图之间的差异，并且在每个尺度相关的重建子网络中引入了局部残差学习。该方案可以通过多尺度频率合成将深度结构从粗到细还原。此外，批量归一化层用于提高深度图去噪的性能。我们的方法在无噪声和有噪声的情况下进行评估。对 17 种最先进的方法进行了全面比较。实验结果表明，基于定性和定量评估，所提出的方法具有更快的收敛速度和改进的性能。我们的方法在无噪声和有噪声的情况下进行评估。对 17 种最先进的方法进行了全面比较。实验结果表明，基于定性和定量评估，所提出的方法具有更快的收敛速度和改进的性能。我们的方法在无噪声和有噪声的情况下进行评估。对 17 种最先进的方法进行了全面比较。实验结果表明，基于定性和定量评估，所提出的方法具有更快的收敛速度和改进的性能。