Deep learning models have been able to generate rain-free images effectively, but the extension of these methods to complex rain conditions where rain streaks show various blurring degrees, shapes, and densities has remained an open problem. Among the major challenges are the capacity to encode the rain streaks and the sheer difficulty of learning multi-scale context features that preserve both global color coherence and exactness of detail. To address the first problem, we design a non-local fusion module (NFM) and an attention fusion module (AFM), and construct the multi-level pyramids’ architecture to explore the local and global correlations of rain information from the rain image pyramid. More specifically, we apply the non-local operation to fully exploit the self-similarity of rain streaks and perform the fusion of multi-scale features along the image pyramid. To address the latter challenge, we additionally design a residual learning branch that is capable of adaptively bridging the gaps (e.g., texture and color information) between the predicted rain-free image and the clean background via a hybrid embedding representation. Extensive results have demonstrated that our proposed method is able to generate much better rain-free images on several benchmark datasets than the state-of-the-art algorithms. Moreover, we conduct the joint evaluation experiments with respect to deraining performance and the detection/segmentation accuracy to further verify the effectiveness of our deraining method for downstream vision tasks/applications. The source code is available at https://github.com/kuihua/MSHFN .

中文翻译：

---

用于单图像去雨的多尺度混合融合网络


深度学习模型已经能够有效地生成无雨图像，但将这些方法扩展到复杂的降雨条件（雨条纹显示出各种模糊程度、形状和密度）仍然是一个悬而未决的问题。主要挑战包括对雨条纹进行编码的能力，以及学习保持全局颜色一致性和细节准确性的多尺度上下文特征的绝对困难。为了解决第一个问题，我们设计了一个非局部融合模块（NFM）和一个注意力融合模块（AFM），并构建了多层金字塔架构来探索雨图像金字塔中雨信息的局部和全局相关性。更具体地说，我们应用非局部操作来充分利用雨条纹的自相似性，并沿着图像金字塔进行多尺度特征的融合。为了解决后一个挑战，我们另外设计了一个残差学习分支，它能够通过混合嵌入表示自适应地弥合预测的无雨图像和干净背景之间的差距（例如，纹理和颜色信息）。大量结果表明，我们提出的方法能够在多个基准数据集上生成比最先进算法更好的无雨图像。此外，我们还进行了去雨性能和检测/分割精度的联合评估实验，以进一步验证我们的去雨方法对下游视觉任务/应用的有效性。源代码可在 https://github.com/kuihua/MSHFN 获取。