Ensemble single image deraining network via progressive structural boosting constraints

Highlights

• The ensemble network has capacity to solve rain removal of multiple types.

• Struct-absolute loss benefits for embedding boosting strategy into network training.

• The ensemble network achieves new SOTA performance on public deraining datasets.

Abstract

Image deraining is an extensively researched topic in low-level computer vision community. During the past, sufficient works have been proposed to address this problem. Though great improvements these methods have achieved, no derain network can confidently declare that it can solve rain removal problem perfectly. Single complex model may lead to overfitting, while simple model is too weak to achieve clear result. Therefore, in this paper, inspired from classic boosting idea, we have proposed an effective ensemble derain framework to aggregate multiple simple weak drain models to obtain a strong derain model. Cascade structural weighting-map are computed for adaptively emphasizing the quality of local derain regions. Struct-absolution losses are proposed to account for pixel-wise and local region-wise differences, and to facilitate embedding boosting idea into network training. The comprehensive experiments on public derain datasets and high-level vision tasks validate that our proposed model which just utilizes three generally weak derain subnets can achieve much better performance than compared state-of-the-art methods. Our ensemble framework has enough capacity that any state-of-the-art DL-based models can be taken as sub-modules to solve rain removal of multiple types within a single framework.

Introduction

Rainy day is one of the most common weather in our daily life. Images captured on rainy condition often suffer poor visual qualities with rain masks. Under different conditions, the rain-masks may appear different styles, such as linear-shape sparse rain-streaks, randomly scattered raindrops and accumulated hazy rain-mist. The image degradation caused by rain-masks brings great challenge to downstream high-level vision tasks, such as video surveillance, object recognition, auto-driving, etc. Therefore, it is important to develop novel and effective rain removal algorithm to restore image’s clear content. In recent years, with increasing demands of industrial applications, image derain becomes a very hot research topic in computer vision communities.

During the past, many state-of-the-art methods have been proposed to solve the image derain problems. Though great improvements have been achieved, there are several deficiencies that need paying much attention on. On one hand, many derain methods were criticized that they were specifically designed for only one rain-mask, especially for rain-streak. Assuming degradation too much on one rain case may oversimplify the derain problem. On another hand, since image derain is a challenge ill-posed image restoration problem, in most cases, there does not exist a single derain model that is perfect for rain removal. Single complex derain model may lead to overfitting, resulting content details smoothed, while simple general model may be too weak to remove rain clearly. Therefore, some ensemble strategies are needed to combine methods for different rain-masks or methods considering different rain information aspects, taking their complementary advantages.

In this paper, we follow the ensemble learning strategy, and propose an ensemble single image derain network inspired from classic adaboost [1] idea. As we know, adaboost algorithm adaptively gives much emphasis on wrongly classified samples. For the derain problem, we take pixel-wise value and region’s structural consistence as “sample labels”. Therefore, we design cascade weighting map computation modules, and construct progressive structural boosting constraints on pre-trained weak derain subnets. Auxiliary struct-absolute losses (SALoss) are calculated on each subnet for fine-tuning. Then coarsely derained results from each “weak” subnet are aggregated and refined to generate final optimal clear output. We have performed comprehensive experiments on two public derain datasets to evaluate the effectiveness our ensemble framework. Though our initial motivation is to validate the boosting thought on derain nets, the experiment results amazingly demonstrate that our proposed ensemble model just aggregating three weak derain subnets can achieve more superior performance than state-of-the-art methods compared in this paper.

The contributions of this paper are summarized into three folds.

• We have successfully constructed an effective ensemble single deraining network via progressive structural boosting constraints. The ensemble framework has enough capacity that any state-of-the-art derain networks can be taken as its sub-modules to potentially solve rain removal of multiple types within a single framework.

• We have proposed a struct-absolute loss to consider both pixel-wise and local region-wise differences for network training. As auxiliary loss for fine-tuning, it can benefit for embedding boosting into network training.

• We have experimented our proposed ensemble network on public synthesized derain datasets and real-world rainy images. The experiment results demonstrate that our ensemble boosting derain network can achieve much better performance than many state-of-the-art methods, though we just only utilize three pre-trained “weak” derain models as subnets. On high-level vision tasks, our experimented model can also benefit high-level vision algorithms for reaching almost the same results on clear ground-truth.

In the following sections, we will briefly review some related works in Section 2. Then in Section 3, we will describe our ensemble boosting derain framework in details. In Section 4, we will comprehensively demonstrate our experiments. Finally, in Section 5, we give our conclusion and discus the future work.