FW-GAN: Underwater image enhancement using generative adversarial network with multi-scale fusion
Introduction
Since the 21st century, with the rapid development of global science and technology, mankind has accelerated the exploration and resource development of the marine environment, all kinds of underwater robots have wide applications in military and livelihood tasks among high-risk and highly polluted waters, such as underwater archaeology and topographic mapping, marine environment observation, marine resources investigation, marine security defense and marine pasture. Visual perception is one of the important means for underwater robots to capture environmental information, it plays an important role in improving autonomous navigation and intelligent operation of robots [1]. However, underwater environments often tend to be harsh and it is difficult for the robot’s vision sensors to obtain high-quality underwater images. This is mainly because the image quality is seriously affected by the complex physical and chemical conditions of underwater environments. For example, the selective attenuation of atmospheric light usually makes underwater images show a bluish or greenish tone; particles suspended underwater absorb most of the light and change its direction before the reflected light reaches the camera, resulting in the degradation of low contrast, blur and haze of the image. The above nonlinear distortion of underwater images will seriously restrict the performance of high-level vision tasks, such as image segmentation [2], object classification and detection [3]. Therefore, studying high-quality underwater image enhancement methods to overcome the deficiency of visual perception caused by image distortion so that make underwater robots have a better sight is a key scientific problem that needs to be solved urgently [4], [5].
To improve the underwater image quality, the existing methods mainly include model-based image restoration method, non-model-based image enhancement method and deep learning-based image enhancement method [1]. Since the physical and chemical conditions differ greatly in different waters, the physical model-based restoration method needs to obtain the empirical parameters of water conditions in advance, which results in the lack of generalization. Traditional enhancement methods can achieve attractive visual effects by adjusting the distribution of image pixel values. However, due to the lack of an underwater imaging model, some scenes are over-/under-enhanced, and some noises and artifacts are introduced.
In recent years, the theory and technology of deep learning have developed rapidly. Convolutional Neural Network (CNN) and Generative Adversarial Network (GAN) [6] have made encouraging progress in many visual computing tasks, especially in the fields of image super-resolution [7], [8], deblurring [9] and deraining [10]. Meanwhile, the underwater image enhancement method based on deep learning has also made good progress [4], [11], [12], but there is still much room for improvement, especially in the performance and generalization capability of the method. To solve those problems, this paper proposes a multi-scale fusion generative adversarial network named Fusion Water-GAN (FW-GAN) to learn the nonlinear mapping between distorted underwater images and high-quality underwater images. The main contributions of this paper are summarized as follows:
- 1.
We propose a multi-scale fusion generator network architecture. Based on the analysis of the underwater environment, an adaptive fusion strategy is proposed to fuse the multi-source and multi-scale features, which can effectively correct the color casts and haze of the image and improve its contrast, it can also avoid blind enhancement of the image and improve the generalization capability of the model.
- 2.
We propose a decoder model combined with channel attention to compute the attention of the prior and decoded feature maps in the fusion process and adjust them adaptively. The aim is to learn the potential associations between the prior features of fusion and the enhanced results.
- 3.
We conducted qualitative and quantitative evaluations and compared FW-GAN with traditional methods and state-of-the-art deep learning-based models. The results show that FW-GAN can effectively improve the quality of underwater images, it has good generalization capability and competitive performance. We carried out application tests and proved that the proposed method had a positive effect on other visual tasks. Finally, we conduct an ablation study to demonstrate the contribution of each core component in our network.
Section snippets
Related works
In the past few years, many underwater image restoration and enhancement methods have been proposed to improve the quality of underwater images. The existing underwater image quality improvement methods can be divided into the following three categories.
Network architecture
In the proposed FW-GAN network, we designed the generator of encoder–decoder structure based on the U-Net. The difference is we propose to use fusion connections instead of skip-connections between the encoding layer and decoding layer of the corresponding size of the network structure. In order to better fuse various prior features into the image decoding learning process, we designed a decoder combined with channel attention. The fusion connections and channel attention decoder are described
Experiment
In this section, we first describe the implementation details and then introduce the experimental settings. In order to evaluate our method, we compared it with the traditional and the state-of-the-art methods under both synthetic and real-world underwater datasets. Finally, we conducted ablation experiments to verify each core component of FW-GAN.
Conclusion
This paper presents an underwater image enhancement model based on GAN. The model has a generator with multiple inputs and convolution branches. We input three prior images to refine the prior features, and then these features are fused into the encoding and decoding process of the original input image through the multi-scale fusion strategy to obtain the enhanced underwater image. Besides, we designed a channel attention decoder to learn the correlation and dependence between prior features to
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was supported in part by the Key Area Research Projects of Universities of Guangdong Province under Grant 2019KZDZX1026, in part by the Natural Science Foundation of Guangdong Province under Grant 2020A1515110255, in part by the Science and Technology Program of Guangzhou under Grant 202102080591, in part by the Natural Science Foundation of China under Grant 61603103, in part by the Innovation Team Project of Universities of Guangdong Province under Grant 2020KCXTD015.
References (61)
- et al.
Underwater scene prior inspired deep underwater image and video enhancement
Pattern Recognit.
(2020) - et al.
Two-step domain adaptation for underwater image enhancement
Pattern Recognit.
(2022) - et al.
Automatic red-channel underwater image restoration
J. Vis. Commun. Image Represent.
(2015) - et al.
Depth-aware total variation regularization for underwater image dehazing
Signal Process., Image Commun.
(2021) Image enhancement by histogram transformation
Comput. Graph. Image Process.
(1977)Contrast limited adaptive histogram equalization
- et al.
Enhancing underwater image via color correction and bi-interval contrast enhancement
Signal Process., Image Commun.
(2021) - et al.
UIEĈ2-Net: CNN-based underwater image enhancement using two color space
Signal Process., Image Commun.
(2021) - et al.
Coupled tensor decompositions for data fusion
- et al.
Augmenting interictal mapping with neurovascular coupling biomarkers by structured factorization of epileptic EEG and fMRI data
NeuroImage
(2021)
DewaterNet: A fusion adversarial real underwater image enhancement network
Signal Process., Image Commun.
Underwater image restoration via backscatter pixel prior and color compensation
Eng. Appl. Artif. Intell.
Twice mixing: a rank learning based quality assessment approach for underwater image enhancement
Signal Process., Image Commun.
An experimental-based review of image enhancement and image restoration methods for underwater imaging
IEEE Access
Semantic segmentation of underwater imagery using deep networks trained on synthetic imagery
J. Mar. Sci. Eng.
Roimix: Proposal-fusion among multiple images for underwater object detection
Enhancing underwater imagery using generative adversarial networks
Fast underwater image enhancement for improved visual perception
IEEE Robot. Autom. Lett.
Generative adversarial nets
Adv. Neural Inf. Process. Syst.
Fast and accurate image super-resolution with deep laplacian pyramid networks
IEEE Trans. Pattern Anal. Mach. Intell.
Deep stacked hierarchical multi-patch network for image deblurring
Multi-stage progressive image restoration
Single image haze removal using dark channel prior
IEEE Trans. Pattern Anal. Mach. Intell.
Underwater depth estimation and image restoration based on single images
IEEE Comput. Graph. Appl.
Initial results in underwater single image dehazing
A rapid scene depth estimation model based on underwater light attenuation prior for underwater image restoration
Sea-thru: A method for removing water from underwater images
Enhancement of underwater images with statistical model of background light and optimization of transmission map
IEEE Trans. Broadcast.
Cited by (15)
Multi-scale cross-layer feature interaction GAN for underwater image enhancement
2024, Digital Signal Processing: A Review JournalPerceptual loss guided Generative adversarial network for saliency detection
2024, Information SciencesWindow-based transformer generative adversarial network for autonomous underwater image enhancement
2023, Engineering Applications of Artificial IntelligenceWSDS-GAN: A weak-strong dual supervised learning method for underwater image enhancement
2023, Pattern RecognitionCMAF: Cross-Modal Augmentation via Fusion for Underwater Acoustic Image Recognition
2024, ACM Transactions on Multimedia Computing, Communications and ApplicationsA Swift Algorithm and Hue-Preserving Based Mechanism for Underwater Image Colour Enhancement
2024, International Journal of Intelligent Systems and Applications in Engineering