Elsevier

Ad Hoc Networks

Volume 107, 1 October 2020, 102272
Ad Hoc Networks

A residual network framework based on weighted feature channels for multispectral image compression

https://doi.org/10.1016/j.adhoc.2020.102272Get rights and content

Abstract

Deep learning has achieved great success in many computer vision tasks, such as recognition, image enhancement and image compression. However, it is difficult to use a residual network that can efficiently consider the characteristics of multispectral images for multispectral image compression. In this paper, a novel end-to-end multispectral image compression framework based on a weighted feature channel residual network is proposed to efficiently remove the spatial and spectral redundancy of multispectral images by extracting the importance of each channel. The multispectral image compression framework includes a forward coding network, a rate-distortion optimizer, a quantizer/inverse quantizer, an entropy encoder/decoder and an inverse decoding network. In the encoder, multispectral images are directly fed into the forward coding network, and the main spectral and spatial features of the multispectral images are extracted by the residual block. Additionally, the weighted feature channel module can explicitly model the relationship between feature channels when extracting features from multispectral images and adaptively allocate different weights for each feature channel through training. The rate-distortion optimizer is added to make the main features compact. Then, the intermediate feature data are quantized and encoded by lossless entropy coding to obtain a code stream. In the decoder, the code stream is approximately restored to the intermediate features through the entropy decoder and the inverse quantizer. Then, the intermediate features are reconstructed to the multispectral images by the inverse decoding network. Experimental results on 7-band multispectral images of the Landsat 8 satellite and 8-band multispectral images of WorldView-3 satellite demonstrate that the proposed algorithm can achieve a better PSNR than conventional 2D schemes (which are JPEG2000 and JPEG in this paper) and 3D scheme (which is 3D-SPIHT in this paper) and can effectively preserve more spectral information of multispectral images.

Introduction

Multispectral data are widely used in many scientific and military fields, such as target detection, material identification, environmental monitoring, mineral exploration, and military surveillance [1], due to abundant spatial and spectral information. With the rapid development of multispectral image sensors, they generate a very large amount of data and lead to great challenges in data transmission and storage [2]. Therefore, efficient compression is of great importance for multispectral data.

Traditional multispectral image compression algorithms include predictive coding [3], vector quantization coding [4], and transform coding [5,6]. In [3], a pixel can be predicted by its spatial and spectral neighbors, and then the errors of predicted pixels are relatively small and easier to compress. For vector quantization coding, multispectral image data are decomposed into a set of vectors, and then each vector is quantized and encoded to achieve compression. These traditional multispectral image compression algorithms can efficiently remove spatial and spectral redundancy and obtain a better compression ratio. However, these algorithms also have obvious shortcomings. Prediction coding algorithms can achieve lossless compression, but the compression rate is relatively low. Vector quantization algorithms are computationally expensive for practical applications. Transform coding algorithms could give rise to some block effects and edge Gibbs effects when the compression ratio is large. Therefore, it is a rather challenging task to make full use of multispectral image features to remove the spatial and spectral redundancy of a multispectral image and achieve high-quality image compression.

Recently, some high-quality compression methods of multispectral images including lossless and lossy compression are proposed and these methods can be roughly divided into two categories including hardware-based compression methods [7], [8], [9] and software-based compression methods [10], [11], [12], [13], [14]. Due to the real-time requirements and storage limitation, the implementations of hardware accelerators are relatively convenient. In [7], [8], [9], the lossless compression methods based on FPGAs are proposed, the characteristics of these methods are lower complexity and inexpensive space-grade electronic devices. Similarly, the software-based compression methods also achieve superior performance and lower complexity. In [10,11], the distributed source-coding principle is adapted to promote the transmission of images and achieves efficient compression performance. In [12,13], the promoted wavelet transform is used to improve the compression performance and reduces computational cost. In [14], the onground convolutional neural networks are coupled with the Consultative Committee for Space Data Systems to achieve superior performance, which indicates the potential of the convolutional neural networks in multispectral compression.

Deep learning, e.g., convolutional neural networks (CNNs) and residual networks (RNs), has achieved great success in image compression [15], [16], [17], [18]. In [15], [16], [17], [18], some algorithms based on deep learning techniques are developed to encode input images and reconstruct images. The errors or distortions caused during training are optimized by a unified end-to-end learning algorithm. For RGB image compression, Toderici et al. [15,16] propose a recurrent neural network to generate bitstreams and reconstruct the image. The network can adjust the compression ratio of the image by controlling the number of iterations. For holographic image compression, Jiao et al. [17] develop a convolutional neural network to remove artifacts generated by JPEG compressed holographic images and resolve the image quality degradation caused by the loss of some high-frequency features during compression. For light field image compression, Bakir et al. [18] use deep learning at the decoding end to reconstruct light field images from the sparse images obtained at the encoding end. The abovementioned methods yield better compression performance than JPEG [19] and JPEG2000 [20] in both objective and subjective performance evaluations.

RGB images contain red, green and blue bands, so RGB images can be viewed as simple multispectral images with 3 bands. Compared with RGB images, multispectral images contain richer spectral information, so the spectral information of multispectral images can be extracted as the feature information for the encoder. Convolutional neural networks can directly extract spatial-spectral features due to the local receptive field. Therefore, multispectral image compression algorithms can be extended by RGB image compression algorithms based on convolutional neural networks. Inspired by the excellent performance of convolutional neural networks for RGB image compression, a multispectral image compression algorithm, termed a residual network based on weighted feature channels, is proposed. The overall framework of this algorithm includes a forward coding network, a rate-distortion optimizer, a quantizer/inverse quantizer, an entropy encoder/decoder and an inverse decoding network. In the encoder, multispectral images are directly fed into the forward coding network, and the main spectral and spatial features of the multispectral images are extracted by the residual block. According to their different importance, different feature channels are adaptively assigned different weights in the weighted feature channel module. The rate-distortion optimizer, which is added between the forward coding network and the quantizer, makes the extracted multispectral feature data more compact so that the entropy encoder can be more effective in compressing multispectral images. Then, the intermediate feature data are quantized and encoded by lossless entropy coding to obtain a code stream. In the decoder, the code stream is decoded by the entropy decoder, the inverse quantizer and the inverse decoding network to reconstruct multispectral images. The results validate that the proposed algorithm has superior performance than conventional 2D schemes (which are JPEG2000 and JPEG in this paper) and 3D scheme (which is 3D-SPIHT in this paper) and can effectively preserve more spectral information of multispectral images.

The contributions of the paper can be summarized as follows:

  • (1)

    The proposed compression framework is compatible with raw multispectral data, which means that the multispectral images can be directly fed into the framework without pretreatment such as registration, rectification and calibration, and the uncalibrated data can be directly tested on the compression framework.

  • (2)

    The proposed compression framework is an end-to-end framework. In the training phase, the features of multispectral images can be adaptively extracted by the encoder avoiding errors caused by manual feature extraction. During testing, the compressed stream and the reconstructed multispectral images can be directly obtained after test images are fed into the framework.

  • (3)

    The weights of different channels can be adaptively assigned during feature extraction. Most of the current algorithms treat each channel as equally important, however, the importance of different channels are different for the reconstruction of multispectral images, the framework can distinguish different importance by further processing features extracted by the residual block based on weighted feature channels.

  • (4)

    Lower complexity of test images. Multispectral image transmission is real-time with limited transmission time, requiring compression system to process data quickly. The proposed compression framework can process images in batches and quickly generate compressed stream and reconstructed images.

This paper is organized as follows. Section 2 introduces the proposed compression framework. Section 3 presents the training parameter settings and the solutions to train the compression framework. Section 4 provides the experimental results. Finally, we conclude in Section 5.

Section snippets

The proposed compression framework

The proposed multispectral image compression framework is shown in Fig. 1. Multispectral images are fed directly into the forward coding network without pretreatment such as registration, rectification and calibration, and the input multispectral images are not divided into blocks (e.g. 8 × 8), which reduces the occurrence of block effects. The main spectral and spatial features of the multispectral images are extracted by the residual block, which is the basic component of the forward coding

Datasets for training and testing

The dataset is derived from multispectral images obtained by the Landsat 8 satellite, which contain 7 bands. The Landsat 8 satellite includes Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS), and the original images size is different in different areas. The first seven bands are selected and synthesized into a whole multispectral image and then it is divided into blocks of the specified size, such as 128 × 128 and 512 × 512, as training images and test images, respectively. We

Evaluation of results

To evaluate the performance of the compression framework, objective quality evaluation and spectral information loss evaluation of the reconstructed images are introduced for comparison with JPEG, JPEG2000 and 3D-SPIHT algorithms. The peak signal-to-noise ratio (PSNR) is used to measure the quality of the reconstructed images. The proposed algorithm is compared with JPEG, JPEG2000 and 3D-SPIHT at 8 different bit rates.

Fig. 5 shows the average PSNR of the 7-band test images at different bit

Conclusion

In this paper, an effective residual framework based on a forward coding network and an inverse decoding network is proposed to compress multispectral images. The forward coding network is used to extract features based on weighted feature channels and generate a compact representation that preserves more useful information for image reconstruction. The inverse decoding network, which is symmetrical to the forward coding network, is used to reconstruct the decoded image with high quality.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by a grant from the National Natural Science Foundation of China (61801214). The authors would also like to thank the anonymous reviewers for their helpful comments regarding the improvement of this paper.

Fanqiang Kong received the Ph.D. degree in information and communication engineering from Xidian University, Xi'an, China, in 2008. He is currently an associate professor at the College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing, China. His research interests include spectral image coding and image analysis, artificial intelligence, and pattern recognition.

References (28)

  • L. Mao et al.

    Optical element surface defect measurement based on multispectral technique

    Chin. J. Lasers

    (2017)
  • X. Liu et al.

    NOMA-based resource allocation for cluster-based cognitive industrial internet of things

    IEEE Trans. Ind. Inf.

    (2020)
  • Y. Li et al.

    Lossless compression of hyperspectral images based on the prediction error block

    Int. Soc. Opt. Photonics

    (2016)
  • C.F. Lee et al.

    A survey of data hiding based on vector quantization.

    Advances in Intelligent Information Hiding and Multimedia Signal Processing

    (2020)
  • S. Shafie et al.

    Zero-padding in DWT satellite image compression

    Pertanika J. Sci. Technol.

    (2017)
  • Y. Nian et al.

    Pairwise KLT-based compression for multispectral images

    Sens. Imaging

    (2016)
  • J. Fjeldtvedt et al.

    An efficient real-time FPGA implementation of the CCSDS-123 compression standard for hyperspectral image

    IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.

    (2018)
  • D. Bascones et al.

    FPGA implementation of the CCSDS 1.2.3 standard for real-time hyperspectral lossless compression

    IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.

    (2017)
  • A. Rodriguez et al.

    Scalable Hardware-based on-board processing for run-time adaptive lossless hyperspectral compression

    IEEE Access

    (2019)
  • J. Zhang et al.

    Distributed lossless coding of hyperspectral images

    IEEE J. Sel. Top. Signal Process.

    (2015)
  • A. Abrardo et al.

    Error-resilient and low-complexity onboard lossless compression of hyperspectral images by means of distributed source coding

    IEEE Trans. Geosci. Remote Sens.

    (2010)
  • K.J. Cheng et al.

    Lossless to lossy dual-tree BEZW compression for hyperspectral images

    IEEE Trans. Geosci. Remote Sens.

    (2014)
  • N. Amrani et al.

    Regression wavelet analysis for lossless coding of remote-sensing data

    IEEE Trans. Geosci. Remote Sens.

    (2016)
  • D. Valsesia et al.

    High-throughput onboard hyperspectral image compression with ground-based CNN reconstruction

    IEEE Trans. Geosci. Remote Sens.

    (2019)
  • Cited by (14)

    • Multi-scale spatial-spectral attention network for multispectral image compression based on variational autoencoder

      2022, Signal Processing
      Citation Excerpt :

      In other words, the spectra of multispectral image are non-stationary. Last but not least, current multispectral image compression frameworks [9–12] directly use a fully factorized entropy model [28] along with an existing adaptive entropy coder. However, this will lead to some statistical dependency remaining in the distribution of latent representation, resulting in suboptimal compression performance [29].

    • MultiTempGAN: Multitemporal multispectral image compression framework using generative adversarial networks

      2021, Journal of Visual Communication and Image Representation
      Citation Excerpt :

      Specifically, here, CNN represents a spectral transform similar to PCA in PCA + JP2K method. Moreover, Kong et al. [30] presented a spectral-spatial feature partitioned extraction-based CNN method for multispectral image compression. The method first applies the data into two independent CNN feature extraction modules to remove spectral and spatial redundancy and then compresses the data using quantization and entropy coding steps.

    • A combination method of stacked autoencoder and 3D deep residual network for hyperspectral image classification

      2021, International Journal of Applied Earth Observation and Geoinformation
    • Multi-spectral image compression by fusing multi-scale feature convolutional neural networks

      2024, Guangxue Jingmi Gongcheng/Optics and Precision Engineering
    View all citing articles on Scopus

    Fanqiang Kong received the Ph.D. degree in information and communication engineering from Xidian University, Xi'an, China, in 2008. He is currently an associate professor at the College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing, China. His research interests include spectral image coding and image analysis, artificial intelligence, and pattern recognition.

    Shunmin Zhao received the B.S. degree in information engineering from Qingdao University, Qingdao, China, in 2018. He is now working towards the M.S. degree at the College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing, China. His current research interests include multispectral image compression, deep learning.

    YunSong Li received the Ph.D. degree in signal processing from Xidian University, Xi'an, China, in 2002. He is a Professor of Communication with Xidian University. His research interests include spectral image coding and image analysis.

    Dan Li received the B.S., M.S., and Ph.D. degrees in control science and engineering from the Harbin Institute of Technology (HIT), Harbin, China, in 2012, 2014, and 2018, respectively. Since 2018, she has been a lecturer with college of Astronautics, Nanjing University of Aeronautics and Astronautics (NUAA). Her research interests include Hyperspectral image classification, signal processing, sparse sampling and reconstruction technology.

    Yongbo Zhou received the B.S. degree in information engineering from the College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing, China, in 2017 and received M.S. degree the M.S. degree in information engineering from the College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing, China. He is now engaged in related work in communications.

    View full text