Elsevier

Signal Processing

Volume 176, November 2020, 107672
Signal Processing

Deep spectral convolution network for hyperspectral image unmixing with spectral library

https://doi.org/10.1016/j.sigpro.2020.107672Get rights and content

Highlights

  • We propose a deep spectral convolution network with spectral library that can be applied for a series of HSIs after training.

  • We design a deeper network architecture to efficiently extract local spectral features and achieve better estimation results.

  • We construct a new loss function, which includes reconstruction error, abundance sparsity, and abundance cross-entropy.

Abstract

Spectral unmixing is an important task for hyperspectral remote sensing image processing, which infers the pure spectral signatures (endmembers) in hyperspectral image (HSI) and their corresponding fractions (abundances). Recently, deep learning has become a powerful tool for HSI analysis, such as HSI classification and HSI super-resolution. In this paper, we propose a new unmixing algorithm that uses the convolutional neural network (CNN) for hyperspectral data incorporating spectral library, which can be applied for a series of HSIs after training. The proposed deep spectral convolution network extracts features and then executes the estimating process from these extracted spectral characteristics to acquire the fractional abundances on a fixed spectral library. Meanwhile, considering the incorporation of spectral library, a deeper convolutional network has been adopted to achieve better results. Moreover, we construct a new loss function, which includes pixel reconstruction error, abundance sparsity, and abundance cross-entropy to train the aforementioned network in an end-to-end manner. Experiments on both simulated and real HSIs indicate the advantage of the proposed method, which can obviously enhance the abundance estimation accuracy.

Introduction

Hyperspectral remote sensing images have been proverbially utilized in geophysical applications, including urban observation, environmental monitoring, emergency response, and so on [1], [2], [3]. However, mixed pixels which commonly exist in hyperspectral image (HSI) because of the low spatial resolution of the imaging spectrometers, critically limit the analyzing and application of HSI [4], [5]. Therefore, hyperspectral unmixing (HU) has been proposed to handle the spectral mixing problem and effectively utilize hyperspectral remote sensing data, which intends to deduce the endmembers and their corresponding abundances in the mixed pixels of HSI [6].

Based on different spectral mixing models, linear mixing model (LMM) and nonlinear mixing model (NLMM) are respectively proposed to deal with the unmixing problem in different cases [7]. For the theory and suitable conditions of the two mixing models, [6] has studied and analyzed in detail. LMM has been extensively applied for unmixing as its simpleness and effectiveness. And it supposes that the mixed spectra in the scene are a linear superposition of land-covers weighted by their fractional abundances [7]. Depended on the model, the existing unmixing approaches are mainly unsupervised methods such as nonnegative matrix factorization (NMF) [8], [9], [10], some typical approaches founded on geometry [11], [12] and statistics [13], [14], and all the aforementioned approaches derive endmembers straightly from the hyperspectral data. However, some of them [8], [9], [10], [13], [14] obtain virtual endmember signatures, and others [11], [12] demand the existence of pure pixels in the scene which is difficult to ensure in real scenarios. Furthermore, assessing the number of endmembers in a given HSI that these approaches needed is also troublesome.

In recent years, sparse unmixing [7], [15] has been proposed to deal with the above problems. As a semisupervised method, it supposes that each mixed pixel in HSI is represented as the combination of a few endmembers known beforehand and accessible in a big spectral library. Spectral libraries consist of pure spectral signals (endmembers) which are valuated on the ground, such as employing an advanced field spectroradiometer [15]. Some publicly obtainable spectral libraries have been given by national government agencies and research centers, such as the U.S. Geological Survey (USGS) spectral library [16], involving over 1300 materials’ spectra. Using the spectral libraries as dictionaries, sparse unmixing aims to searching the optimal subcollection of endmembers in the library that can best reconstruct the mixed pixels. And in this way, it needs not to suppose the presence of pure pixels or estimate the number of endmember signatures in HSI.

A significant amount of sparse unmixing methods have been raised in [15], [17], [18], [19]. And a latest tendency to solve this problem is to combine spatial or spectral information under a weighted sparse regression framework to enhance the estimated abundance fractions [20], [21], [22]. The algorithms in [15], [17], [19] handle the sparse regression and collaborative sparse regression problems separately employing the classical l1 and l2,1 norm minimization based approaches [21], [22], [23], [24] apply weighted sparse regularizers incorporating both spatial and spectral information to further promote sparsity on the abundance maps. The algorithms in [15], [17], [19] handle the sparse regression and collaborative sparse regression problems separately employing the classical l1 and l2,1 norm minimization based approaches [23], [24]. The algorithms in [21], [22] apply weighted sparse regularizers incorporating both spatial and spectral information to further promote sparsity on the abundance maps. Utilizing the alternating direction method of multipliers [25], the relevant majorization problems are convex and therefore able to be effectively solved. Readers can also find the concurrent developments of sparse HU in the literature [26], [27], [28].

Normally, LMM has good effects in scenarios where they are viewed as linearly mixing, either intrinsicly or macroscopically. For real scenarios which are hard to exactly depict, the composite NLMM will be more applicable. A Bayesian method and the generalized bilinear model are studied in [29] to estimate the abundance fractions and the noise variance [30] presents the autoassociative neural network (AANN) for pixel unmixing in HSI, and it extends the potential of utilizing artificial neural networks (ANNs) for solving the nonlinear unmixing problem. A Bayesian method and the generalized bilinear model are studied in [29] to estimate the abundance fractions and the noise variance. In [30], the autoassociative neural network (AANN) is presented for pixel unmixing in HSI, and it extends the potential of utilizing artificial neural networks (ANNs) for solving the nonlinear unmixing problem. It includes reducing the dimensionality of input vector and executing the estimation from the reduced vector to acquire the abundance maps. However, the feature extraction step and abundance estimating process have to be trained independently.

More recently, deep learning has drawn widespread advertency and has been successfully implemented in the fields of image denoising and super-resolution [31], [32]. Nonnegative sparse and denoising autoencoders have superior noise reduction and inherent self-adaptation potentiality. In [33], [34], [35], they have been applied to acquire the endmembers and abundances simultaneously for HU. The further application of sparse autoencoder [36] and multiple hidden layer autoencoder [37] in HU indicates its great potential. Meanwhile, the convolutional neural network (CNN) has also been applied to HU benefitting from its automatically extracting related characteristics. A new HU method based on deep spectral convolution network is proposed in [38] and spectral convolutions are applied to extract local spectral features. They are all unsupervised unmixing networks, and exhibit higher accuracy than traditional unmixing approaches, such as NMF-based methods and statistical methods. To make full use of abundance supervision information, [39] proposes an end-to-end HU approach based on the CNN and takes the spatial and spectral characteristic of HSI into the abundance estimating process by the way of patches, which improves the unmixing capability. Under the situation that the supervision information can be provided, the addition of abundance information improves the accuracy of unmixing network and reduces the waste of hyperspectral resources.

Inspired by the successful implantation of spectral libraries in sparse HU, in this paper, a new deep spectral convolution network incorporating spectral library called SCNL is proposed for HU, which can be applied for a series of HSIs after training. The main contributions of this paper can be summarized as follows.

  • We propose a deep spectral convolution network with spectral library. The introduction of spectral library allows the network to be used for a number of HSIs, rather than a single HSI, after training. This greatly reduces the waste of resources and improves the utilization of the network. Meanwhile, the endmember signatures in spectral library are also used as supervised information to be added to the network, which improves the representation capabilities of the network.

  • We use spectral convolutions to extract local spectral features to take full advantage of the spectral information of HSIs. We adopt a deeper network to deal with the incorporation of spectral library because it needs to extract more features. A deeper network architecture leads to better feature representation and better estimation results.

  • We construct a new loss function, which includes reconstruction error, abundance sparsity, and abundance cross-entropy to train the proposed network in an end-to-end manner. They ensure that the network has better image reconstruction effects, sparser abundance fractions, and closer abundance values to the groundtruth, which leads to better experimental results.

The rest of this paper is structured as follows. Section 2 describes the proposed deep spectral convolution network architecture with spectral library. Experimental results and a detailed comparison with other existing approaches are illustrated in Section 3. Our experimental results on both simulated and real HSIs demonstrate that the proposed SCNL shows good performance, particularly in noisy real scenes. Finally, we conclude with remarks in Section 4.

Section snippets

Proposed framework

In this section, we describe the proposed deep spectral convolution network with spectral library for HU, which can be used for a serious of HSIs. We first express the problem to clarify the comprehensibility of the proposed approach, and then supply associated details regarding the proposed architecture.

Experiments and analysis

In this section, we investigate the validity of the proposed SCNL for HU on both simulated and real HSIs. The results acquired by the linear spectral unmixing (LSU), AANN [30], extended support vector machine (eSVM) [49], and pixel-based CNN [39] algorithms are shown for comparative motivation. The number of units for the hidden layers of AANN is set to 40, 4, and 40. And the learning rate is set to 1e5. The parameter setting of pixel-based CNN is consistent with [39]. The fully constrained

Conclusion

In this paper, we have proposed a deep spectral convolution network with spectral library for HU. In order to integrate the prior knowledge of spectral library in the HU process, we have designed a deeper convolutional network based on the spectral library for unmixing. It utilizes the CNN to extract spectral characteristics, then acquires the abundance fractions of mixed pixels through the fully connected layers, and finally, reconstructs the input pixels by another fully connected layer.

CRediT authorship contribution statement

Lin Qi: Conceptualization, Writing - original draft, Writing - review & editing. Jie Li: Conceptualization, Writing - original draft, Writing - review & editing. Ying Wang: Conceptualization, Writing - original draft, Writing - review & editing. Mingyu Lei: Conceptualization, Writing - original draft, Writing - review & editing. Xinbo Gao: Conceptualization, Writing - original draft, Writing - review & editing.

Declaration of Competing Interest

All authors have participated in (a) conception and design, or analysis and interpretation of the data; (b) drafting the article or revising it critically for important intellectual content; and (c) approval of the final version. This manuscript has not been submitted to, nor is under review at, another journal or other publishing venue. The authors have no affiliation with any organization with a direct or indirect financial interest in the subject matter discussed in the manuscript.

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China under Grant 61432014, 61772402, U1605252 and 61671339, in part by the National Key Research and Development Program of China under Grant 2016QY01W0200, and in part by National High-Level Talents Special Support Program of China under Grant CS31117200001.

References (50)

  • N. Keshava et al.

    Spectral unmixing

    IEEE Signal Process. Mag.

    (2002)
  • J.M. Bioucas-Dias et al.

    Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based approaches

    IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.

    (2012)
  • X. Lu et al.

    Manifold regularized sparse nmf for hyperspectral unmixing

    IEEE Trans. Geosci. Remote Sens.

    (2013)
  • L. Tong et al.

    Nonnegative-matrix-factorization-based hyperspectral unmixing with partially known endmembers

    IEEE Trans. Geosci. Remote Sens.

    (2016)
  • M.E. Winter

    N-findr: An algorithm for fast autonomous spectral end-member determination in hyperspectral data

    Proc. SPIE Conf. Imaging Spectrometry V

    (1999)
  • J.M. Nascimento et al.

    Vertex component analysis: a fast algorithm to unmix hyperspectral data

    IEEE Trans. Geosci. Remote Sens.

    (2005)
  • J.D. Bayliss et al.

    Analyzing hyperspectral data with independent component analysis

    Proc. AIPR Workshop: Exploiting New Image Sources Sensors

    (1998)
  • M. Berman et al.

    Ice: a statistical approach to identifying endmembers in hyperspectral images

    IEEE Trans. Geosci. Remote Sens.

    (2004)
  • M.-D. Iordache et al.

    Sparse unmixing of hyperspectral data

    IEEE Trans. Geosci. Remote Sens.

    (2011)
  • R.N. Clark, G.A. Swayze, R. Wise, K.E. Livo, T. Hoefen, R.F. Kokaly, S.J. Sutley, Usgs digital spectral library...
  • M.-D. Iordache et al.

    Total variation spatial regularization for sparse hyperspectral unmixing

    IEEE Trans. Geosci. Remote Sens.

    (2012)
  • Y. Zhong et al.

    Non-local sparse unmixing for hyperspectral remote sensing imagery

    IEEE J. Sel. Topics Appl. Earth Observations Remote Sens.

    (2014)
  • M.-D. Iordache et al.

    Collaborative sparse regression for hyperspectral unmixing

    IEEE Trans. Geosci. Remote Sens.

    (2014)
  • R. Wang et al.

    Hyperspectral unmixing using double reweighted sparse regression and total variation

    IEEE Geosci. Remote Sens. Lett.

    (2017)
  • S. Zhang et al.

    Spatial discontinuity-weighted sparse unmixing of hyperspectral images

    IEEE Trans. Geosci. Remote Sens.

    (2018)
  • Cited by (30)

    • Spectral-spatial adversarial network for nonlinear hyperspectral unmixing of imbalanced datasets

      2023, International Journal of Applied Earth Observation and Geoinformation
    • Investigation of the data fusion of spectral and textural data from hyperspectral imaging for the near geographical origin discrimination of wolfberries using 2D-CNN algorithms

      2022, Infrared Physics and Technology
      Citation Excerpt :

      To our knowledge, HSI technology and spectral and textural fusion have not been used in published studies to identify the near geographical origins of wolfberries. Deep learning was a new area of machine learning research that possessed the deep structure of artificial neural networks and was capable of handling very large-scale data [25]. Deep convolutional neural networks have been applied to HSI recognition [26].

    • Deep-learning-based latent space encoding for spectral unmixing of geological materials

      2022, ISPRS Journal of Photogrammetry and Remote Sensing
      Citation Excerpt :

      The difficulty in obtaining the ground-truth abundances and the non-consideration of spectral noises affected the practical implementation of these approaches for real-world data (Karami et al., 2014; Zhao et al., 2014). A 3D-convolution-based feature transformation, followed by sparsity estimation, was adopted to achieve acceptable unmixing results (Qi et al., 2020). The model-inspired neural networks (Qian et al., 2020) integrated physics-based and data-based priors to improve interpretability and reduce the vast training sample requirement.

    View all citing articles on Scopus
    View full text