Deep spectral convolution network for hyperspectral image unmixing with spectral library
Introduction
Hyperspectral remote sensing images have been proverbially utilized in geophysical applications, including urban observation, environmental monitoring, emergency response, and so on [1], [2], [3]. However, mixed pixels which commonly exist in hyperspectral image (HSI) because of the low spatial resolution of the imaging spectrometers, critically limit the analyzing and application of HSI [4], [5]. Therefore, hyperspectral unmixing (HU) has been proposed to handle the spectral mixing problem and effectively utilize hyperspectral remote sensing data, which intends to deduce the endmembers and their corresponding abundances in the mixed pixels of HSI [6].
Based on different spectral mixing models, linear mixing model (LMM) and nonlinear mixing model (NLMM) are respectively proposed to deal with the unmixing problem in different cases [7]. For the theory and suitable conditions of the two mixing models, [6] has studied and analyzed in detail. LMM has been extensively applied for unmixing as its simpleness and effectiveness. And it supposes that the mixed spectra in the scene are a linear superposition of land-covers weighted by their fractional abundances [7]. Depended on the model, the existing unmixing approaches are mainly unsupervised methods such as nonnegative matrix factorization (NMF) [8], [9], [10], some typical approaches founded on geometry [11], [12] and statistics [13], [14], and all the aforementioned approaches derive endmembers straightly from the hyperspectral data. However, some of them [8], [9], [10], [13], [14] obtain virtual endmember signatures, and others [11], [12] demand the existence of pure pixels in the scene which is difficult to ensure in real scenarios. Furthermore, assessing the number of endmembers in a given HSI that these approaches needed is also troublesome.
In recent years, sparse unmixing [7], [15] has been proposed to deal with the above problems. As a semisupervised method, it supposes that each mixed pixel in HSI is represented as the combination of a few endmembers known beforehand and accessible in a big spectral library. Spectral libraries consist of pure spectral signals (endmembers) which are valuated on the ground, such as employing an advanced field spectroradiometer [15]. Some publicly obtainable spectral libraries have been given by national government agencies and research centers, such as the U.S. Geological Survey (USGS) spectral library [16], involving over 1300 materials’ spectra. Using the spectral libraries as dictionaries, sparse unmixing aims to searching the optimal subcollection of endmembers in the library that can best reconstruct the mixed pixels. And in this way, it needs not to suppose the presence of pure pixels or estimate the number of endmember signatures in HSI.
A significant amount of sparse unmixing methods have been raised in [15], [17], [18], [19]. And a latest tendency to solve this problem is to combine spatial or spectral information under a weighted sparse regression framework to enhance the estimated abundance fractions [20], [21], [22]. The algorithms in [15], [17], [19] handle the sparse regression and collaborative sparse regression problems separately employing the classical l1 and l2,1 norm minimization based approaches [21], [22], [23], [24] apply weighted sparse regularizers incorporating both spatial and spectral information to further promote sparsity on the abundance maps. The algorithms in [15], [17], [19] handle the sparse regression and collaborative sparse regression problems separately employing the classical l1 and l2,1 norm minimization based approaches [23], [24]. The algorithms in [21], [22] apply weighted sparse regularizers incorporating both spatial and spectral information to further promote sparsity on the abundance maps. Utilizing the alternating direction method of multipliers [25], the relevant majorization problems are convex and therefore able to be effectively solved. Readers can also find the concurrent developments of sparse HU in the literature [26], [27], [28].
Normally, LMM has good effects in scenarios where they are viewed as linearly mixing, either intrinsicly or macroscopically. For real scenarios which are hard to exactly depict, the composite NLMM will be more applicable. A Bayesian method and the generalized bilinear model are studied in [29] to estimate the abundance fractions and the noise variance [30] presents the autoassociative neural network (AANN) for pixel unmixing in HSI, and it extends the potential of utilizing artificial neural networks (ANNs) for solving the nonlinear unmixing problem. A Bayesian method and the generalized bilinear model are studied in [29] to estimate the abundance fractions and the noise variance. In [30], the autoassociative neural network (AANN) is presented for pixel unmixing in HSI, and it extends the potential of utilizing artificial neural networks (ANNs) for solving the nonlinear unmixing problem. It includes reducing the dimensionality of input vector and executing the estimation from the reduced vector to acquire the abundance maps. However, the feature extraction step and abundance estimating process have to be trained independently.
More recently, deep learning has drawn widespread advertency and has been successfully implemented in the fields of image denoising and super-resolution [31], [32]. Nonnegative sparse and denoising autoencoders have superior noise reduction and inherent self-adaptation potentiality. In [33], [34], [35], they have been applied to acquire the endmembers and abundances simultaneously for HU. The further application of sparse autoencoder [36] and multiple hidden layer autoencoder [37] in HU indicates its great potential. Meanwhile, the convolutional neural network (CNN) has also been applied to HU benefitting from its automatically extracting related characteristics. A new HU method based on deep spectral convolution network is proposed in [38] and spectral convolutions are applied to extract local spectral features. They are all unsupervised unmixing networks, and exhibit higher accuracy than traditional unmixing approaches, such as NMF-based methods and statistical methods. To make full use of abundance supervision information, [39] proposes an end-to-end HU approach based on the CNN and takes the spatial and spectral characteristic of HSI into the abundance estimating process by the way of patches, which improves the unmixing capability. Under the situation that the supervision information can be provided, the addition of abundance information improves the accuracy of unmixing network and reduces the waste of hyperspectral resources.
Inspired by the successful implantation of spectral libraries in sparse HU, in this paper, a new deep spectral convolution network incorporating spectral library called SCNL is proposed for HU, which can be applied for a series of HSIs after training. The main contributions of this paper can be summarized as follows.
- •
We propose a deep spectral convolution network with spectral library. The introduction of spectral library allows the network to be used for a number of HSIs, rather than a single HSI, after training. This greatly reduces the waste of resources and improves the utilization of the network. Meanwhile, the endmember signatures in spectral library are also used as supervised information to be added to the network, which improves the representation capabilities of the network.
- •
We use spectral convolutions to extract local spectral features to take full advantage of the spectral information of HSIs. We adopt a deeper network to deal with the incorporation of spectral library because it needs to extract more features. A deeper network architecture leads to better feature representation and better estimation results.
- •
We construct a new loss function, which includes reconstruction error, abundance sparsity, and abundance cross-entropy to train the proposed network in an end-to-end manner. They ensure that the network has better image reconstruction effects, sparser abundance fractions, and closer abundance values to the groundtruth, which leads to better experimental results.
The rest of this paper is structured as follows. Section 2 describes the proposed deep spectral convolution network architecture with spectral library. Experimental results and a detailed comparison with other existing approaches are illustrated in Section 3. Our experimental results on both simulated and real HSIs demonstrate that the proposed SCNL shows good performance, particularly in noisy real scenes. Finally, we conclude with remarks in Section 4.
Section snippets
Proposed framework
In this section, we describe the proposed deep spectral convolution network with spectral library for HU, which can be used for a serious of HSIs. We first express the problem to clarify the comprehensibility of the proposed approach, and then supply associated details regarding the proposed architecture.
Experiments and analysis
In this section, we investigate the validity of the proposed SCNL for HU on both simulated and real HSIs. The results acquired by the linear spectral unmixing (LSU), AANN [30], extended support vector machine (eSVM) [49], and pixel-based CNN [39] algorithms are shown for comparative motivation. The number of units for the hidden layers of AANN is set to 40, 4, and 40. And the learning rate is set to . The parameter setting of pixel-based CNN is consistent with [39]. The fully constrained
Conclusion
In this paper, we have proposed a deep spectral convolution network with spectral library for HU. In order to integrate the prior knowledge of spectral library in the HU process, we have designed a deeper convolutional network based on the spectral library for unmixing. It utilizes the CNN to extract spectral characteristics, then acquires the abundance fractions of mixed pixels through the fully connected layers, and finally, reconstructs the input pixels by another fully connected layer.
CRediT authorship contribution statement
Lin Qi: Conceptualization, Writing - original draft, Writing - review & editing. Jie Li: Conceptualization, Writing - original draft, Writing - review & editing. Ying Wang: Conceptualization, Writing - original draft, Writing - review & editing. Mingyu Lei: Conceptualization, Writing - original draft, Writing - review & editing. Xinbo Gao: Conceptualization, Writing - original draft, Writing - review & editing.
Declaration of Competing Interest
All authors have participated in (a) conception and design, or analysis and interpretation of the data; (b) drafting the article or revising it critically for important intellectual content; and (c) approval of the final version. This manuscript has not been submitted to, nor is under review at, another journal or other publishing venue. The authors have no affiliation with any organization with a direct or indirect financial interest in the subject matter discussed in the manuscript.
Acknowledgment
This work was supported in part by the National Natural Science Foundation of China under Grant 61432014, 61772402, U1605252 and 61671339, in part by the National Key Research and Development Program of China under Grant 2016QY01W0200, and in part by National High-Level Talents Special Support Program of China under Grant CS31117200001.
References (50)
- et al.
Material based salient object detection from hyperspectral images
Pattern Recognit.
(2018) - et al.
Structured sparse method for hyperspectral unmixing
ISPRS J. Photogramm. Remote Sens.
(2014) Algorithms for simultaneous sparse approximation. Part ii: convex relaxation
Signal Process.
(2006)- et al.
A novel joint dictionary framework for sparse hyperspectral unmixing incorporating spectral library
Neurocomputing
(2019) - et al.
Edge-preserving image denoising using a deep convolutional neural network
Signal Process.
(2019) - et al.
CNN based spectral super-resolution of remote sensing images
Signal Process.
(2020) - et al.
Urban observation: integration of remote sensing and social media data
IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.
(2019) - et al.
Nonlocal similarity based nonnegative tucker decomposition for hyperspectral image denoising
IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.
(2018) Why use hyperspectral imagery?
Photogramm. Eng. Remote Sens.
(2004)- et al.
Foreword to the special issue on spectral unmixing of remotely sensed data
IEEE Trans. Geosci. Remote Sens.
(2011)
Spectral unmixing
IEEE Signal Process. Mag.
Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based approaches
IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.
Manifold regularized sparse nmf for hyperspectral unmixing
IEEE Trans. Geosci. Remote Sens.
Nonnegative-matrix-factorization-based hyperspectral unmixing with partially known endmembers
IEEE Trans. Geosci. Remote Sens.
N-findr: An algorithm for fast autonomous spectral end-member determination in hyperspectral data
Proc. SPIE Conf. Imaging Spectrometry V
Vertex component analysis: a fast algorithm to unmix hyperspectral data
IEEE Trans. Geosci. Remote Sens.
Analyzing hyperspectral data with independent component analysis
Proc. AIPR Workshop: Exploiting New Image Sources Sensors
Ice: a statistical approach to identifying endmembers in hyperspectral images
IEEE Trans. Geosci. Remote Sens.
Sparse unmixing of hyperspectral data
IEEE Trans. Geosci. Remote Sens.
Total variation spatial regularization for sparse hyperspectral unmixing
IEEE Trans. Geosci. Remote Sens.
Non-local sparse unmixing for hyperspectral remote sensing imagery
IEEE J. Sel. Topics Appl. Earth Observations Remote Sens.
Collaborative sparse regression for hyperspectral unmixing
IEEE Trans. Geosci. Remote Sens.
Hyperspectral unmixing using double reweighted sparse regression and total variation
IEEE Geosci. Remote Sens. Lett.
Spatial discontinuity-weighted sparse unmixing of hyperspectral images
IEEE Trans. Geosci. Remote Sens.
Cited by (30)
Hyperspectral image denoising via self-modulating convolutional neural networks
2024, Signal ProcessingQuantification of polycyclic aromatic hydrocarbons in roasted Tan lamb using fluorescence hyperspectral imaging technology
2023, Journal of Food Composition and AnalysisUnmixing and pigment identification using visible and short-wavelength infrared: Reflectance vs logarithm reflectance hyperspaces
2023, Journal of Cultural HeritageSpectral-spatial adversarial network for nonlinear hyperspectral unmixing of imbalanced datasets
2023, International Journal of Applied Earth Observation and GeoinformationInvestigation of the data fusion of spectral and textural data from hyperspectral imaging for the near geographical origin discrimination of wolfberries using 2D-CNN algorithms
2022, Infrared Physics and TechnologyCitation Excerpt :To our knowledge, HSI technology and spectral and textural fusion have not been used in published studies to identify the near geographical origins of wolfberries. Deep learning was a new area of machine learning research that possessed the deep structure of artificial neural networks and was capable of handling very large-scale data [25]. Deep convolutional neural networks have been applied to HSI recognition [26].
Deep-learning-based latent space encoding for spectral unmixing of geological materials
2022, ISPRS Journal of Photogrammetry and Remote SensingCitation Excerpt :The difficulty in obtaining the ground-truth abundances and the non-consideration of spectral noises affected the practical implementation of these approaches for real-world data (Karami et al., 2014; Zhao et al., 2014). A 3D-convolution-based feature transformation, followed by sparsity estimation, was adopted to achieve acceptable unmixing results (Qi et al., 2020). The model-inspired neural networks (Qian et al., 2020) integrated physics-based and data-based priors to improve interpretability and reduce the vast training sample requirement.