M2H-Net: A Reconstruction Method For Hyperspectral Remotely Sensed Imagery
Introduction
The Hyperspectral image contains not only the spatial information of the observed objects but also the rich spectral information of dozens or even hundreds of narrow bands in each pixel, which greatly improves the ability of human beings to recognize the world. Since the 1970s, hyperspectral imaging technology has developed rapidly and has been widely used in remote sensing (Goetz et al., 1985, Goetz, 2009), military (Tiwari et al., 2011, Shimoni et al., 2019), agriculture (Ke et al., 2016, Zhu et al., 2020), biomedical (Lu and Fei, 2014, Offerhaus et al., 2019), food detection (Ravikanth et al., 2017, Xing et al., 2019) and other fields. It has become very popular because of its great potential and value in research and application.
However, the acquisition of hyperspectral data is currently the bottleneck of the research and application of hyperspectral technology. Despite great progress has been made in both hardware and software, there are still many problems that cannot be properly solved. Firstly, the cost of hyperspectral camera is very high because of the manufacturing technology of sensors and optical components (Gutiérrez et al., 2019); secondly, high spectral resolution is often achieved at the cost of losing spatial and temporal resolution (Brady, 2009, Jung et al., 2015, Behmann et al., 2018); finally, handling the huge amount of hyperspectral data is time, computing and storage resources consuming (Signoroni et al., 2019). These factors obviously make it limited in scientific research and extensive large-scale practical application.
Although many miniature hyperspectral imagers (Basedow et al., 1995, Gat, 2000, Gonzalez et al., 2016) have been developed in recent years, however, due to the physical limitations, difficult choices have to be made among spectral, spatial and temporal resolution, hardware performance and many other factors. Therefore, researchers tried to start with software/algorithm to find solutions with higher imaging quality, efficiency and lower cost, such as pan/multi-sharpening (Loncan et al., 2015, Vivone et al., 2019, Zhou et al., 2016) and principal component analysis (Maloney, 1986, Agahian et al., 2008), etc. Although the accuracy was not high, the extremely low cost attracted the interest of many researchers. Later, sparse coding technology was used to greatly improve the accuracy of reconstruction of hyperspectral image with RGB image (Arad and Ben-Shahar, 2016, Aeschbacher et al., 2017, Fu et al., 2018). In recent years, deep learning (DL) technology (Lin and Finlayson, 2020, Ma et al., 2019, Reichstein et al., 2019, Yuan et al., 2020) has made a breakthrough in hyperspectral image reconstruction. For example, Alvarez-Gila (Alvarez-Gila et al., 2017) achieved relatively high accurate reconstructed hyperspectral image by generating a generative adversarial network, Koundinya (Koundinya et al., 2018) used RGB images to reconstruct hyperspectral images based on 3D-CNN, and Shi (Shi et al., 2018) constructed a hyperspectral reconstruction network by using the idea of dense and residual connection. New trends in image restoration and enhancement (NTIRE) held two competitions in 2018 (Arad et al., 2018) and 2020 (Arad et al., 2020) to promote the research and application of hyperspectral reconstruction technology, and various networks (Fubara et al., 2020, Li et al., 2020, Zhao et al., 2020) have achieved promising results.
However, it should be pointed out that at present, most research on hyperspectral image reconstruction focuses only on the visible spectrum between 400 nm and 700 nm (Signoroni et al., 2019), which is too narrow to meet the needs of many fields. For example, it has been known that in vegetation remote sensing, the red edge (REG) band and/or near-infrared (NIR) band (from ~ 700 nm to ~ 1000 nm) can better show the vegetation status than visible bands, but the spectral curve will oscillate drastically from visible to NIR. It is not known whether the methods and conclusions of previous studies can be extended to a wider spectral range for remote sensing. In addition, most of the previous studies focused on the visible three-band RGB images, while the sensors used in remote sensing basically have more than three channels. Nevertheless, most of the existing frameworks are only applicable to three bands, which are unable to give full play to the advantages of multispectral images, thus, it will lead to a great waste of information. In terms of the reconstructed hyperspectral image, since many research results are trained and compared based on public datasets with only 31 bands (Yasuma et al., 2010, Arad and Ben-Shahar, 2016, Arad et al., 2018, Li et al., 2020), the number of reconstructed bands is too small to demonstrate the ability of reconstruction algorithm in spectral continuity and accuracy. Last but not least, to the best of our knowledge, few researches have explored such large-scale and highly complex scenes as satellite remote sensing. Instead, most of them made only some trials in the relatively small area with simple objects.
In view of these issues, the main objective of this paper is to develop a multispectral to hyperspectral network (M2H-Net), which can reconstruct hyperspectral image from multispectral image with high reconstruction accuracy and is more responsive to remote sensing, i.e., wide spectral range applicability, input/output band customization, complex scene and large-scale stability. In particular, the study addresses the following research questions: 1) How to build a DL network with good spectral and spatial resolution and high reconstruction accuracy in a wide spectral range (380–2500 nm)? 2) What combination of multispectral bands can be used as the input of the network to reconstruct hyperspectral image more efficiently and accurately? 3) How applicable is the network to different remote sensing sensors and complex scenes?
To this end, five sets of multispectral and hyperspectral combinatorial datasets are developed for different scenarios, which come from different platforms (UAVs and satellites), and different sensor types (frame and pushbroom), with different spectral and spatial resolutions. M2H-Net is applied on these datasets, and MRAE and RMSE are used to analyze and discuss the reconstructed hyperspectral image results.
Section snippets
Study area and data acquisition
Four test fields with different characteristics were selected as study areas, as shown in Fig. 1. Study area 1 is located in the suburb of Daxing District, Beijing, China. It is an agricultural experimental field with flat and open terrain and flourishing soybean, maize and other vegetation (trees, shrubs, weeds). Study area 2 is a stone quarry located on the outskirts of Fangshan District, Beijing, which has an obvious relief and complex environment. The surface mainly covers stones, trees,
Overall accuracy
Table 6 shows the accuracy of the reconstructed hyperspectral images using the testing-set in datasets 1–5. On the whole, the MRAE and RMSE values of the five datasets are low, less than 0.075 and 0.016 respectively, indicating that the overall accuracy of reconstruction is high; the running time per image for all datasets is less than 1 s, which indicates the high efficiency of the M2H-Net.
Specifically, we can see that the MRAE and RMSE values of dataset 1 and 5 are the lowest, indicating that
Conclusions
In this paper, a deep convolution network (M2H-Net) for reconstructing hyperspectral images from multispectral images is proposed. It has the following characteristics: 1) it can produce hyperspectral images with high accurate and continuous spectrum in the range of 380–2500 nm, which is commonly-used in earth observation remote sensing; 2) it can effectively use more bands as input, which can be used between hyperspectral sensors with similar spectral response function, as well as between
Declaration of Competing Interest
The authors declared that they have no conflicts of interest to this work.
Acknowledgment
We thank the anonymous reviewers and the editors, whose comments and advice improve the quality of the paper. This research was supported by Capacity Building for Sci-Tech Innovation - Fundamental Scientific Research Funds (NO.: 20530290059).
References (66)
- et al.
The effect of spatial resolution on radiometric and geometric performances of a UAV-mounted hyperspectral 2D imager
ISPRS J. Photogrammetry Remote Sensing
(2018) Three decades of hyperspectral remote sensing of the Earth: A personal view
Remote Sens. Environ.
(2009)- et al.
Deep learning in remote sensing applications: A meta-analysis and review
ISPRS J. Photogrammetry Remote Sensing
(2019) - et al.
An assessment of independent component analysis for detection of military targets from hyperspectral images
Int. J. Appl. Earth Obs. Geoinf.
(2011) - et al.
Deep learning in environmental remote sensing: Achievements and challenges
Remote Sens. Environ.
(2020) - et al.
In defense of shallow learned spectral reconstruction from rgb images
Proceedings of the IEEE International Conference on Computer Vision Workshops
(2017) - et al.
Reconstruction of reflectance spectra using weighted principal component analysis
Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur
(2008) - et al.
Adversarial networks for spatial context-aware spectral image reconstruction from rgb
Proceedings of the IEEE International Conference on Computer Vision Workshops
(2017) - et al.
Ntire 2018 challenge on spectral reconstruction from rgb images
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops
(2018) - et al.
Ntire 2020 challenge on spectral reconstruction from an rgb image
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops
(2020)
Sparse recovery of hyperspectral signal from natural rgb images
HYDICE system: Implementation and performance
Specim IQ: evaluation of a new, miniaturized handheld hyperspectral camera and its application for plant phenotyping and disease detection
Sensors
Optical imaging and spectroscopy
UAV-based structural and spectral data for the assessment and monitoring of oil palm biomass
Rgb to spectral reconstruction via learned basis functions and weights
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
Spectral reflectance recovery from a single rgb image
IEEE Transactions on Computational Imaging
A novel CMOS-compatible, monolithically integrated line-scan hyperspectral imager covering the VIS-NIR range. Next-Generation Spectroscopic Technologies IX
International Society for Optics and Photonics
Imaging spectroscopy using tunable filters: a review. In, Wavelet Applications VII
Semantic segmentation for urban planning maps based on U-Net. IGARSS 2018-2018
Imaging spectrometry for earth remote sensing
Science
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Proceedings of the IEEE international conference on computer vision
Deep residual learning for image recognition
Proceedings of the IEEE conference on computer vision and pattern recognition
Image quality metrics: PSNR vs. SSIM
2010 20th International Conference on Pattern Recognition, Istanbul
Densely connected convolutional networks
Proceedings of the IEEE conference on computer vision and pattern recognition
Multilevel SIFT matching for large-size VHR image registration
IEEE Geosci. Remote Sens. Lett.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey
Proceedings of Machine Learning Research
Assessing the accuracy of high resolution digital surface models computed by PhotoScan® and MicMac® in sub-optimal survey conditions
Remote Sensing
Cited by (24)
Hyper-ES<sup>2</sup>T: Efficient Spatial–Spectral Transformer for the classification of hyperspectral remote sensing images
2022, International Journal of Applied Earth Observation and GeoinformationCitation Excerpt :Hyperspectral imaging, a rapidly developing technology, enhances the capacity of human beings to recognize the world, providing a novel prospect for earth observation. It becomes very popular for its broad scope and significance in terms of research and application, which is widely employed in a variety of fields, including remote sensing, military, agriculture, biomedical (Deng et al., 2021). Since the 2000s, the researches on hyperspectral images (HSIs) processing have been further boosted by the development of machine learning (ML), and deep learning (DL) (Pande and Banerjee, 2022).
Analysing biological colour patterns from digital images: An introduction to the current toolbox
2024, Ecology and EvolutionMulti-sensor multispectral reconstruction framework based on projection and reconstruction
2024, Science China Information SciencesIMU-CNN: implementing remote sensing image restoration framework based on Mask-Upgraded Cascade R-CNN and deep autoencoder
2024, Multimedia Tools and ApplicationsS<sup>2</sup>DCN: Spectral-Spatial Difference Convolution Network for Hyperspectral Image Classification
2024, IEEE Journal of Selected Topics in Applied Earth Observations and Remote SensingUAV Remote-Sensing Image Semantic Segmentation Strategy Based on Thermal Infrared and Multispectral Image Features
2023, IEEE Journal on Miniaturization for Air and Space Systems
- 1
Joint first author.