M2H-Net: A Reconstruction Method For Hyperspectral Remotely Sensed Imagery

Abstract

Hyperspectral remote sensing can get spatially and spectrally continuous data simultaneously. However, the imaging equipment is usually expensive and complex, along with the low spatial resolution. In recent years, reconstruction of hyperspectral image by deep learning from the widely used low-cost, high spatial resolution RGB camera, has attracted extensive attention in many fields. However, most research is limited to three bands in the range of 400–700 nm, which greatly restrains its application in remote sensing. In this study, a more suitable for remote sensing multispectral to hyperspectral network (M2H-Net) is proposed, which can take many bands as input and output hyperspectral images with any number of bands within a wider spectral range (380–2500 nm). Its characteristics include adding residual connection on U-Net to reduce vanishing gradients; adding convolution combinations with different kernel sizes (1 × 1 and 3 × 3) to balance the spectral and spatial relationships. It is applied on images from different platforms (UAVs and Satellites), different imaging modes (frame and pushbroom) and different spectral response functions (narrow and wide bandwidth), and the results show that: 1) it has a very high accuracy of hyperspectral image reconstruction. The mean relative absolute error (MRAE) and root mean squared error (RMSE) are between 0.039 and 0.074 and 0.010–0.016, respectively, which are 69.2% and 41.2% lower than those of U-Net; 2) it has high efficiency with fast convergence (about 40 epochs) and stable performance. Compared with many algorithms won in the new trends in image restoration and enhancement (NTIRE) competition, M2H-Net ranked 7th in accuracy, but took less time (0.44 s); 3) it has strong generalization ability. Using the pre-trained M2H-Nets to reconstruct Cubert S185 and GF-5 hyperspectral images in different locations, different times and complex scenes, high accuracy (MRAE = 0.072, RMSE = 0.011) can still be obtained. This method is more suitable for remote sensing to meet the needs of multiple bands, spectrum width and complex scenes, thus provides the possibility to generate the global coverage hyperspectral imagery by using the massive in-orbit or historical archived multispectral images, which will not only greatly save the R&D and investment on hyperspectral imaging equipment, but also conduct data collection with higher efficiency and lower complexity. Due to the ability to reconstruct hyperspectral images in specified bands on demand, M2H-Net is also of great value in hyperspectral image processing, such as data compression, storage and transmission, etc.

Introduction

The Hyperspectral image contains not only the spatial information of the observed objects but also the rich spectral information of dozens or even hundreds of narrow bands in each pixel, which greatly improves the ability of human beings to recognize the world. Since the 1970s, hyperspectral imaging technology has developed rapidly and has been widely used in remote sensing (Goetz et al., 1985, Goetz, 2009), military (Tiwari et al., 2011, Shimoni et al., 2019), agriculture (Ke et al., 2016, Zhu et al., 2020), biomedical (Lu and Fei, 2014, Offerhaus et al., 2019), food detection (Ravikanth et al., 2017, Xing et al., 2019) and other fields. It has become very popular because of its great potential and value in research and application.

However, the acquisition of hyperspectral data is currently the bottleneck of the research and application of hyperspectral technology. Despite great progress has been made in both hardware and software, there are still many problems that cannot be properly solved. Firstly, the cost of hyperspectral camera is very high because of the manufacturing technology of sensors and optical components (Gutiérrez et al., 2019); secondly, high spectral resolution is often achieved at the cost of losing spatial and temporal resolution (Brady, 2009, Jung et al., 2015, Behmann et al., 2018); finally, handling the huge amount of hyperspectral data is time, computing and storage resources consuming (Signoroni et al., 2019). These factors obviously make it limited in scientific research and extensive large-scale practical application.

Although many miniature hyperspectral imagers (Basedow et al., 1995, Gat, 2000, Gonzalez et al., 2016) have been developed in recent years, however, due to the physical limitations, difficult choices have to be made among spectral, spatial and temporal resolution, hardware performance and many other factors. Therefore, researchers tried to start with software/algorithm to find solutions with higher imaging quality, efficiency and lower cost, such as pan/multi-sharpening (Loncan et al., 2015, Vivone et al., 2019, Zhou et al., 2016) and principal component analysis (Maloney, 1986, Agahian et al., 2008), etc. Although the accuracy was not high, the extremely low cost attracted the interest of many researchers. Later, sparse coding technology was used to greatly improve the accuracy of reconstruction of hyperspectral image with RGB image (Arad and Ben-Shahar, 2016, Aeschbacher et al., 2017, Fu et al., 2018). In recent years, deep learning (DL) technology (Lin and Finlayson, 2020, Ma et al., 2019, Reichstein et al., 2019, Yuan et al., 2020) has made a breakthrough in hyperspectral image reconstruction. For example, Alvarez-Gila (Alvarez-Gila et al., 2017) achieved relatively high accurate reconstructed hyperspectral image by generating a generative adversarial network, Koundinya (Koundinya et al., 2018) used RGB images to reconstruct hyperspectral images based on 3D-CNN, and Shi (Shi et al., 2018) constructed a hyperspectral reconstruction network by using the idea of dense and residual connection. New trends in image restoration and enhancement (NTIRE) held two competitions in 2018 (Arad et al., 2018) and 2020 (Arad et al., 2020) to promote the research and application of hyperspectral reconstruction technology, and various networks (Fubara et al., 2020, Li et al., 2020, Zhao et al., 2020) have achieved promising results.

However, it should be pointed out that at present, most research on hyperspectral image reconstruction focuses only on the visible spectrum between 400 nm and 700 nm (Signoroni et al., 2019), which is too narrow to meet the needs of many fields. For example, it has been known that in vegetation remote sensing, the red edge (REG) band and/or near-infrared (NIR) band (from ~ 700 nm to ~ 1000 nm) can better show the vegetation status than visible bands, but the spectral curve will oscillate drastically from visible to NIR. It is not known whether the methods and conclusions of previous studies can be extended to a wider spectral range for remote sensing. In addition, most of the previous studies focused on the visible three-band RGB images, while the sensors used in remote sensing basically have more than three channels. Nevertheless, most of the existing frameworks are only applicable to three bands, which are unable to give full play to the advantages of multispectral images, thus, it will lead to a great waste of information. In terms of the reconstructed hyperspectral image, since many research results are trained and compared based on public datasets with only 31 bands (Yasuma et al., 2010, Arad and Ben-Shahar, 2016, Arad et al., 2018, Li et al., 2020), the number of reconstructed bands is too small to demonstrate the ability of reconstruction algorithm in spectral continuity and accuracy. Last but not least, to the best of our knowledge, few researches have explored such large-scale and highly complex scenes as satellite remote sensing. Instead, most of them made only some trials in the relatively small area with simple objects.

In view of these issues, the main objective of this paper is to develop a multispectral to hyperspectral network (M2H-Net), which can reconstruct hyperspectral image from multispectral image with high reconstruction accuracy and is more responsive to remote sensing, i.e., wide spectral range applicability, input/output band customization, complex scene and large-scale stability. In particular, the study addresses the following research questions: 1) How to build a DL network with good spectral and spatial resolution and high reconstruction accuracy in a wide spectral range (380–2500 nm)? 2) What combination of multispectral bands can be used as the input of the network to reconstruct hyperspectral image more efficiently and accurately? 3) How applicable is the network to different remote sensing sensors and complex scenes?

To this end, five sets of multispectral and hyperspectral combinatorial datasets are developed for different scenarios, which come from different platforms (UAVs and satellites), and different sensor types (frame and pushbroom), with different spectral and spatial resolutions. M2H-Net is applied on these datasets, and MRAE and RMSE are used to analyze and discuss the reconstructed hyperspectral image results.