Skip to main content
SpringerLink
Log in

Learning hyperspectral images from RGB images via a coarse-to-fine CNN

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Hyperspectral remote sensing is well-known for its extraordinary spectral distinguishability to discriminate different materials. However, the cost of hyperspectral image (HSI) acquisition is much higher compared to traditional RGB imaging. In addition, spatial and temporal resolutions are sacrificed to obtain very high spectral resolution owing to the limitations of sensor technologies. Therefore, in this paper, HSIs are reconstructed using easily acquired RGB images and a convolutional neural network (CNN). As a result, high spatial and temporal resolution RGB images can be inherited to HSIs. Specifically, a two-stage CNN, referred to as the spectral super-resolution network (SSR-Net), is designed to learn the transformation model between RGB images and HSIs from training data, including a band prediction network (BP-Net) to estimate hyperspectral bands from RGB images and a refinement network (RF-Net) to further reduce spectral distortion in the band prediction step. As a result, the learned joint features in the proposed SSR-Net can directly predict HSIs from their corresponding scenes in RGB images without prior knowledge. Experimental results obtained on several benchmark datasets demonstrate that the proposed SSR-Net outperforms several state-of-the-art methods by ensuring higher quality in HSI reconstruction, and significantly improves the performance of traditional RGB images in classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Li J, Marpu P R, Plaza A, et al. Generalized composite kernel framework for hyperspectral image classification. IEEE Trans Geosci Remote Sens, 2013, 51: 4816–4829

    Article  Google Scholar 

  2. Li W, Du Q, Zhang B. Combined sparse and collaborative representation for hyperspectral target detection. Pattern Recogn, 2015, 48: 3904–3916

    Article  Google Scholar 

  3. Huang X, Zhang L. An SVM ensemble approach combining spectral, structural, and semantic features for the classification of high-resolution remotely sensed imagery. IEEE Trans Geosci Remote Sens, 2013, 51: 257–272

    Article  Google Scholar 

  4. Ma M, Mei S, Wan S, et al. Video summarization via block sparse dictionary selection. Neurocomputing, 2020, 378: 197–209

    Article  Google Scholar 

  5. Zhang Z J, Pang Y W. CGNet: cross-guidance network for semantic segmentation. Sci China Inf Sci, 2020, 63: 120104

    Article  Google Scholar 

  6. Ma S, Pang Y W, Pan J, et al. Preserving details in semantics-aware context for scene parsing. Sci China Inf Sci, 2020, 63: 120106

    Article  Google Scholar 

  7. Xie J, Pang Y W, Cholakkal H, et al. PSC-Net: learning part spatial co-occurrence for occluded pedestrian detection. Sci China Inf Sci, 2021, 64: 120103

    Article  MathSciNet  Google Scholar 

  8. Cao J, Pang Y, Li X. Learning multilayer channel features for pedestrian detection. IEEE Trans Image Process, 2017, 26: 3210–3220

    Article  MathSciNet  MATH  Google Scholar 

  9. Alparone L, Wald L, Chanussot J, et al. Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest. IEEE Trans Geosci Remote Sens, 2007, 45: 3012–3021

    Article  Google Scholar 

  10. Bendoumi M A, He M Y, Mei S H. Hyperspectral image resolution enhancement using high-resolution multispectral image based on spectral unmixing. IEEE Trans Geosci Remote Sens, 2014, 52: 6574–6583

    Article  Google Scholar 

  11. Zhang Y. Spatial resolution enhancement of hyperspectral image based on the combination of spectral mixing model and observation model. In: Proceedings of SPIE, 2014. 9244: 201–204

    Google Scholar 

  12. Li X, Ling F, Foody G M, et al. Generating a series of fine spatial and temporal resolution land cover maps by fusing coarse spatial resolution remotely sensed images and fine spatial resolution land cover maps. Remote Sens Environ, 2017, 196: 293–311

    Article  Google Scholar 

  13. Fu Y, Zheng Y, Huang H, et al. Hyperspectral image super-resolution with a Mosaic RGB image. IEEE Trans Image Process, 2018, 27: 5539–5552

    Article  MathSciNet  Google Scholar 

  14. Zhang L, Wei W, Zhang Y, et al. Cluster sparsity field: an internal hyperspectral imagery prior for reconstruction. Int J Comput Vis, 2018, 126: 797–821

    Article  MATH  Google Scholar 

  15. Liebel L, Körner M. Single-image super resolution for multispectral remote sensing data using convolutional neural networks. Int Arch Photogramm Remote Sens Spatial Inf Sci, 2016, XLI-B3: 883–890

    Article  Google Scholar 

  16. Li Y, Hu J, Zhao X, et al. Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing, 2017, 266: 29–41

    Article  Google Scholar 

  17. Hu J, Li Y, Xie W. Hyperspectral image super-resolution by spectral difference learning and spatial error correction. IEEE Geosci Remote Sens Lett, 2017, 14: 1825–1829

    Article  Google Scholar 

  18. Mei S, Yuan X, Ji J, et al. Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sens, 2017, 9: 1139

    Article  Google Scholar 

  19. Mei S, Jiang R, Li X, et al. Spatial and spectral joint super-resolution using convolutional neural network. IEEE Trans Geosci Remote Sens, 2020. doi: https://doi.org/10.1109/TGRS.2020.2964288

  20. Nguyen R M H, Prasad D K, Brown M S. Training-based spectral reconstruction from a single RGB image. In: Proceedings of European Conference on Computer Vision, 2014. 186–201

  21. Arad B, Ben-Shahar O. Sparse recovery of hyperspectral signal from natural RGB images. In: Proceedings of European Conference on Computer Vision. Berlin: Springer, 2016. 19–34

    Google Scholar 

  22. Yi C, Zhao Y Q, Chan J C W. Spectral super-resolution for multispectral image based on spectral improvement strategy and spatial preservation strategy. IEEE Trans Geosci Remote Sens, 2019, 57: 9010–9024

    Article  Google Scholar 

  23. Jia Y, Zheng Y, Gu L, et al. From RGB to spectrum for natural scenes via manifold-based mapping. In: Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV), 2017. 4715–4723

  24. Arad B, Ben-Shahar O, Timofte R, et al. NTIRE 2018 challenge on spectral reconstruction from RGB images. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Los Alamitos, 2018

  25. Can Y B, Timofte R. An efficient CNN for spectral reconstruction from RGB images. 2018. ArXiv: 1804.04647

  26. Han X, Yu J, Xue J, et al. Spectral super-resolution for RGB images using class-based BP neural networks. In: Proceedings of 2018 Digital Image Computing: Techniques and Applications (DICTA), 2018. 1–7

  27. Koundinya S, Sharma H, Sharma M, et al. 2D-3D CNN based architectures for spectral reconstruction from RGB images. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018

  28. Zhang L, Zhang L, Du B. Deep learning for remote sensing data: a technical tutorial on the state of the art. IEEE Geosci Remote Sens Mag, 2016, 4: 22–40

    Article  Google Scholar 

  29. Mei S, Ji J, Hou J, et al. Learning sensor-specific spatial-spectral features of hyperspectral images via convolutional neural networks. IEEE Trans Geosci Remote Sens, 2017, 55: 4520–4533

    Article  Google Scholar 

  30. Yuan Q, Zhang Q, Li J, et al. Hyperspectral image denoising employing a spatial-spectral deep residual convolutional neural network. IEEE Trans Geosci Remote Sens, 2019, 57: 1205–1218

    Article  Google Scholar 

  31. Zhang M, Li W, Du Q. Diverse region-based CNN for hyperspectral image classification. IEEE Trans Image Process, 2018, 27: 2623–2634

    Article  MathSciNet  MATH  Google Scholar 

  32. Yasuma F, Mitsunaga T, Iso D, et al. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans Image Process, 2010, 19: 2241–2253

    Article  MathSciNet  MATH  Google Scholar 

  33. Choi I, Jeon D S, Nam G, et al. High-quality hyperspectral reconstruction using a spectral prior. ACM Trans Graph, 2017, 36: 1–13

    Article  Google Scholar 

  34. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. ArXiv: 1409.1556

  35. Chen L, Papandreou G, Schroff F, et al. Rethinking atrous convolution for semantic image segmentation. 2017. ArXiv: 1706.05587

  36. Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. In: Proceedings of International Conference on Learning Representations, 2016

  37. Zhao H, Gallo O, Frosio I, et al. Loss functions for image restoration with neural networks. IEEE Trans Comput Imag, 2017, 3: 47–57

    Article  Google Scholar 

  38. Saxe A M, Mcclelland J L, Ganguli S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In: Proceedings of International Conference on Learning Representations, 2014. 1–22

  39. Kingma D P, Ba J. Adam: a method for stochastic optimization. 2014. ArXiv:1412.6980

  40. Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. 2015. ArXiv: 1502.03167

  41. Camps-Valls G, Gomez-Chova L, Munoz-Mari J, et al. Composite kernels for hyperspectral image classification. IEEE Geosci Remote Sens Lett, 2006, 3: 93–97

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by National Natural Science Foundation of China (Grant No. 61671383). Seed Foundation of Innovation and Creation for Graduate Students in Northwestern Polytechnical University (Grant No. CX2020020), and Hong Kong RGC (Grant No. 9042820 (CityU 11219019)).

Author information

Authors and Affiliations

  1. School of Electronics and Information, Northwestern Polytechnical University, Xi’an, 710072, China

    Shaohui Mei & Yunhao Geng

  2. Department of Computer Science, City University of Hong Kong, Hong Kong, 999077, China

    Junhui Hou

  3. Department of Electrical and Computer Engineering, Mississippi State University, Starkville, MS, 39762, USA

    Qian Du

Authors
  1. Shaohui Mei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunhao Geng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junhui Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qian Du
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shaohui Mei.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mei, S., Geng, Y., Hou, J. et al. Learning hyperspectral images from RGB images via a coarse-to-fine CNN. Sci. China Inf. Sci. 65, 152102 (2022). https://doi.org/10.1007/s11432-020-3102-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-020-3102-9

Keywords

Search

Navigation