Abstract
High-resolution is essential to achieve remarkable performance in many applications using hyperspectral images. However, the acquired hyperspectral images are often low-resolution due to the limitations of sensors. Recently, deep learning-based methods have been widely studied to address the super-resolution problem, and most super-resolution methods are learned with the simulated dataset. However, the pre-defined down-sampling function employed in the simulation is simple, thus the trained model often suffers from the poor generalization for real-world images. In this paper, we propose an unsupervised super-resolution approach that does not require the paired dataset. The method is conducted in two stages: unsupervised image generation and supervised image super-resolution. Specifically, the image generation model generates a low-resolution image through a generative adversarial network in an unsupervised manner. Then, the generated low-resolution images are used to train the super-resolution model with high-resolution images in a supervised manner. To explore the high correlation of hyperspectral image in spectral and spatial domain, group convolution, attention mechanism, and multi-level features are employed in the super-resolution model. Experiments on the public dataset CAVE and Harvard show that the proposed model provides outperforming super-resolution ability over the compared methods. Also, the results on images with unknown degradation show a promising generalization of the proposed model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Code availability
Custom code.
References
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)
Liebel, L., Körner, M.: Single-image super resolution for multispectral remote sensing data using convolutional neural networks. ISPRS-Int. Arch. Photogr., Remote Sens. Spatial Inf. Sci. 41, 883–890 (2016)
Li, Y., Hu, J., Zhao, X., Xie, W., Li, J.J.: Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing (2017). https://doi.org/10.1016/j.neucom.2017.05.024
Mei, S., Yuan, X., Ji, J., Zhang, Y., Wan, S., Du, Q.: Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sens. 9(11), 1139 (2017). https://doi.org/10.3390/rs9111139
Zheng, K., Gao, L., Ran, Q., Cui, X., Zhang, B., Liao, W., Jia, S.: Separable-spectral convolution and inception network for hyperspectral image super-resolution. Int. J. Mach. Learn. Cybern. (2019). https://doi.org/10.1007/s13042-018-00911-4
Lugmayr, A., Danelljan, M., Timofte, R.: Unsupervised learning for real-world super-resolution. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) 2019, pp. 3408–3416. IEEE
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z.: Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In: CVPR 2017, vol. 3, p. 4
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018
Dai, T., Cai, J., Zhang, Y., Xia, S.-T., Zhang, L.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition 2019, pp. 11065–11074
Shi, Z., Chen, C., Xiong, Z., Liu, D., Zha, Z.-J., Wu, F.: Deep residual attention network for spectral image super-resolution. In: European Conference on Computer Vision 2018, pp. 214–229. Springer
Shocher, A., Cohen, N., Irani, M.: “zero-shot” super-resolution using deep internal learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018, pp. 3118–3126
Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., Lin, L.: Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops 2018, pp. 701–710
Zhang, Y., Liu, S., Dong, C., Zhang, X., Yuan, Y.: Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution. IEEE Trans. Image Process. 29, 1101–1112 (2019)
Bulat, A., Yang, J., Tzimiropoulos, G.: To learn image super-resolution, use a gan to learn how to do image degradation first. In: Proceedings of the European conference on computer vision (ECCV) 2018, pp. 185–200
Chen, S., Han, Z., Dai, E., Jia, X., Liu, Z., Xing, L., Zou, X., Xu, C., Liu, J., Tian, Q.: Unsupervised image super-resolution with an indirect supervised path. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops 2020, pp. 468–469
Zhang, N., Wang, Y., Zhang, X., Xu, D., Wang, X.: An Unsupervised Remote Sensing Single-Image Super-Resolution Method Based on Generative Adversarial Network. IEEE Access 8, 29027–29039 (2020)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems 2014, pp. 2672–2680
Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint (2018). arXiv:1802.05957
Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. 19(9), 2241–2253 (2010). https://doi.org/10.1109/TIP.2010.2046811
Chakrabarti, A., Zickler, T.: Statistics of real-world hyperspectral images. In: CVPR 2011 2011, pp. 193–200. IEEE
Funding
This work is jointly supported by the National Natural Science Foundation of China (Grant o: 61403397) and the Natural Science Basic Research Plan in Shaanxi Province of China (Grant No: 2020JM-358).
Author information
Authors and Affiliations
Contributions
Zhang proposed method and conducted experiments, and Zhao is responsible for data preparation and pre-processing. Fu and Wang proposed improvements to experimental plan and amendments to the article.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no conflict of interests.
Availability of data and material
CAVE dataset is available from https://www.cs.columbia.edu/CAVE/databases /multispectral/ Harvard dataset is available from http://vision.seas.harvard.edu/ hyperspec/ index.html.
Rights and permissions
About this article
Cite this article
Zhang, S., Fu, G., Wang, H. et al. Degradation learning for unsupervised hyperspectral image super-resolution based on generative adversarial network. SIViP 15, 1695–1703 (2021). https://doi.org/10.1007/s11760-021-01902-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-021-01902-9