Skip to main content
SpringerLink
Log in

Multi-FAN: multi-spectral mosaic super-resolution via multi-scale feature aggregation network

  • Short Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

This paper introduces a novel method to super-resolve multi-spectral images captured by modern real-time single-shot mosaic image sensors, also known as multi-spectral cameras. Our contribution is twofold. Firstly, we super-resolve multi-spectral images from mosaic images rather than image cubes, which helps to take into account the spatial offset of each wavelength. Secondly, we introduce an external multi-scale feature aggregation network (Multi-FAN) which concatenates the feature maps with different levels of semantic information throughout a super-resolution (SR) network. A cascade of convolutional layers then implicitly selects the most valuable feature maps to generate a mosaic image. This mosaic image is then merged with the mosaic image generated by the SR network to produce a quantitatively superior image. We apply our Multi-FAN to RCAN (residual channel attention network), which is the state-of-the-art SR algorithm. We show that Multi-FAN improves both quantitative results (PSNR and SSIM) and inference time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Ahn, N., Kang, B., Sohn, K.A.: Fast, accurate, and, lightweight super-resolution with cascading residual network. arXiv preprint arXiv:1803.08664 (2018)

  2. Bell, J.F., Wellington, D., Hardgrove, C., Godber, A., Rice, M.S., Johnson, J.R., Fraeman, A.: Multispectral imaging of mars from the mars science laboratory mastcam instruments: Spectral properties and mineralogic implications along the gale crater traverse. In: AAS/Division for Planetary Sciences Meeting Abstracts, vol. 48 (2016)

  3. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)

  4. Chang, C.I.: An information-theoretic approach to spectral variability, similarity, and discrimination for hyperspectral image analysis. IEEE Trans. Inf. Theory 46(5), 1927–1932 (2000)

    Article  Google Scholar 

  5. Doering, D., Vizzotto, M., Bredemeier, C., da Costa, C., Henriques, R., Pignaton, E., Pereira, C.: Mde-based development of a multispectral camera for precision agriculture. IFAC-PapersOnLine 49(30), 24–29 (2016)

    Article  Google Scholar 

  6. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. TPAMI (2016)

  7. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: ECCV (2016)

  8. Fu, Y., Zheng, Y., Huang, H., Sato, I., Sato, Y.: Hyperspectral image super-resolution with a mosaic RGB image. IEEE Trans. Image Process. 27(11), 5539–5552 (2018)

    Article  MathSciNet  Google Scholar 

  9. Fujimoto, A., Ogawa, T., Yamamoto, K., Matsui, Y., Yamasaki, T., Aizawa, K.: Manga109 dataset and creation of metadata. In: Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding, p. 2. ACM (2016)

  10. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS (2014)

  11. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014)

  12. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017)

  13. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

  14. Jaiswal, S.P., Fang, L., Jakhetiya, V., Pang, J., Mueller, K., Au, O.C.: Adaptive multispectral demosaicking based on frequency-domain analysis of spectral correlation. IEEE Trans. Image Process. 26(2), 953–968 (2016)

    Article  MathSciNet  Google Scholar 

  15. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)

  16. Kim, J.H., Choi, J.H., Cheon, M., Lee, J.S.: Ram: Residual attention module for single image super-resolution. arXiv preprint arXiv:1811.12043 (2018)

  17. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  18. Lahoud, F., Zhou, R., Süsstrunk, S.: Multi-modal spectral image super-resolution. In: European Conference on Computer Vision, pp. 35–50. Springer (2018)

  19. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network

  20. Li, Y., Hu, J., Zhao, X., Xie, W., Li, J.: Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266, 29–41 (2017)

    Article  Google Scholar 

  21. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: CVPRW (2017)

  22. Martin, D., Fowlkes, C., Tal, D., Malik, J., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Iccv Vancouver: (2001)

  23. Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. In: Advances in Neural Information Processing Systems, pp. 2204–2212 (2014)

  24. Monno, Y., Kikuchi, S., Tanaka, M., Okutomi, M.: A practical one-shot multispectral imaging system using a single image sensor. IEEE Trans. Image Process. 24(10), 3048–3059 (2015)

    Article  MathSciNet  Google Scholar 

  25. Najafi, M., Namin, S.T., Petersson, L.: Classification of natural scene multi spectral images using a new enhanced crf. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3704–3711. IEEE (2013)

  26. Shi, Z., Chen, C., Xiong, Z., Liu, D., Zha, Z.J., Wu, F.: Deep residual attention network for spectral image super-resolution. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 0 (2018)

  27. Shoeiby, M., Lars, P., Aliakbarian, S., Armin, A., Robles-kelly, A.: Super-resolved chromatic mapping of snapshot mosaic image sensors via a texture sensitive residual network

  28. Shoeiby, M., Robles-Kelly, A., Timofte, R., Zhou, R., Lahoud, F., Susstrunk, S., Xiong, Z., Shi, Z., Chen, C., Liu, D., Zha, Z.J., Wu, F., Wei, K., Zhang, T., Wang, L., Fu, Y., Zhong, Z., Nagasubramanian, K., Singh, A.K., Singh, A., Sarkar, S., Baskar, G.: PIRM2018 challenge on spectral image super-resolution: Methods and results. In: European Conference on Computer Vision Workshops (ECCVW) (2018)

  29. Shoeiby, M., Robles-Kelly, A., Wei, R., Timofte, R.: Pirm2018 challenge on spectral image super-resolution: Dataset and study (2019)

  30. Takumi, K., Watanabe, K., Ha, Q., Tejero-De-Pablos, A., Ushiku, Y., Harada, T.: Multispectral object detection for autonomous vehicles. In: Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp. 35–43. ACM (2017)

  31. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: ICCV (2017)

  32. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  33. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: CVPR (2018)

  34. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. arXiv preprint arXiv:1807.02758 (2018)

  35. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: European Conference on Computer Vision, pp. 294–310. Springer (2018)

  36. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: CVPR (2018)

  37. Zhou, R., Achanta, R., Süsstrunk, S.: Deep residual network for joint demosaicing and super-resolution. In: Color and Imaging Conference, vol. 2018, pp. 75–80. Society for Imaging Science and Technology (2018)

Download references

Author information

Authors and Affiliations

  1. CSIRO-DATA61, Canberra, Australia

    Mehrdad Sheoiby, Saeed Anwar & Lars Petersson

  2. CSIRO-DATA61/ANU, Canberra, Australia

    Sadegh Aliakbarian

Authors
  1. Mehrdad Sheoiby
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sadegh Aliakbarian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saeed Anwar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lars Petersson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehrdad Sheoiby.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sheoiby, M., Aliakbarian, S., Anwar, S. et al. Multi-FAN: multi-spectral mosaic super-resolution via multi-scale feature aggregation network. Machine Vision and Applications 32, 47 (2021). https://doi.org/10.1007/s00138-021-01174-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-021-01174-w

Keywords

Search

Navigation