Abstract
Because of the ill-posedness of multi-frame super resolution (MSR), the regularization method plays an important role in the MSR field. Various regularization terms have been proposed to constrain the image to be estimated. However, artifacts also exist in the estimated image due to the artificial tendency in the manually designed prior model. To solve this problem, we propose a novel regularization-based MSR method with learned prior knowledge. By using the variable splitting technique, the fidelity term and regularization term are separated. The fidelity term is associated with an “
Funding source: National Natural Science Foundation of China
Award Identifier / Grant number: 61802213
Funding source: Natural Science Foundation of Shandong Province
Award Identifier / Grant number: ZR2017LF016
Award Identifier / Grant number: ZR2018LF004
Funding statement: This work was supported by the National Natural Science Foundation of China (No. 61802213) and Shandong Provincial Natural Science Found (No. ZR2017LF016, No. ZR2018LF004).
References
[1] S. D. Babacan, R. Molina and A. K. Katsaggelos, Parameter estimation in TV image restoration using variational distribution approximation, IEEE Trans. Image Process. 17 (2008), no. 3, 326–339. 10.1109/TIP.2007.916051Search in Google Scholar PubMed
[2] A. Brifman, Y. Romano and M. Elad, Unified single-image and video super-resolution via denoising algorithms, IEEE Trans. Image Process. 28 (2019), no. 12, 6063–6076. 10.1109/TIP.2019.2924173Search in Google Scholar PubMed
[3] S. H. Chan, Performance analysis of plug-and-play ADMM: A graph signal processing perspective, IEEE Trans. Comput. Imaging 5 (2019), no. 2, 274–286. 10.1109/TCI.2019.2892123Search in Google Scholar
[4] S. H. Chan, X. Wang and O. A. Elgendy, Plug-and-play ADMM for image restoration: Fixed-point convergence and applications, IEEE Trans. Comput. Imaging 3 (2017), no. 1, 84–98. 10.1109/TCI.2016.2629286Search in Google Scholar
[5] K. Chang, P. L. K. Ding and B. X. Li, Single image super resolution using joint regularization, IEEE Signal Proc. Letters 25 (2018), no. 4, 596–600. 10.1109/LSP.2018.2815003Search in Google Scholar
[6] C. B. Chen, H. Liang, S. R. Zhao, Z. H. Lyu and M. Sarem, A novel multi-image super-resolution reconstruction method using anisotropic fractional order adaptive norm, Vis. Comput. 31 (2015), no. 9, 1217–1231. 10.1007/s00371-014-1007-5Search in Google Scholar
[7] J. Chen, J. L. Nunez-Yanez and A. Achim, Bayesian video super-resolution with heavy-tailed prior models, IEEE Trans. Circuits Syst. Video Technol. 24 (2014), no. 6, 905–914. 10.1109/TCSVT.2014.2302549Search in Google Scholar
[8] C. Dong, C. C. Loy, K. He and X. Tang, Learning a deep convolutional network for image super-resolution, Computer Vision—ECCV 2014, Springer, Cham (2014), 184–199. 10.1007/978-3-319-10593-2_13Search in Google Scholar
[9] I. El Mourabit, M. El Rhabi, A. Hakim, A. Laghrib and E. Moreau, A new denoising model for multi-frame super-resolution image reconstruction, Signal Proc. 132 (2017), 51–65. 10.1016/j.sigpro.2016.09.014Search in Google Scholar
[10] S. Farsiu, M. D. Robinson, M. Elad and P. Milanfar, Fast and robust multiframe super resolution, IEEE Trans. Image Process. 13 (2004), no. 10, 1327–1344. 10.1109/TIP.2004.834669Search in Google Scholar PubMed
[11] D. Geman and C. Yang, Nonlinear image recovery with half-quadratic regularization, IEEE Trans. Image Process. 4 (1995), no. 7, 932–946. 10.1109/83.392335Search in Google Scholar PubMed
[12] D. Ghosh, N. Kaabouch and W. C. Hu, A robust iterative super-resolution mosaicking algorithm using an adaptive and directional Huber–Markov regularization, J. Visual Commun. Image Represent. 40 (2016), 98–110. 10.1016/j.jvcir.2016.06.008Search in Google Scholar
[13] T. Honda, D. Sugimura and T. Hamamoto, Multi-frame RGB/NIR imaging for low-light color image super-resolution, IEEE Trans. Comput. Imag. 6 (2020), 248–262. 10.1109/TCI.2019.2948779Search in Google Scholar
[14] S. Huang, J. Sun, Y. Yang, Y. Fang, P. Lin and Y. Que, Robust single-image super-resolution based on adaptive edge-preserving smoothing regularization, IEEE Trans. Image Process. 27 (2018), no. 6, 2650–2663. 10.1109/TIP.2018.2809472Search in Google Scholar PubMed
[15] S. Huang, J. Wu, Y. Yang and P. Lin, Multi-frame image super-resolution reconstruction based on spatial information weighted fields of experts, Multidimens. Syst. Signal Process. 31 (2020), no. 1, 1–20. 10.1007/s11045-019-00648-5Search in Google Scholar
[16] H. Irmak, G. B. Akar and S. E. Yuksel, A MAP-based approach for hyperspectral imagery super-resolution, IEEE Trans. Image Process. 27 (2018), no. 6, 2942–2951. 10.1109/TIP.2018.2814210Search in Google Scholar PubMed
[17] B. Jacob Maiseli, N. Ally and H. J. Gao, A noise-suppressing and edge-preserving multiframe super-resolution image reconstruction method, Signal Proc. Image Commun. 34 (2015), 1–13. 10.1016/j.image.2015.03.001Search in Google Scholar
[18] R. C. Jin, S. R. Zhao, X. Y. Xu and E. Song, Multiframe super-resolution based on half-quadratic prior with artifacts suppress, J. Visual Commun. Image Represent. 40 (2016), 656–670. 10.1016/j.jvcir.2016.08.006Search in Google Scholar
[19] T. Kato, H. Hino and N. Murata, Double sparsity for multi-frame super resolution, Neurocomput. 240 (2017), 115–126. 10.1016/j.neucom.2017.02.043Search in Google Scholar
[20] J. Kim, J. Kwon Lee and K. M. Lee, Accurate image super-resolution using very deep convolutional networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE Press, Piscataway (2016), 1646–1654. 10.1109/CVPR.2016.182Search in Google Scholar
[21] Y. Kim, B. Ham, M. N. Do and K. Sohn, Structure-texture image decomposition using deep variational priors, IEEE Trans. Image Process. 28 (2019), no. 6, 2692–2704. 10.1109/TIP.2018.2889531Search in Google Scholar PubMed
[22] D. P. Kingma and J. L. Ba, Adam: A method for stochastic optimization, in: International Conference on Learning Representations, 2015. Search in Google Scholar
[23] A. Laghrib, A. Ben-loghfyry, A. Hadri and A. Hakim, A nonconvex fractional order variational model for multi-frame image super-resolution, Signal Proc. Image Commun. 67 (2018), 1–11. 10.1016/j.image.2018.05.011Search in Google Scholar
[24] A. Laghrib, A. Hadri, A. Hakim and S. Raghay, A new multiframe super-resolution based on nonlinear registration and a spatially weighted regularization, Inform. Sci. 493 (2019), 34–56. 10.1016/j.ins.2019.04.029Search in Google Scholar
[25] V. Lempitsky, A. Vedaldi and D. Ulyanov, Deep image prior, Computer Vision and Pattern Recognition, IEEE Press, Piscataway (2018), 9446–9454. 10.1109/CVPR.2018.00984Search in Google Scholar
[26] D. Liu, Z. Wang, B. Wen, J. Yang, W. Han and T. S. Huang, Robust single image super-resolution via deep networks with sparse prior, IEEE Trans. Image Process. 25 (2016), no. 7, 3194–3207. 10.1109/TIP.2016.2564643Search in Google Scholar PubMed
[27] X. Liu, L. Chen, W. Wang and J. Zhao, Robust multi-frame super-resolution based on spatially weighted half-quadratic estimation and adaptive BTV regularization, IEEE Trans. Image Process. 27 (2018), no. 10, 4971–4986. 10.1109/TIP.2018.2848113Search in Google Scholar PubMed
[28] B. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, Proceedings of Imaging Understanding Workshop, Scientific Research, Wuhan (1981), 121–130. Search in Google Scholar
[29] S. Ono, Primal-dual plug-and-play image restoration, IEEE Signal Proc. Lett. 24 (2017), no. 8, 1108–1112. 10.1109/LSP.2017.2710233Search in Google Scholar
[30] C. Paris, J. Bioucas-Dias and L. Bruzzone, A novel sharpening approach for superresolving multiresolution optical images, IEEE Trans. Geosci. Remote Sens. 57 (2019), no. 3, 1545–1546. 10.1109/TGRS.2018.2867284Search in Google Scholar
[31] Y. Romano, M. Elad and P. Milanfar, The little engine that could: regularization by denoising (RED), SIAM J. Imaging Sci. 10 (2017), no. 4, 1804–1844. 10.1137/16M1102884Search in Google Scholar
[32] S. Roth and M. J. Black, Fields of experts: A framework for learning image priors, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Press, Piscataway (2005), 860–867. 10.1109/CVPR.2005.160Search in Google Scholar
[33] L. I. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Phys. D 60 (1992), no. 1–4, 259–268. 10.1016/0167-2789(92)90242-FSearch in Google Scholar
[34] P. Shamsolmoali, X. F. Li and R. L. Wang, Single image resolution enhancement by efficient dilated densely connected residual network, Signal Proc. Image Commun. 79 (2019), 13–23. 10.1016/j.image.2019.08.008Search in Google Scholar
[35] P. Shamsolmoali, M. Zareapoor, D. K. Jain, V. K. Jain and J. Yang, Deep convolution network for surveillance records super-resolution, Multimedia Tools Appl. 78 (2019), no. 17, 23815–23829. 10.1007/s11042-018-5915-7Search in Google Scholar
[36] P. Shamsolmoali, M. Zareapoor, J. H. Zhang and J. Yang, Image super resolution by dilated dense progressive network, Image Vis. Comput. 88 (2019), 9–18. 10.1016/j.imavis.2019.03.006Search in Google Scholar
[37] B. Shi, Q. Lian and X. Fan, Ppr: Plug-and-play regularization model for solving nonlinear imaging inverse problems, Signal Proc. 162 (2019), 83–96. 10.1016/j.sigpro.2019.04.013Search in Google Scholar
[38] K. C. Sun, T.-H. Tran, R. Krawtschenko and S. Simon, Multi-frame super-resolution reconstruction based on mixed Poisson–Gaussian noise, Signal Proc. Image Commun. 82 (2020), Article ID 115736. 10.1016/j.image.2019.115736Search in Google Scholar
[39] Y. Tai, J. Yang and X. Liu, Image super-resolution via deep recursive residual network, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press, Piscataway (2017), 2790–2798. 10.1109/CVPR.2017.298Search in Google Scholar
[40] A. M. Teodoro, J. M. Bioucas-Dias and M. A. T. Figueiredo, A convergent image fusion algorithm using scene-adapted Gaussian-mixture-based denoising, IEEE Trans. Image Process. 28 (2019), no. 1, 451–463. 10.1109/TIP.2018.2869727Search in Google Scholar PubMed
[41] T. Tirer and R. Giryes, Super-resolution via image-adapted denoising cnns: Incorporating external and internal learning, IEEE Signal Proc. Lett. 26 (2019), no. 7, 1080–1084. 10.1109/LSP.2019.2920250Search in Google Scholar
[42] L. P. Wang, S. B. Zhou and A. Karim, Super-resolution image reconstruction method using homotopy regularization, Multimedia Tools Appl. 75 (2016), no. 23, 15993–16016. 10.1007/s11042-015-2910-0Search in Google Scholar
[43] Q. Wang, S. Li, H. Qin and A. Hao, Super-resolution of multi-observed RGB-D images based on nonlocal regression and total variation, IEEE Trans. Image Process. 25 (2016), no. 3, 1425–1440. 10.1109/TIP.2016.2521180Search in Google Scholar PubMed
[44] J. W. Zhang, J. S. Pan, W. S. Lai, R. W. H. Lau and M. H. Yang, Learning fully convolutional networks for iterative non-blind deconvolution, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press, Piscataway (2017), 6969–6977. 10.1109/CVPR.2017.737Search in Google Scholar
[45] K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Trans. Image Process. 26 (2017), no. 7, 3142–3155. 10.1109/TIP.2017.2662206Search in Google Scholar PubMed
[46] K. Zhang, W. M. Zuo, S. H. Gu and L. Zhang, Learning deep CNN denoiser prior for image restoration, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press, Piscataway (2017), 2808–2817. 10.1109/CVPR.2017.300Search in Google Scholar
[47] G. Zhong, S. Xiang, P. Zhou and L. Yu, Spatially adaptive tensor total variation-Tikhonov model for depth image super resolution, IEEE Access, 5 (2017), 13857–13867. 10.1109/ACCESS.2017.2715981Search in Google Scholar
[48] X. Zhu, P. Jin, X. X. Wang and N. Ai, Multi-frame image super-resolution reconstruction via low-rank fusion combined with sparse coding, Multimedia Tools Appl. 78 (2019), no. 6, 7143–7154. 10.1007/s11042-018-6495-2Search in Google Scholar
© 2020 Walter de Gruyter GmbH, Berlin/Boston
