Skip to main content
Log in

Denoising Color Images Based on Local Orientation Estimation and CNN Classifier

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

A structure-adaptive vector filter for removal of impulse noise from color images is presented. The proposed method is based on local orientation estimation. A color image is represented in quaternion form, and then, quaternion Fourier transform is used to compute the orientation of the pattern in a local neighborhood. Since the computation in quaternion frequency domain is extremely time-consuming, we prove a theorem that the integral of the product of frequency variables and the magnitude of quaternion frequency signals can be computed directly in spatial domain, which results that the color orientation detection problem can be solved in spatial domain. Based on the local orientation and orientation strength, the size, shape, and orientation of the support window of vector median filter (VMF) are adaptively determined, leading to an effective structure-adaptive VMF. Unlike the classical VMF restricting the output to the existing color samples, this paper computes the output of VMF over the entire 3D data space, which boosts the filtering performance effectively. To further improve denoising effect, a deep convolutional neural network is employed to detect impulse noise in color images and integrated into the proposed denoising framework. The experimental results exhibit the effectiveness of the proposed denoiser by showing significant performance improvements both in noise suppression and in detail preservation, compared to other color image denoising methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Shen, Y., Han, B., Braverman, E.: Removal of mixed gaussian and impulse noise using directional tensor product complex tight framelets. J. Math. Imaging Vis. 54(1), 64–77 (2016)

    MathSciNet  MATH  Google Scholar 

  2. Plataniotis, K.N., Venetsanopoulos, A.N.: Color Image Processing and Applications. Springer, Berlin (2000)

    Google Scholar 

  3. Villar, S.A., Torcida, S., Acosta, G.G.: Median filtering: a new insight. J. Math. Imaging Vis. 58(1), 130–146 (2017)

    MATH  Google Scholar 

  4. Astola, J., Haavisto, P., Neuvo, Y.: Vector median filters. Proc. IEEE 78(4), 678–689 (1990)

    Google Scholar 

  5. Trahanias, P.E., Venetsanopoulos, A.N.: Vector directional filters: a new class of multichannel image processing filters. IEEE Trans. Image Process. 2(4), 528–534 (1993)

    Google Scholar 

  6. Karakos, D.G., Trahanias, P.E.: Generalized multichannel image-filtering structures. IEEE Trans. Image Process. 6(7), 1038–1045 (1997)

    Google Scholar 

  7. Lukac, R.: Adaptive vector median filtering. Pattern Recogn. Lett. 24(12), 1889–1899 (2003)

    Google Scholar 

  8. Smolka, B., Chydzinski, A.: Fast detection and impulsive noise removal in color images. Real Time Images 11(5/6), 389–402 (2005)

    Google Scholar 

  9. Malinski, L., Smolka, B.: Fast averaging peer group filter for the impulsive noise removal in color images. J. Real Time Image Process. 11(3), 427–444 (2016)

    Google Scholar 

  10. Celebi, M.E., Aslandogan, Y.A.: Robust switching vector median filter for impulsive noise removal. J. Electron. Imaging 17(4), 043006 (2008)

    Google Scholar 

  11. Shen, Y., Barner, K.E.: Fast adaptive optimization of weighted vector median filters. IEEE Trans. Signal Process. 54(7), 2497–2510 (2006)

    MATH  Google Scholar 

  12. Lukac, R., Smolka, B., Plataniotis, K.N., Venetsanopoulos, A.N.: Selection weighted vector directional filters. Comput. Vis. Image Underst. 94(1/3), 140–167 (2004)

    MATH  Google Scholar 

  13. Plataniotis, K.N., Androutsos, D., Venetsanopoulos, A.N.: Color image processing using adaptive vector directional filters. IEEE Trans. Circuits Syst. II Analog Digit. Signal Process 45(10), 1414–1419 (1998)

    Google Scholar 

  14. Camarena, J.-G., Gregori, V., Morillas, S., Sapena, A.: A simple fuzzy method to remove mixed gaussian-impulsive noise from color images. IEEE Trans. Fuzzy Syst. 21(5), 971–978 (2013)

    Google Scholar 

  15. Ananthi, V.P., Balasubramaniam, P.: A new image denoising method using interval-valued intuitionistic fuzzy sets for the removal of impulse noise. Signal Process. 121, 81–93 (2016)

    Google Scholar 

  16. Portilla, J., Strela, V., Wainwright, M.J., Simoncelli, E.P.: Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans. Image Process. 12(11), 1338–1351 (2003)

    MathSciNet  MATH  Google Scholar 

  17. Fathi, A., Naghsh, A.R.: Efficient image denoising method based on a new adaptive wavelet packet thresholding function. IEEE Trans. Image Process. 21(9), 3981–3990 (2012)

    MathSciNet  MATH  Google Scholar 

  18. Kim, S.: PDE-based image restoration: a hybrid model and color image denoising. IEEE Trans. Image Process. 15(5), 1163–1170 (2006)

    Google Scholar 

  19. Tiirola, J.: Image denoising using directional adaptive variable exponents model. J. Math. Imaging Vis. 57(1), 56–74 (2017)

    MathSciNet  MATH  Google Scholar 

  20. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 629–639 (1990)

    Google Scholar 

  21. Hajiaboli, M.R.: An anisotropic fourth-order diffusion filter for image noise removal. Int. J. Comput. Vis. 92(2), 177–191 (2011)

    MathSciNet  MATH  Google Scholar 

  22. Weickert, J.: Coherence-enhancing diffusion of colour images. Image Vis. Comput. 17(3–4), 201–212 (1999)

    Google Scholar 

  23. Estellers, V., Soatto, S., Bresson, X.: Adaptive regularization with the structure tensor. IEEE Trans. Image Process. 24(6), 1777–1790 (2015)

    MathSciNet  MATH  Google Scholar 

  24. Felsberg, M.: Autocorrelation-driven diffusion filtering. IEEE Trans. Image Process. 20(7), 1797–1806 (2011)

    MathSciNet  MATH  Google Scholar 

  25. Liu, F., Liu, J.: Anisotropic diffusion for image denoising based on diffusion tensors. J. Vis. Commun. Image Represent. 23(3), 516–521 (2012)

    Google Scholar 

  26. Jin, L., Zhu, Z., Xu, X., Li, X.: Two-stage quaternion switching vector filter for color impulse noise removal. Signal Process. 128, 171–185 (2016)

    Google Scholar 

  27. Gai, S., Yang, G., Wan, M., Wang, L.: Denoising color images by reduced quaternion matrix singular value decomposition. Multidimens. Syst. Signal Process. 26(1), 307–320 (2015)

    MathSciNet  Google Scholar 

  28. Jin, L., Liu, H., Xu, X., Song, E.: Quaternion-based impulse noise removal from color video sequences. IEEE Trans. Circuits Syst. Video Technol. 23(5), 741–755 (2013)

    Google Scholar 

  29. Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)

    MathSciNet  Google Scholar 

  30. Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2008)

    MathSciNet  MATH  Google Scholar 

  31. Chen, C.L.P., Liu, L., Chen, L., Tang, Y.Y., Zhou, Y.: Weighted couple sparse representation with classified regularization for impulse noise removal. IEEE Trans. Image Process. 24(11), 4014–4026 (2015)

    MathSciNet  MATH  Google Scholar 

  32. Jiang, J., Zhang, L., Yang, J.: Mixed noise removal by weighted encoding with sparse nonlocal regularization. IEEE Trans. Image Process. 23(6), 2651–2662 (2014)

    MathSciNet  MATH  Google Scholar 

  33. Ji, H., Huang, S., Shen, Z., Xu, Y.: Robust video restoration by joint sparse and low rank matrix approximation. SIAM J. Imaging Sci. 4(4), 1122–1142 (2011)

    MathSciNet  MATH  Google Scholar 

  34. Wang, R., Pakleppa, M., Trucco, E.: Low-rank prior in single patches for nonpointwise impulse noise removal. IEEE Trans. Image Process. 24(5), 1485–1496 (2015)

    MathSciNet  MATH  Google Scholar 

  35. Jin, K.H., Ye, J.C.: Sparse and low-rank decomposition of a Hankel structured matrix for impulse noise removal. IEEE Trans. Image Process. 27(3), 1448–1461 (2018)

    MathSciNet  MATH  Google Scholar 

  36. Roy, A., Laskar, R.H.: Multiclass SVM based adaptive filter for removal of high density impulse noise from color images. Appl. Soft Comput. 46, 816–826 (2016)

    Google Scholar 

  37. Lin, T.-C.: Decision-based filter based on SVM and evidence theory for image noise removal. Neural Comput. Appl. 21(4), 695–703 (2012)

    Google Scholar 

  38. Turkmen, I.: The ANN based detector to remove random-valued impulse noise in images. J. Vis. Commun. Image Represent. 34, 28–36 (2016)

    Google Scholar 

  39. Nair, M.S., Shankar, V.: Predictive-based adaptive switching median filter for impulse noise removal using neural network-based noise detector. Signal Image Video Process. 7(6), 1041–1070 (2013)

    Google Scholar 

  40. Kaliraj, G., Baskar, S.: An efficient approach for the removal of impulse noise from the corrupted image using neural network based impulse detector. Image Vis. Comput. 28, 458–466 (2010)

    Google Scholar 

  41. Liang, S.F., Lu, S.M., Chang, J.Y., Lin, C.T.: A novel two-stage impulse noise removal technique based on neural networks and fuzzy decision. IEEE Trans. Fuzzy Syst. 16(4), 863–873 (2008)

    Google Scholar 

  42. Forstner, M.A., Gulch, E.: A fast operator for detection and precise location of distinct points, corners and centers of circular features. In: Proceedings of the ISPRS Intercommission Conference on Fast Processing of Phonogrammic Data, pp. 281–305 (1987)

  43. Hamilton, W.R.: Elements of Quaternions. Longmans Green, London (1866)

    Google Scholar 

  44. Xu, D., Mandic, D.P.: The theory of quaternion matrix derivatives. IEEE Trans. Signal Process. 63(6), 1543–1556 (2015)

    MathSciNet  MATH  Google Scholar 

  45. Subakan, O.N., Vemuri, B.C.: A quaternion framework for color image smoothing and segmentation. Int. J. Comput. Vis. 91(3), 233–250 (2011)

    MathSciNet  MATH  Google Scholar 

  46. Chen, B., Shu, H., Coatrieux, G., Chen, G., Sun, X., Coatrieux, J.L.: Color image analysis by quaternion-type moments. J. Math. Imaging. Vis. 51(1), 124–144 (2014)

    MathSciNet  MATH  Google Scholar 

  47. Yang, G.Z., Burger, P., Firmin, D.N., Underwood, S.R.: Structure adaptive anisotropic image filtering. Image Vis. Comput. 14(2), 135–145 (1996)

    Google Scholar 

  48. Greenberg, S., Kogan, D.: Improved structure-adaptive anisotropic filter. Pattern Recognit. Lett. 27(1), 59–65 (2006)

    Google Scholar 

  49. Jin, L., Jin, M., Xu, X., Song, E.: Structure-adaptive vector median filter for impulse noise removal in color images. In: IEEE International Conference on Image Processing (ICIP), pp. 690–694 (2017)

  50. Ell, T.A., Sangwine, S.J.: Hypercomplex Fourier transforms of color images. IEEE Trans. Image Process. 16(1), 22–35 (2007)

    MathSciNet  MATH  Google Scholar 

  51. Said, S., Le Bihan, N., Sangwine, S.J.: Fast complexified quaternion Fourier transform. IEEE Trans. Signal Process. 56(4), 1522–1531 (2008)

    MathSciNet  MATH  Google Scholar 

  52. Pei, S.-C., Ding, J.-J., Chang, J.-H.: Efficient implementation of quaternion Fourier transform, convolution, and correlation by 2-D complex FFT. IEEE Trans. Signal Process. 49(11), 2783–2797 (2001)

    MathSciNet  MATH  Google Scholar 

  53. Jin, L., Liu, H., Xu, X., Song, E.: Improved direction estimation for Di Zenzo’s multichannel image gradient operator. Pattern Recognit. 45(12), 4300–4311 (2012)

    MATH  Google Scholar 

  54. Di Zenzo, S.: A note on the gradient of a multi-image. Comput. Vis. Graph. Image Process. 33(1), 116–125 (1986)

    MATH  Google Scholar 

  55. Shui, P.-L.: Image denoising algorithm via doubly local Wiener filtering with directional windows in wavelet domain. IEEE Signal Process. Lett. 12(10), 681–684 (2005)

    Google Scholar 

  56. Xiao, S., Hu, J., Wang, Y.: Non-local means with steer search window and adaptive parameter. In: IEEE International Conference on Image Processing, pp. 76–80 (2018)

  57. Vardi, Y., Zhang, C.-H.: A modified Weiszfeld algorithm for the Fermat-Weber location problem. Mathematical Programming A 90, 559–566 (2001)

    MathSciNet  MATH  Google Scholar 

  58. Spence, C., Fancourt, C.: An iterative method for vector median filtering. In: IEEE IEEE International Conference on Image Processing (ICIP), pp. 265–268 (2007)

  59. Zhang, W., Jin, L., Song, E., Xu, X.: Removal of impulse noise in color images based on convolution neural network. Appl. Soft Comput. J. 82, 10558 (2019)

    Google Scholar 

  60. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)

    MathSciNet  MATH  Google Scholar 

  61. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)

  62. Zhang, L., Zhang, L., Mou, X., Zhang, D.: FSIM: a feature similarity index for image quality assessment. IEEE Trans. Image Process. 20(8), 2378–2386 (2011)

    MathSciNet  MATH  Google Scholar 

  63. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of International Conference on Learning Representations, (2015)

Download references

Acknowledgements

The authors would like to thank Dr. Kyong Hwan Jin for providing the program for the algorithm ALOHA [35]. This work was supported by the National Natural Science Foundation of China under Grant No. 61370181.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Lianghai Jin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Wenhua Zhang—deceased.

Appendices

Appendix 1: Proofs of Lemmas 2 and 3

Note that in the following proofs, j denotes the imaginary unit of a complex number.

Proof of Lemma 2

$$ \begin{aligned} F_{f}^{*} (u,v) = & \left( {\int_{ - \infty }^{ + \infty } {\int_{ - \infty }^{ + \infty } {f(x,y){\text{e}}^{ - j(ux + vy)} {\text{d}}x{\text{d}}y} } } \right)^{*} \\ = & \int_{ - \infty }^{ + \infty } {\int_{ - \infty }^{ + \infty } {f^{*} (x,y){\text{e}}^{ - j(ux + vy)} {\text{d}}x{\text{d}}y} } \\ = & F_{{f^{*} }} ( - u, - v) \\ \end{aligned} $$

To prove Lemma 3, the following conclusion is needed:

Lemma 4

Let\( f(x,y) \)denote a complex function defined in\( [ - \infty ,\; + \infty ] \times [ - \infty ,\; + \infty ] \)and\( F_{f} (u,v) \)be its CFT. Then, the following conclusions hold:

  1. (1)

    \( F_{{f_{x}^{{\prime }} }} (u,v) = ju \cdot F_{f} (u,v) \)when\( \mathop {\lim }\limits_{\left| x \right| \to \infty } \left| {f(x,y)} \right| = 0. \)

  2. (2)

    \( F_{{f_{y}^{\prime } }} (u,v) = jv \cdot F_{f} (u,v) \)when\( \mathop {\lim }\limits_{\left| y \right| \to \infty } \left| {f(x,y)} \right| = 0. \)

The proof of Lemma 4 can be found in general textbooks and documents. We no longer prove it again and only give the proof of Lemma 3.

Proof of Lemma 3

$$ \begin{aligned} & \int {\int { \, uv \cdot F(u,v)G^{*} (u,v){\text{d}}u{\text{d}}v} } \\ & \quad = \int {\int {\left( {{\text{j}}u \cdot F(u,v)} \right) \cdot \left( { - jv \cdot G^{*} (u,v)} \right)} } {\text{d}}u{\text{d}}v \\ & \quad = \int {\int { \, \left( {{\text{j}}u \cdot F(u,v)} \right) \cdot \left( { - jv \cdot \int {\int {g^{*} (x,y){\text{e}}^{j(ux + vy)} {\text{d}}x{\text{d}}y} } } \right)} } {\text{d}}u{\text{d}}v \\ & \quad = \int {\int { \, F_{{f_{x}^{{\prime }} }} (u,v) \cdot F_{{\left( {g^{*} } \right)_{y}^{{\prime }} }} ( - u, - v)} } {\text{d}}u{\text{d}}v \\ & \quad = \int {\int { \, F_{{f_{x}^{{\prime }} }} (u,v) \cdot \left( {\int {\int {\frac{{\partial g^{*} (x,y)}}{\partial y}{\text{e}}^{j(ux + vy)} {\text{d}}x{\text{d}}y} } } \right)} } {\text{d}}u{\text{d}}v \\ & \quad = \int {\int { \, \left( {\int {\int {F_{{f_{x}^{{\prime }} }} (u,v) \cdot {\text{e}}^{j(ux + vy)} {\text{d}}u{\text{d}}v} } } \right)} } \cdot \frac{{\partial g^{*} (x,y)}}{\partial y}{\text{d}}x{\text{d}}y \\ & \quad = \frac{1}{{4\pi^{2} }}\int {\int { \, \frac{\partial f(x,y)}{\partial x} \cdot \frac{{\partial g^{*} (x,y)}}{\partial y}} {\text{d}}x{\text{d}}y} \\ \end{aligned} $$

Similarly, by letting \( \int {\int {uv \cdot F(u,v)G^{*} (u,v){\text{d}}u{\text{d}}v} } = \int {\int { \, \left( {jv \cdot F(u,v)} \right) \cdot \left( { - ju \cdot G^{*} (u,v)} \right)} } {\text{d}}u{\text{d}}v, \) we can obtain

$$ \int {\int {uv \cdot F(u,v)G^{*} (u,v){\text{d}}u{\text{d}}v} } = \frac{1}{{4\pi^{2} }}\int {\int { \, \frac{\partial f(x,y)}{\partial y}\frac{\partial g*(x,y)}{\partial x}} {\text{ d}}x{\text{d}}y} $$

In the same way, we have

$$ \begin{aligned} & \int {\int { \, u^{2} \cdot F(u,v)G^{*} (u,v){\text{d}}u{\text{d}}v} } \\ & \quad = \int {\int { \, \left( {ju \cdot F(u,v)} \right) \cdot \left( { - ju \cdot G^{*} (u,v)} \right)} } {\text{d}}u{\text{d}}v \\ & \quad = \int {\int { \, \left( {ju \cdot F(u,v)} \right) \cdot \left( { - ju \cdot \int {\int {g^{*} (x,y){\text{e}}^{j(ux + vy)} {\text{d}}x{\text{d}}y} } } \right)} } {\text{d}}u{\text{d}}v \\ & \quad = \int {\int { \, F_{{f_{x}^{{\prime }} }} (u,v) \cdot F_{{\left( {g^{*} } \right)_{x}^{{\prime }} }} ( - u, - v)} } {\text{d}}u{\text{d}}v \\ & \quad = \int {\int { \, F_{{f_{x}^{{\prime }} }} (u,v) \cdot \left( {\int {\int {\frac{{\partial g_{{}}^{*} (x,y)}}{\partial x}{\text{e}}^{j(ux + vy)} {\text{d}}x{\text{d}}y} } } \right)} } {\text{d}}u{\text{d}}v \\ & \quad = \int {\int { \, \left( {\int {\int {F_{{f_{x}^{{\prime }} }} (u,v) \cdot {\text{e}}^{j(ux + vy)} {\text{d}}u{\text{d}}v} } } \right)} } \cdot \frac{{\partial g^{*} (x,y)}}{\partial x}{\text{d}}x{\text{d}}y \\ & \quad = \frac{1}{{4\pi^{2} }}\int {\int { \, \frac{\partial f(x,y)}{\partial x} \cdot \frac{{\partial g^{*} (x,y)}}{\partial x}} {\text{d}}x{\text{d}}y} \\ \end{aligned} $$

Similarly, we have

$$ \int {\int { \, v^{2} \cdot F(u,v)G^{*} (u,v){\text{d}}u{\text{d}}v} } = \frac{1}{{4\pi^{2} }}\int {\int { \, \frac{\partial f(x,y)}{\partial y} \cdot \frac{{\partial g^{*} (x,y)}}{\partial y}} {\text{d}}x{\text{d}}y} $$

This finishes the proof of Lemma 3.

Appendix 2: Performance Evaluation on Individual Images

This appendix gives the PSNR, MAE, NCD, FSIMc scores on each of the images shown in Fig. 4.

Tables 5, 6, 7, 8, 9, 10.

Table 5 Comparison of the PSNR and MAE results on 16 slightly contaminated images with impulse noise of \( p = 0.1 \)
Table 6 Comparison of the NCD and FSIMc results on 16 slightly contaminated images with impulse noise of \( p = 0.1 \)
Table 7 Comparison of the PSNR and MAE results on 16 mediumly contaminated images with impulse noise of \( p = 0.3 \)
Table 8 Comparison of the NCD and FSIMc results on 16 mediumly contaminated images with impulse noise of \( p = 0.3 \)
Table 9 Comparison of the PSNR and MAE results on 16 heavily contaminated images with impulse noise of \( p = 0.6 \)
Table 10 Comparison of the NCD and FSIMc results on 16 heavily contaminated images with impulse noise of \( p = 0.6 \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, L., Song, E. & Zhang, W. Denoising Color Images Based on Local Orientation Estimation and CNN Classifier. J Math Imaging Vis 62, 505–531 (2020). https://doi.org/10.1007/s10851-019-00942-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-019-00942-8

Keywords

Navigation