Skip to main content
SpringerLink
Log in

Deep CNN Based Lmser and Strengths of Two Built-In Dualities

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Least mean square error reconstruction for the self-organizing network (Lmser) was proposed in 1991, featured by a bidirectional architecture with several built-in natures. In this paper, we developed Lmser into CNN based Lmser (CLmser), highlighted by new findings on strengths of two major built-in natures of Lmser, namely duality in connection weights (DCW) and duality in paired neurons (DPN). Shown by experimental results on several real benchmark datasets, DCW and DPN bring to us relative strengths in different aspects. While DCW and DPN can both improve the generalization ability of the reconstruction model on small-scale datasets and ease the gradient vanishing problem, DPN plays the main role. Meanwhile, DPN can form shortcut links from the encoder to the decoder to pass detailed information, so it can enhance the performance of image reconstruction and enables CLmser to outperform some recent state-of-the-art methods in image inpainting with irregularly and densely distributed point-shaped masks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://github.com/naoto0804/pytorch-inpainting-with-partial-conv.

  2. https://github.com/researchmm/PEN-Net-for-Inpainting.

  3. https://github.com/lyndonzheng/Pluralistic-Inpainting.

References

  1. Ballard DH (1987) Modular learning in neural networks. In: Proceedings of the sixth national conference on artificial intelligence—volume 1, AAAI’87, pp 279–284

  2. Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on computer graphics and interactive techniques, SIGGRAPH ’00. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, pp 417–424

  3. Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010. Springer, pp 177–186

  4. Bourlard H, Kamp Y (1988) Auto-association by multilayer perceptrons and singular value decomposition. Biol Cybern 59(4–5):291–294

    Article  MathSciNet  Google Scholar 

  5. Coates A, Ng A, Lee H (2011) An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp 215–223

  6. Han K, Wen H, Zhang Y, Fu D, Culurciello E, Liu Z (2018) Deep predictive coding network with local recurrent processing for object recognition. In: Advances in neural information processing systems, pp 9201–9213

  7. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  8. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems, pp 6626–6637

  9. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  10. Huang G, Liu Z, Weinberger KQ (2016) Densely connected convolutional networks. CoRR arXiv:1608.06993

  11. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: CVPR

  12. Karras T, Aila T, Laine S, Lehtinen J (2018) Progressive growing of GANs for improved quality, stability, and variation. In: International conference on learning representations. https://openreview.net/forum?id=Hk99zCeAb

  13. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. CoRR arXiv:1412.6980

  14. Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images

  15. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  16. Liu G, Reda FA, Shih KJ, Wang T, Tao A, Catanzaro B (2018) Image inpainting for irregular holes using partial convolutions. CoRR arXiv:1804.07723

  17. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of the IEEE international conference on computer vision, pp. 3730–3738

  18. Mao X, Shen C, Yang YB (2016) Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in neural information processing systems, pp 2802–2810

  19. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241

  20. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536

    Article  Google Scholar 

  21. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  22. Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747

  23. Xu L (1991) Least MSE reconstruction for self-organization: (i)&(ii). In: Proceedings of 1991 international joint conference on neural networks, pp 2363–2373

  24. Xu L (1993) Least mean square error reconstruction principle for self-organizing neural-nets. Neural Netw 6(5):627–648

    Article  Google Scholar 

  25. Xu L (2019) An overview and perspectives on bidirectional intelligence: Lmser duality, double IA harmony, and causal computation. IEEE/CAA J Autom Sin 6(4):865–893

    Article  MathSciNet  Google Scholar 

  26. Zeng Y, Fu J, Chao H, Guo B (2019) Learning pyramid-context encoder network for high-quality image inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1486–1494

  27. Zheng C, Cham TJ, Cai J (2019) Pluralistic image completion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1438–1447

  28. Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A (2017) Places: a 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1452–1464

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Science and Technology Innovation 2030 Major Project (2018AAA0100700) of the Ministry of Science and Technology of China, and SJTU Medical Engineering Cross-cutting Research Foundation (ZH2018ZDA07), as well as ZhiYuan Chair Professorship Start-up Grant (WF220103010) from Shanghai Jiao Tong University.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Wenjing Huang, Shikui Tu & Lei Xu

Authors
  1. Wenjing Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shikui Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Xu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

A Implementations

For PConv [16], we use an implementation in PyTorch.Footnote 1 For PEN-Net [26], we use their official implementationFootnote 2 and follow all their original settings. For PICNet [27], we use their official implementationFootnote 3 and follow all the settings in their original work except that we start training from the provided pre-trained models and training for 1,000,000 iterations for CelebA-HQ and 2,000,000 iterations for Places2.

B Additional Results on Reconsturction Capability

To further investigate the reconstruction capability of CLmser-n and AE, we conduct experiments on Places2 dataset. We train the two models on 180,346 samples randomly chosen from the train set of Places2 and test them on 10,000 images which are randomly chosen from the test set of Places2. The results are listed in Table 8.

Table 8 Performance of image reconstruction on Places2
Full size table

The results show that the design of the symmetric weights (DCW) does not affect the reconstruction performance.

C Qualitative Examples

See Fig. 8.

Fig. 8
figure 8

Qualitative comparisons of image inpainting on Places2: (columns left to right) masked image, CLmser, CLmser-w, CLmser-n, CAE, PConv, PEN-Net, PICNet, ground truth

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, W., Tu, S. & Xu, L. Deep CNN Based Lmser and Strengths of Two Built-In Dualities. Neural Process Lett 54, 3565–3581 (2022). https://doi.org/10.1007/s11063-020-10341-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-020-10341-5

Keywords

Search

Navigation