Skip to main content
SpringerLink
Log in

Identity-Preserved Face Beauty Transformation with Conditional Generative Adversarial Networks

  • THE PAPER OF IAPR K.-S. FU PRIZE WINNER
  • Published:
Pattern Recognition and Image Analysis Aims and scope Submit manuscript

Abstract

Identity-preserved face beauty transformation aims to change the beauty scale of a face image while preserving the identity of the original face. In our framework of conditional generative adversarial networks (cGANs), the synthesized face produced by the generator would have the same beauty scale indicated by the input condition. Unlike the discrete class labels used in most cGANs, the condition of target beauty scale in our framework is given by a continuous real-valued beauty score in the range [1–5], which makes the work challenging. To tackle the problem, we have implemented a triple structure, in which the conditional discriminator is divided into a normal discriminator and a separate face beauty predictor. We have also developed another new structure called Conditioned Instance Normalization to replace the original concatenation used in cGANs, which makes the combination of the input image and condition more effective. Furthermore, self-consistency loss is introduced as a new parameter to improve the stability of training and quality of the generated image. In the end, the objectives of beauty transformation and identity preservation are evaluated by the pretrained face beauty predictor and state-of-the-art face recognition network. The result is encouraging, and it also shows that certain facial features could be synthesized by the generator according to the target beauty scale, while preserving the original identity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

Similar content being viewed by others

REFERENCES

  1. G. Antipov, M. Baccouche, and J. Dugelay, “Face aging with conditional generative adversarial networks,” in Proc. IEEE International Conference on Image Processing (ICIP) (2017), pp. 2089–2093.

  2. M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in Proceedings of the 34th International Conference on Machine Learning (2017), Vol. 70, pp. 214–223.

  3. Y. Choi, M. Choi, M. Kim, J. W. Ha, S. Kim, and J. Choo, “StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018), pp. 8789–8797.

  4. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 248–255.

  5. N. Diamant, D. Zadok, C. Baskin, E. Schwartz, and A. M. Bronstein, “Beholder-GAN: Generation and beautification of facial images with conditioning on their beauty level,” in Proc. IEEE International Conference on Image Processing (ICIP) (2019), pp. 739–743.

  6. V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” arXiv (2016). arXiv:1610.07629

  7. L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), pp. 2414–2423.

  8. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” Adv. Neural Inf. Process. Syst. 27, 2672–2680 (2014).

    Google Scholar 

  9. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), pp. 770–778.

  10. X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in Proc. IEEE International Conference on Computer Vision (ICCV) (2017), pp. 1501–1510.

  11. P. Isola, J. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), pp. 5967–5976.

  12. T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” arXiv (2017). arXiv:1710.10196

  13. LI, C., T. Xu, J. Zhu, and B. Zhang, “Triple generative adversarial nets,” Adv. Neural Inf. Process. Syst. 30, 4088–4098 (2017).

    Google Scholar 

  14. T. Li, R. Qian, C. Dong, S. Liu, Q. Yan, W. Zhu, and L. Lin, “BeautyGAN: Instance-level facial makeup transfer with deep generative adversarial network,” in Proc. ACM International Conference on Multimedia (2018), pp. 645–653.

  15. L. Liang, L. Lin, L. Jin, D. Xie, and M. Li, “SCUT-FBP5500: A diverse benchmark dataset for multi-paradigm facial beauty prediction,” in Proc. 24th International Conference on Pattern Recognition (ICPR) (2018), pp. 1598–1603.

  16. L. Lin, L. Liang, and L. Jin, “R2-ResNeXt: A ResNeXt-based regression model with relative ranking for facial beauty prediction,” in Proc. 24th International Conference on Pattern Recognition (ICPR) (2018), pp. 85–90.

  17. X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley, “Least squares generative adversarial networks,” in Proc. IEEE International Conference on Computer Vision (ICCV) (2017), pp. 2813–2821.

  18. M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv (2014). arXiv:1411.1784

  19. O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in Proceedings of the British Machine Vision Conference (BMVC) (2015), pp. 41.1–41.12.

  20. T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” Adv. Neural Inf. Process. Syst. 29, 901–909 (2016).

    Google Scholar 

  21. S. Shi, F. Gao, X. Meng, X. Xu, and J. Zhu, “Improving facial attractiveness prediction via co-attention learning,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2019), pp. 4045–4049.

  22. D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), pp. 4105–4113.

  23. Z. Wang, X. Tang, W. Luo, and S. Gao, “Face aging with identity-preserved conditional generative adversarial networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018), pp. 7939–7947.

  24. J. Xu, L. Jin, L. Liang, Z. Feng, and D. Xie, “A new humanlike facial attractiveness predictor with cascaded fine-tuning deep learning model,” arXiv (2015). arXiv:1511.02465

  25. L. Xu, H. Fan, and J. Xiang, “Hierarchical multi-task network for race, gender and facial attractiveness recognition,” in Proc. IEEE International Conference on Image Processing (ICIP) (2019), pp. 3861–3865.

  26. J. Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. IEEE International Conference on Computer Vision (ICCV) (2017), pp. 2223–2232.

Download references

ACKNOWLEDGMENTS

We would like to appreciate the facilities provided by Concordia University and CENPARMI.

Funding

The work is supported by the Natural Sciences and Engineering Research Council of Canada.

Author information

Authors and Affiliations

  1. CENPARMI (Centre for Pattern Recognition and Machine Intelligence), Department of Computer Science and Software Engineering, Concordia University, Montreal, QC, Canada

    Zhitong Huang & Ching Yee Suen

Authors
  1. Zhitong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ching Yee Suen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zhitong Huang or Ching Yee Suen.

Ethics declarations

COMPLIANCE WITH ETHICAL STANDARDS

This manuscript is a completely original work of its authors; it has not been published before and will not be published in other sources.

CONFLICT OF INTEREST

The content of the article does not give grounds for raising the issue of a conflict of interest.

Additional information

Zhitong Huang. Bachelor of Engineering in Microelectronics at the Sun Yat-Sen University, Bachelor of Engineering in Electronics at the Hong Kong Polytechnic University, and Master of Computer Science at the Concordia University.

Ching Yee Suen. Dr. Ching Y. Suen is the Director of CENPARMI and the Concordia Honorary Chair in AI & Pattern Recognition. He received his PhD degree from UBC Vancouver) and his Master degree from the University of Hong Kong. He has served as the Chairman of the Department of Computer Science and as the Associate Dean (Research) of the Faculty of Engineering and Computer Science of Concordia University. He has guided 120 Doctoral and Master students and hosted more than 100 long-term visitors.

A recipient of the 2020 King-Sun Fu Prize, the ICDAR Award, the ITAC/NSERC National Award and numerous other awards, Prof. Suen has served at numerous national and international professional societies as President, Vice-President, Governor, and Director. He has given 45 invited/keynote and 550 journal and conference papers, and 200 invited talks at various industries and academic institutions around the world. He has been the Principal Investigator or Consultant of 30 industrial projects. His research projects have been funded by the ENCS Faculty and the Distinguished Chair Programs at Concordia University, FCAR (Quebec), NSERC (Canada), the National Networks of Centres of Excellence (Canada), the Canadian Foundation for Innovation, and the industrial sectors in various countries, including Canada, France, Japan, Italy, and the United States. He has served the Pattern Recognition Community as the Editor-in-Chief of the Pattern Recognition journal, and numerous other journals. He is the founder of ICDAR, ICFHR, VI, and ICPRAI and has served as the General Chair and Honourable Chair of numerous international conferences.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, Z., Suen, C.Y. Identity-Preserved Face Beauty Transformation with Conditional Generative Adversarial Networks. Pattern Recognit. Image Anal. 31, 364–375 (2021). https://doi.org/10.1134/S1054661821030123

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S1054661821030123

Keywords:

Search

Navigation