Abstract
Digital try-on systems for e-commerce have the potential to change people’s lives and provide notable economic benefits. However, their development is limited by practical constraints, such as accurate sizing of the body and realism of demonstrations. We enumerate three open challenges remaining for a complete and easy-to-use try-on system that recent advances in machine learning make increasingly tractable. For each, we describe the problem, introduce state-of-the-art approaches, and provide future directions.
Article PDF
Similar content being viewed by others
References
Zheng, Z. H.; Zhang, H. T.; Zhang, F. L.; Mu, T. J. Image-based clothes changing system. Computational Visual Media Vol. 3, No. 4, 337–347, 2017.
Dibra, E.; Jain, H.; Öztireli, C.; Ziegler, R.; Gross, M. HS-Nets: Estimating human body shape from silhouettes with convolutional neural networks. In: Proceedings of the 4th International Conference on 3D Vision, 108–117, 2016.
Bălan, A. O.; Black, M. J. The naked truth: Estimating body shape under clothing. In: Computer Vision -ECCV 2008. Lecture Notes in Computer Science, Vol. 5303. Forsyth, D.; Torr, P.; Zisserman, A. Eds. Springer Berlin, 15–29, 2008.
Lassner, C.; Romero, J.; Kiefel, M.; Bogo, F.; Black, M. J.; Gehler, P. V. Unite the people: Closing the loop between 3D and 2D human representations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4704–4713, 2017.
Loper, M.; Mahmood, N.; Romero, J.; Pons-Moll, G.; Black, M. J. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics Vol. 34, No. 6, Article No. 248, 2015.
Wei, S.-E.; Ramakrishna, V.; Kanade, T.; Sheikh, Y. Convolutional pose machines. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 4724–4732, 2016.
Cao, Z.; Simon, T.; Wei, S.; Sheikh, Y. Realtime multiperson 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1302–1310, 2017.
Mehta, D.; Sridhar, S.; Sotnychenko, O.; Rhodin, H.; Shafiei, M.; Seidel, H.-P.; Xu, W.; Casas, D.; Theobalt, C. VNect: Realtime 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 44, 2017.
Alldieck, T.; Magnor, M.; Xu, W.; Theobalt, C.; Pons-Moll, G. Video based reconstruction of 3D people models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8387–8397, 2018.
Kanazawa, A.; Black, M. J.; Jacobs, D. W.; Malik, J. End-to-end recovery of human shape and pose. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7122–7131, 2018.
Varol, G.; Ceylan, D.; Russell, B.; Yang, J.; Yumer, E.; Laptev, I. Bodynet: Volumetric inference of 3D human body shapes. In: Proceedings of the European Conference on Computer Vision, 20–36, 2018.
Zheng, Z.; Yu, T.; Wei, Y.; Dai, Q.; Liu, Y. Deephuman: 3D human reconstruction from a single image. In: Proceedings of the IEEE International Conference on Computer Vision, 7739–7749, 2019.
Saito, S.; Huang, Z.; Natsume, R.; Morishima, S.; Kanazawa, A.; Li, H. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: Proceedings of the IEEE International Conference on Computer Vision, 2304–2314, 2019.
Xu, Y.; Zhu, S.-C.; Tung, T. Denserac: Joint 3D pose and shape estimation by dense render-and-compare. In: Proceedings of the IEEE International Conference on Computer Vision, 7760–7770, 2019.
Smith, D.; Loper, M.; Hu, X.; Mavroidis, P.; Romero, J. FACSIMILE: Fast and accurate scans from an image in less than a second. In: Proceedings of the IEEE International Conference on Computer Vision, 5329–5338, 2019.
Alldieck, T.; Magnor, M.; Bhatnagar, B. L.; Theobalt, C.; Pons-Moll, G. Learning to reconstruct people in clothing from a single RGB camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1175–1186, 2019.
Kolotouros, N.; Pavlakos, G.; Black, M. J.; Daniilidis, K. Learning to reconstruct 3D human pose and shape via modelfitting in the loop. In: Proceedings of the IEEE International Conference on Computer Vision, 2252–2261, 2019.
Liang, J.; Lin, M. C. Shape-aware human pose and shape reconstruction using multi-view images. In: Proceedings of the IEEE International Conference on Computer Vision, 4352–4362, 2019.
Yang, S.; Pan, Z. R.; Amert, T.; Wang, K.; Yu, L. C.; Berg, T.; Lin, M. C. Physics-inspired garment recovery from a single-view image. ACM Transactions on Graphics Vol. 37, No. 5, Article No. 170, 2018.
Yang, S.; Liang, J.; Lin, M. C.; Learning-based cloth material recovery from video. In: Proceedings of the IEEE International Conference on Computer Vision, 4383–4393, 2017.
Qiao, Y. L.; Liang, J. B.; Koltun, V.; Lin, M. C. Scalable differentiable physics for learning and control. arXiv preprint arXiv:2007.02168, 2020.
De Avila Belbute-Peres, F.; Smith, K. A.; Allen, K.; Tenenbaum, J.; Kolter, J. Z. End-to-end differentiable physics for learning and control. In: Proceedings of the Advances in Neural Information Processing Systems, 2018.
Degrave, J.; Hermans, M.; Dambre, J.; Wyffels, F. A differentiable physics engine for deep learning in robotics. Frontiers in Neurorobotics Vol. 13, 6, 2019.
Liang, J.; Lin, M.; Koltun, V. Differentiable cloth simulation for inverse problems. In: Proceedings of the 33rd Conference on Neural Information Processing Systems, 2019.
Hu, Y.; Liu, J.; Spielberg, A.; Tenenbaum, J. B.; Freeman, W. T.; Wu, J.; Rus, D.; Matusik, W. ChainQueen: A real-time differentiable physical simulator for soft robotics. In: Proceedings of the International Conference on Robotics and Automation, 6265–6271, 2019.
Hu, Y. M.; Anderson, L.; Li, T. M.; Sun, Q.; Carr, N.; Ragan-Kelley, J.; Durand, F. DiffTaichi: Differentiable programming for physical simulation. arXiv preprint arXiv:1910.00935, 2019.
Liu, K. X.; Zeng, X. Y.; Bruniaux, P.; Tao, X. Y.; Yao, X. F.; Li, V.; Wang, J. 3D interactive garment patternmaking technology. Computer-Aided Design Vol. 104, 113–124, 2018.
Huang, P.; Yao, J.; Zhao, H. Automatic realistic 3D garment generation based on two images. In: Proceedings of the International Conference on Virtual Reality and Visualization, 250–257, 2016.
Wang, T. Y.; Ceylan, D.; Popović, J.; Mitra, N. J. Learning a shared shape space for multimodal garment design. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 203, 2018.
Acknowledgements
This research was supported in part by the Iribe Professorship and the National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Additional information
Junbang Liang is a 4th-year Ph.D. student in the University of Maryland, College Park. He received his B.E. degree from Tsinghua University in 2016, and his M.S. degree from the University of North Carolina in 2018. His research interests are physics-based cloth simulation, computer vision, and machine learning.
Ming C. Lin is a Distinguished University Professor and Elizabeth Stevinson Iribe Chair of Computer Science at the University of Maryland College Park and John R. and Louise S. Parker Distinguished Professor Emerita of Computer Science at the University of North Carolina, Chapel Hill. She obtained her B.S., M.S., and Ph.D. degrees in electrical engineering and computer science from the University of California, Berkeley. She is a Fellow of ACM, IEEE, and Eurographics, and a member of ACM SIGGRAPH Academy.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Liang, J., Lin, M.C. Machine learning for digital try-on: Challenges and progress. Comp. Visual Media 7, 159–167 (2021). https://doi.org/10.1007/s41095-020-0189-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-020-0189-1