Skip to main content
SpringerLink
Log in

Fashion clothes matching scheme based on Siamese Network and AutoEncoder

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Owing to the rise of living standard, people attach greater importance to personal appearance, especially clothes matching. With image processing and machine learning technology, we can analyze the pattern of clothes matching for recommendation on clothes images. However, we still face great challenges. To be more specific, there exist excessive complicated factors influencing relation among clothes items, such as color or material, and we also struggle against the problem about how to extract efficient and accurate features. Thus, with the purpose of dealing with such challenges, this paper proposes an efficient clothes matching scheme with Siamese Network and AutoEncoder based on both labeled data from dataset FashionVC and unlabeled data from MicroBlog. More specifically, at first, except for clothes suiting with text from FashionVC, the gallery data also include matching clothes outfits recommended by fashionista in MicroBlog (MbFashion). Meanwhile, a semi-supervised clustering based on assembling was also proposed to generate negative samples to form a comprehensive dataset. Secondly, with consideration of matching patterns from MbFashion, we promoted the Siamese Network properly to more efficiently extract vision features on the constructed training dataset. After that, the traditional features are also extracted, while the Triple AutoEncoder and Bayesian Personalized Ranking are used to map the three kinds of features into the same latent space to learn the compatibility between tops and bottoms. Finally, we conducted a series of experiments and evaluated our results to demonstrate the usefulness and effectiveness of the whole scheme on FashionVC and MbFashion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Liu, S., Liu, L., Yan, S.: Fashion analysis: current techniques and future directions. IEEE Multimed 21(2), 72–79 (2014)

    Article  MathSciNet  Google Scholar 

  2. Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8), 30–37 (2009)

    Article  Google Scholar 

  3. Salakhutdinov, R., Mnih, R.: Bayesian probabilistic matrix factorization using markov chain monte carlo. In: Proceedings of the 25th ACM international conference on machine learning, pp. 880–887 (2008)

  4. Kalantidis, Y., Kennedy, L., Li, L.: Getting the look: clothing recognition and segmentation for automatic product suggestions in everyday photos. In: Proceedings of Int’l Conf. on Multimedia Retrieval, pp. 105–112 (2013)

  5. Song, X., Feng, F., Liu, J., et al.: NeuroStylist: neural compatibility modeling for clothing matching. In: Proceedings of the ACM Int’l Conf. on Multimedia, pp. 753–761 (2017)

  6. Rendle, S., Freudenthaler, C., Gantner, Z., et al.: BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of Conf. on Uncertainty in Artificial Intelligence, pp. 452–461 (2009)

  7. Bromley, J., Guyon, I., Lecun, Y., Sckinger, E., Shah, R.: Signature verification using a “Siamese” time delay neural network. In: Proceedings of NIPS, pp. 737–744 (1994)

    Google Scholar 

  8. Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: Proceedings of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 4353–4361 (2015)

  9. Yamaguchi, K., Hadi Kiapour, M., Berg, T. L.: Paper doll parsing: Retrieving similar styles to parse clothing items. In: Proceedings of IEEE Int’l Conf. on Computer Vision, pp. 3519–3526 (2013)

  10. Yamaguchi, K., Kiapour, M. H., Ortiz, L. E., Berg, T. L.: Parsing clothing in fashion photographs. In: Proceedings of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 3570–3577 (2012)

  11. Dong, J., Chen, Q., Shen, X., Yang, J., Yan, S.: Towards unified human parsing and pose estimation. In: Proceedings of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 843–850 (2014)

  12. Kalantidis, Y., Kennedy, L., Li, L.: Getting the look: clothing recognition and segmentation for automatic product suggestions in everyday photos. In: Proceeding of Int’l Conf. on Multimedia Retrieval, pp. 105–112 (2013)

  13. Liu, S., Song, Z., Liu, G., Xu, C., Lu, H., Yan, S.: Street-to-shop: cross-scenario clothing retrieval via parts alignment and auxiliary set. In: in IEEE Confer. on Computer Vision and Pattern Recognition, pp. 3330–3337 (2012)

  14. Kiapour, M., Han, X., Lazebnik, S., Berg, A. C., Berg, T. L.: Where to buy it: matching street clothing photos in online shops. In: Proceedings of IEEE Int’l Conf. On Computer Vision, pp. 3343–3351 (2015)

  15. Huang, C. M., Wei, C. P., Wang, Y. C. F.: Active learning based clothing image recommendation with implicit user preferences. In: Proceedings of IEEE Int’l Conf. on Multimedia and Expo Workshops, pp. 1–4 (2013)

  16. Hu, Y., Yi, X., Davis, L. S.: Collaborative fashion recommendation: a functional tensor factorization approach. In: Proceedings of the 23rd ACM Int’l Conf. on Multimedia, pp. 129–138 (2015)

  17. McAuley, J., Targett, C., Shi, Q., Van Den Hengel. A.: Image-based recommendations on styles and substitutes. In: Proceedings of the Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval, pp. 43–52 (2015)

  18. Chen, J., Zhang, H., He, X., Nie, L., Liu, W., Chua, T.-S.: Attentive collaborative filtering: multimedia recommendation with item- and component-level attention. In: Proceeding of ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 335–344 (2017)

  19. He, X., He, Z., Song, J., Liu, Z., Jiang, Y.-G., Chua, T.-S.: NAIS: neural attentive item similarity model for recommendation. IEEE Trans. Knowl. Data Eng. 30(12), 2354–2366 (2018)

    Article  Google Scholar 

  20. Liu, S., Feng, J., Song, Z., Zhang, T., Lu, H., Xu, C., Yan, S.: Hi, magic closet, tell me what to wear!. In: Proceedings of the ACM Int’l Conf. on Multimedia, pp. 1333–1334 (2012)

  21. Vartak, M., Madden, S.: CHIC: a combination-based recommendation system. In: Proceedings of ACM SIGMOD Int’l Conf. on Management of Data, pp. 981–984 (2013)

  22. Veit, A., Kovacs, B., Bell, S. et al.: Learning visual clothing style with heterogeneous dyadic co-occurrences. In: Proceedings of IEEE Int’l Conf. on Computer Vision, pp. 4642–4650 (2015)

  23. Liu, S., Liu, L., Yan, S.: Fashion Analysis: current techniques and future directions. IEEE Multimed. 21(2), 72–79 (2014)

    Article  MathSciNet  Google Scholar 

  24. Yu, W., Zhang, H., He, X., Chen, X., Xiong, L., Qin, Z.: Aesthetic-based clothing recommendation. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web, pp. 649–658 (2018)

  25. Song, X., Feng, F., Han, X., Yang, X., Liu, W., Nie, L.: Neural Compatibility Modeling with Attentive Knowledge Distillation. In: Proceedings of the 41st Int’l ACM SIGIR Conf. on Research & Development in Information Retrieval, pp. 5–14 (2018)

  26. He, X., He, Z., Du, X., Chua, T. S.: Adversarial personalized ranking for recommendation. In: Proceedings of The 41st Int’l ACM SIGIR Conf. on Research & Development in Information Retrieval, pp. 355–364 (2018)

  27. Liu, Z., Cheng, L., Liu, A., Zhang, L., He, X., Zimmermann, R.: Multiview and multimodal pervasive indoor localization. In: Proceedings of the ACM on Multimedia Conference, pp. 109–117 (2017)

  28. Chen, J., Song, X., Nie, L., Wang, X., Zhang, H., Chua, T.-S.: Micro Tells Macro: Predicting the Popularity of Micro-Videos via a Transductive Model. In: Proceedings of the ACM on Multimedia, pp. 898–907 (2016)

  29. Demers, D., Cottrell, G. W.: Non-linear dimensionality reduction. In: Proceedings of Neural Information Processing Systems, pp. 580–587 (1992)

  30. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  Google Scholar 

  31. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)

    Article  Google Scholar 

  32. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7), 971–987 (2002)

    Article  Google Scholar 

  33. Smith, A. R.: Color gamut transformation pairs. In: Proceedings of the 5th annual Conf. on Computer Graphics and Interactive Techniques, pp. 12–19 (1978)

Download references

Author information

Authors and Affiliations

  1. Beijing Institute of Technology, Beijing, China

    Guangyu Gao, Liling Liu & Yihang Zhang

  2. Earth Observation System and Data Center, Beijing, China

    Li Wang

Authors
  1. Guangyu Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liling Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yihang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangyu Gao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the Joint Funds of the National Natural Science Foundation of China (Grant no. U1736117).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, G., Liu, L., Wang, L. et al. Fashion clothes matching scheme based on Siamese Network and AutoEncoder. Multimedia Systems 25, 593–602 (2019). https://doi.org/10.1007/s00530-019-00617-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-019-00617-9

Keywords

Search

Navigation