Skip to main content
SpringerLink
Log in

A novel feature transform framework using deep neural network for multimodal floor plan retrieval

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

In recent past, there has been a steep increase in the use of online platforms for the search of desired products. Real estate industry is no exception and has started initiating rent/sale of houses through online platforms. In this paper, we propose a deep neural network framework to facilitate automatic search of homes based on their floor plans. The salient features of this framework are that the query can be either an image (existing floor plan) or a sketch through a sketch pad interface. Our proposed framework automatically determines the type of query (image or sketch) and retrieves similar floor plan images from the database. The critical contributions of our proposed approach are: (1) a novel unified floor plan retrieval framework using multimodal query, i.e., an intuitive and convenient sketch query mode as well as a query by example mode ; (2) a conjunction of autoencoder, Cyclic GAN and CNN for the task of domain mapping and floor plan image retrieval. We have reported results of extensive experimentation and comparison with baseline results to establish the effectiveness of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ahmed, S., Liwicki, M., Weber, M., Dengel, A.: Automatic room detection & room labeling from architectural floor plans. In: DAS (2012)

  2. Ahmed, S., Weber, M., Liwicki, M., Dengel, A., Petzold, F.: Automatic analysis and sketch-based retrieval of architectural floor plans. PRL 35, 91–100 (2014)

    Article  Google Scholar 

  3. Chechik, G., Shalit, U., Sharma, V., Bengio, S.: An online algorithm for large scale image similarity learning. In: NIPS (2009)

  4. Chen, W., Hays, J.: Sketchygan: towards diverse and realistic sketch to image synthesis. arXiv:1801.02753 (2018)

  5. Conte, D., Foggia, P., Sansone, C., Vento, M.: Thirty years of graph matching in pattern recognition. IJPRAI 18(03), 265–298 (2004)

    Google Scholar 

  6. Creswell, A., Bharath, A.A.: Task specific adversarial cost function. arXiv:1609.08661 (2016)

  7. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005)

  8. de las Heras, L., Mota, D.F., Fornés, A., Valveny, E., Sánchez, G., Lladós, J.: Runlength histogram image signature for perceptual retrieval of architectural floor plans. In: GREC (2013)

  9. de las Heras, L.P., Terrades, O.R., Robles, S., Sánchez, G.: CVC-FP and SGT6: a new database for structural floor plan analysis and its groundtruthing tool. IJDAR 18(1), 15–30 (2015)

    Article  Google Scholar 

  10. Delalandre, M., Valveny, E., Ramel, J.-Y. Recent contributions on the SESYD dataset for performance evaluation of symbol spotting systems (2011). https://www.semanticscholar.org/paper/Recent-contributions-on-the-SESYD-dataset-for-of-Delalandre-Valveny/42a3d89544393fe80acb6d6c4eae0239c9c96b99. Accessed Dec 2017

  11. Dutta, A., Lladós, J., Bunke, H., Pal, U.: Near convex region adjacency graph and approximate neighborhood string matching for symbol spotting in graphical documents. In: ICDAR (2013)

  12. Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: benchmark and bag-of-features descriptors. TVCG 17, 1624–1636 (2011)

    Google Scholar 

  13. Ghifary, M., Kleijn, W.B., Zhang, M., Balduzzi, D., Li, W.: Deep reconstruction-classification networks for unsupervised domain adaptation. In: ECCV, pp. 597–613. Springer (2016)

  14. Goncu, C., Madugalla, A., Marinai, S., Marriott, K.: Accessible on-line floor plans. In: Proceedings of the 24th international conference on world wide web, pp. 388–398. International World Wide Web Conferences Steering Committee (2015)

  15. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. arXiv:1408.5093 (2014)

  16. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: ANIPS (2012)

  17. Lamiroy, B., Ogier, J.M.: Analysis and interpretation of graphical documents. In: Doermann, D., Tombre, K. (eds.) Handbook of Document Image Processing and Recognition, pp. 553–590. Springer, London

    Chapter  Google Scholar 

  18. Le Bodic, P., Héroux, P., Adam, S., Lecourtier, Y.: An integer linear program for substitution-tolerant subgraph isomorphism and its use for symbol spotting in technical drawings. PR 45(12), 4214–4224 (2012)

    Google Scholar 

  19. Lladós, J., Martí, E., Villanueva, J.J.: Symbol recognition by error-tolerant subgraph matching between region adjacency graphs. PAMI 23(10), 1137–1143 (2001)

    Article  Google Scholar 

  20. Locteau, H., Adam, S., Trupin, E., Labiche, J., Héroux, P.: Symbol spotting using full visibility graph representation. In: GREC, pp. 49–50 (2007)

  21. Lowe, D.G.: Object recognition from local scale-invariant features. In: ICCV (1999)

  22. Messmer, B.T., Bunke, H.: Automatic learning and recognition of graphical symbols in engineering drawings. In: GREC, pp. 123–134. Springer (1995)

  23. Nayef, N.: Geometric-based symbol spotting and retrieval in technical line drawings (2013)

  24. Qureshi, R.J., Ramel, J.Y., Barret, D., Cardot, H.: Spotting symbols in line drawing images using graph representations. In: International workshop on graphics recognition, pp. 91–103. Springer (2007)

  25. Santosh, K.: Complex and composite graphical symbol recognition and retrieval: a quick review. In: International conference on recent trends in IP and PR, pp. 3–15. Springer (2016)

  26. Santosh, K.: Document Image Analysis—Current Trends and Challenges in Graphics Recognition, pp. 1–174. Springer, Berlin (2018)

    Book  Google Scholar 

  27. Santosh, K., Lamiroy, B., Wendling, L.: Integrating vocabulary clustering with spatial relations for symbol recognition. IJDAR 17(1), 61–78 (2014)

    Article  Google Scholar 

  28. Santosh, K., Wendling, L.: Graphical Symbol Recognition, pp. 1–22. Wiley, Hoboken (1999)

    Google Scholar 

  29. Santosh, K., Wendling, L., Lamiroy, B.: Bor: bag-of-relations for symbol retrieval. IJPRAI 28(06), 1450017 (2014)

    Google Scholar 

  30. Sharma, D., Chattopadhyay, C.: High-level feature aggregation for fine-grained architectural floor plan retrieval. IET CV 12, 702–709 (2018)

    Google Scholar 

  31. Sharma, D., Chattopadhyay, C.: ROBIN and S-ROBIN dataset (2017). https://github.com/gesstalt/ROBIN.git (2018). Accessed 10 Jan 2018

  32. Sharma, D., Gupta, N., Chattopadhyay, C., Mehta, S.: DANIEL: a deep architecture for automatic analysis and retrieval of building floor plans. In: ICDAR (2017)

  33. Vento, M.: A long trip in the charming world of graphs for pattern recognition. PR 48(2), 291–301 (2015)

    MATH  Google Scholar 

  34. Weber, M., Liwicki, M., Dengel, A.: A scatch-a sketch-based retrieval for architectural floor plans (2010)

  35. Yamasaki, T., Zhang, J., Takada, Y.: Apartment structure estimation using fully convolutional networks and graph model. In: ACM workshop on multimedia for real estate tech, pp. 1–6. ACM (2018)

  36. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv:1703.10593 (2017)

  37. Zhuang, F., Cheng, X., Luo, P., Pan, S.J., He, Q.: Supervised representation learning: transfer learning with deep autoencoders. In: IJCAI, pp. 4119–4125 (2015)

  38. Ziran, Z., Marinai, S.: Object detection in floor plan images. In: IAPR workshop on artificial neural networks in pattern recognition, pp. 383–394. Springer (2018)

Download references

Acknowledgements

This research was partially funded by Science and Engineering Research Board (SERB), Government of India Grant Number ECR/2016/000953.

Author information

Authors and Affiliations

  1. Indian Institute of Technology Jodhpur, Jodhpur, 342037, India

    D. Sharma & C. Chattopadhyay

  2. IBM Research Labs, New Delhi, India

    N. Gupta & S. Mehta

Authors
  1. D. Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. N. Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. C. Chattopadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. S. Mehta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to C. Chattopadhyay.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, D., Gupta, N., Chattopadhyay, C. et al. A novel feature transform framework using deep neural network for multimodal floor plan retrieval. IJDAR 22, 417–429 (2019). https://doi.org/10.1007/s10032-019-00340-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-019-00340-1

Keywords

Search

Navigation