Skip to main content
SpringerLink
Log in

Deep traffic sign detection and recognition without target domain real images

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Deep learning has become a standard approach to machine vision in recent years. Despite several advances, it requires large amounts of annotated data. Nonetheless, in many applications, large-scale data acquisition and annotation is expensive and data imbalance is an intrinsic problem. To address these challenges, we propose a novel synthetic database generation method that only requires (i) arbitrary natural images, i.e., does not demand real images from the target domain, and (ii) templates of the traffic signs. Our method does not aim at overcoming the training with real data but to be a compatible option when there is a lack of real data. Results with data of multiple countries show that the synthetic database generated without human effort is effective for training a deep traffic sign detector. On large datasets, training with a fully synthetic dataset almost matches the performance of training with a real one. When compared to training with a smaller dataset of real images, training with synthetic images increased the accuracy by 12.25%. The proposed method also improves the performance of the detector when target-domain data are available.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. The link will be available upon acceptance.

  2. https://github.com/endernewton/tf-faster-rcnn.

References

  1. Álvaro, Arcos-García., Álvarez García, J..A., Soria-Morillo, L..M.: Evaluation of deep neural networks for traffic sign detection systems. Neurocomputing 316, 332–344 (2018)

    Article  Google Scholar 

  2. Badue, C., Guidolini, R., Carneiro, R.V., Azevedo, P., Cardoso, V.B., Forechi, A., Jesus, L., Berriel, R., Paixão, T.M., Mutz, F., et al.: Self-driving cars: a survey. Expert Syst. Appl. 165, 113816 (2020)

    Article  Google Scholar 

  3. Barnes, N., Zelinsky, A., Fletcher, L.S.: Real-time speed sign detection using the radial symmetry detector. IEEE Trans. Intell. Transp. Syst. 9(2), 322–332 (2008)

    Article  Google Scholar 

  4. Berriel, R.F., Torres, L.T., Cardoso, V.B., Guidolini, R., Badue, C., Souza, A.F.D., Oliveira-Santos, T.: Heading direction estimation using deep learning with automatic large-scale data acquisition. In: International Joint Conference on Neural Networks, (IJCNN) (2018)

  5. Carrasco, J..P., de la Escalera, A..d.l, Armingol, J..M., et al.: Recognition stage for a speed supervisor based on road sign detection. Sensors 12(9), 12153–12168 (2012)

    Article  Google Scholar 

  6. Ciresan, D.C., Meier, U., Masci, J., Schmidhuber, J.: A committee of neural networks for traffic sign classification. In: International Joint Conference on Neural Networks (IJCNN) (2011)

  7. Cireşan, D., Meier, U., Masci, J., Schmidhuber, J.: Multi-column deep neural network for traffic sign classification. Neural Netw. 32, 333–338 (2012)

    Article  Google Scholar 

  8. De La Escalera, A., Armingol, J.M., Salichs, M.: Traffic sign detection for driver support systems. In: International Conference on Field and Service Robotics (2001)

  9. Devaranjan, J., Kar, A., Fidler, S.: Meta-sim2: unsupervised learning of scene structure for synthetic data generation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) European Conference on Computer Vision (ECCV) (2020)

  10. Dvornik, N., Mairal, J., Schmid, C.: Modeling visual context is key to augmenting object detection datasets. In: European Conference on Computer Vision (ECCV) (2018)

  11. Dwibedi, D., Misra, I., Hebert, M.: Cut, paste and learn: surprisingly easy synthesis for instance detection. In: International Conference on Computer Vision (ICCV) (2017)

  12. Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge: a retrospective. Int. J. Comput. Vis. (IJCV) 111(1), 98–136 (2015)

    Article  Google Scholar 

  13. Fang, C.Y., Chen, S.W., Fuh, C.S.: Road-sign detection and tracking. IEEE Trans. Veh. Technol. 52(5), 1329–1341 (2003)

    Article  Google Scholar 

  14. Feng, D., Harakeh, A., Waslander, S.L., Dietmayer, K.: A review and comparative study on probabilistic object detection in autonomous driving. IEEE Trans. Intell. Transp. Syst. 1–20 (2021)

  15. Georgakis, G., Mousavian, A., Berg, A., Kosecka, J.: Synthesizing training data for object detection in indoor scenes. In: Robotics: Science and Systems XIII. Robotics: Science and Systems Foundation (2017). https://doi.org/10.15607/RSS.2017.XIII.043

  16. Grigorescu, S.M.: Generative one-shot learning (GOL): a semi-parametric approach to one-shot learning in autonomous vision. In: International Conference on Robotics and Automation (ICRA) (2018)

  17. Gupta, A., Vedaldi, A., Zisserman, A.: Synthetic data for text localisation in natural images. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  19. Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., Igel, C.: Detection of traffic signs in real-world images: the German traffic sign detection benchmark. In: International Joint Conference on Neural Networks (IJCNN) (2013)

  20. Kim, H.K., Park, J.H., Jung, H.Y.: An efficient color space for deep-learning based traffic light recognition. J. Adv. Transp. 2018 (2018)

  21. Kim, J., Oh, T.H., Lee, S., Pan, F., Kweon, I.S.: Variational prototyping-encoder: one-shot learning with prototypical images. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  22. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2012)

  23. Lin, C.C., Wang, M.S.: Road sign recognition with fuzzy adaptive pre-processing models. Sensors 12(5), 6415–6433 (2012)

    Article  MathSciNet  Google Scholar 

  24. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: European Conference on Computer Vision (ECCV) (2014)

  25. Loy, G., Barnes, N.: Fast shape-based road sign detection for a driver assistance system. In: International Conference on Intelligent Robots and Systems (IROS) (2004)

  26. Maldonado-Bascon, S., Lafuente-Arroyo, S., Gil-Jimenez, P., Gomez-Moreno, H., Lopez-Ferreras, F.: Road-sign detection and recognition based on support vector machines. IEEE Trans. Intell. Transp. Syst. 8(2), 264–278 (2007)

    Article  Google Scholar 

  27. Møgelmose, A., Trivedi, M.M., Moeslund, T.B.: Learning to detect traffic signs: comparative evaluation of synthetic and real-world datasets. In: International Conference on Pattern Recognition (ICPR) (2012)

  28. Moiseev, B., Konev, A., Chigorin, A., Konushin, A.: Evaluation of traffic sign recognition methods trained on synthetically generated data. In: Advanced Concepts for Intelligent Vision Systems, Lecture Notes in Computer Science, pp. 576–583. Springer (2013)

  29. Peng, X., Sun, B., Ali, K., Saenko, K.: Learning deep object detectors from 3D models. In: International Conference on Computer Vision (ICCV) (2015)

  30. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), 313–318 (2003)

    Article  Google Scholar 

  31. Possatti, L.C., Guidolini, R., Cardoso, V.B., Berriel, R.F., Paixão, T.M., Badue, C., De Souza, A.F., Oliveira-Santos, T.: Traffic light recognition using deep learning and prior maps for autonomous cars. In: International Joint Conference on Neural Networks (IJCNN) (2019)

  32. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)

    Article  Google Scholar 

  33. Ruta, A., Li, Y., Liu, X.: Detection, tracking and recognition of traffic signs from video input. In: International Conference on Intelligent Transportation Systems (2008)

  34. Saadna, Y., Behloul, A.: An overview of traffic sign detection and classification methods. Int. J. Multim. Inf. Retr. 6(3), 193–210 (2017)

    Article  Google Scholar 

  35. Sarcinelli, R., Guidolini, R., Cardoso, V.B., Paixão, T.M., Berriel, R.F., Azevedo, P., Souza, A.F.D., Badue, C., Oliveira-Santos, T.: Handling pedestrians in self-driving cars using image tracking and alternative path generation with Frenét frames. Comput. Graph. 84, 173–184 (2019)

    Article  Google Scholar 

  36. Shen, L., You, L., Peng, B., Zhang, C.: Group multi-scale attention pyramid network for traffic sign detection. Neurocomputing 452, 1–14 (2021)

    Article  Google Scholar 

  37. Stallkamp, J., Schlipsing, M., Salmen, J., Igel, C.: Man vs. computer: benchmarking machine learning algorithms for traffic sign recognition. Neural Netw. 32, 323–332 (2012)

    Article  Google Scholar 

  38. Stergiou, A., Kalliatakis, G., Chrysoulas, C.: Traffic sign recognition based on synthesised training data. Big Data Cognit. Comput. 2, 19 (2018)

    Article  Google Scholar 

  39. Temel, D., Chen, M.H., AlRegib, G.: Traffic sign detection under challenging conditions: a deeper look into performance variations and spectral characteristics. IEEE Trans. Intell. Transp. Syst. 21, 3663–3673 (2019)

    Article  Google Scholar 

  40. Timofte, R., Zimmermann, K., Van Gool, L.: Multi-view traffic sign detection, recognition, and 3D localisation. Mach. Vis. Appl. 25(3), 633–647 (2014). https://doi.org/10.1007/s00138-011-0391-3

    Article  Google Scholar 

  41. Torres, L.T., Paixão, T.M., Berriel, R.F., Souza, A.F.D., Badue, C., Sebe, N., Oliveira-Santos, T.: Effortless deep training for traffic sign detection using templates and arbitrary natural images. In: International Joint Conference on Neural Networks, (IJCNN) (2019)

  42. Varun, S., Singh, S., Kunte, R.S., Samuel, R.S., Philip, B.: A road traffic signal recognition system based on template matching employing tree classifier. In: International Conference on Computational Intelligence and Multimedia Applications (ICCIMA) (2007)

  43. Wali, S.B., Abdullah, M.A., Hannan, M.A., Hussain, A., Samad, S.A., Ker, P.J., Mansor, M.B.: Vision-based traffic sign detection and recognition systems: current trends and challenges. Sensors 19(9), 2093 (2019). https://doi.org/10.3390/s19092093

    Article  Google Scholar 

  44. Wang, H., Wang, Q., Yang, F., Zhang, W., Zuo, W.: Data augmentation for object detection via progressive and selective instance-switching (2019). arXiv:1906.00358 [cs]

  45. Wu, Y., Liu, Y., Li, J., Liu, H., Hu, X.: Traffic sign detection based on convolutional neural networks. In: International Joint Conference on Neural Networks (IJCNN) (2013)

  46. Zaklouta, F., Stanciulescu, B.: Real-time traffic-sign recognition using tree classifiers. IEEE Trans. Intell. Transp. Syst. 13(4), 1507–1514 (2012)

    Article  Google Scholar 

  47. Zhang, J., Xie, Z., Sun, J., Zou, X., Wang, J.: A cascaded R-CNN with multiscale attention and imbalanced samples for traffic sign detection. IEEE Access 8, 29742–29754 (2020)

    Article  Google Scholar 

  48. Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., Hu, S.: Traffic-sign detection and classification in the wild. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

Download references

Acknowledgements

This study was financed by the Coordenação de Aperfeiçoamento de Pessoal de Ní­vel Superior - Brasil (CAPES) - Finance Code 001, Conselho Nacional de Desenvolvimento Cientí­fico e Tecnológico (CNPq, Brazil), PIIC UFES, Fundaçao de Amparo à Pesquisa do Espí­rito Santo - Brasil (FAPES) - grants 2021-07kj2 and 84412844, and the European Commission under European Horizon 2020 Programme, grant number 951911 - AI4Media. The authors thank NVIDIA Corporation for the donation of the GPUs used in this research.

Author information

Authors and Affiliations

  1. Universidade Federal do Espírito Santo, Vitória, Brazil

    Lucas Tabelini, Rodrigo Berriel, Alberto F. De Souza, Claudine Badue & Thiago Oliveira-Santos

  2. Instituto Federal do Espírito Santo, Serra, Brazil

    Thiago M. Paixão

  3. Università degli Studi di Trento, Trento, Italy

    Nicu Sebe

Authors
  1. Lucas Tabelini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rodrigo Berriel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thiago M. Paixão
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alberto F. De Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Claudine Badue
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nicu Sebe
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Thiago Oliveira-Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lucas Tabelini.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 11090 KB)

Supplementary file 2 (mp4 8337 KB)

Supplementary file 3 (mp4 27226 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tabelini, L., Berriel, R., Paixão, T.M. et al. Deep traffic sign detection and recognition without target domain real images. Machine Vision and Applications 33, 50 (2022). https://doi.org/10.1007/s00138-022-01302-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-022-01302-0

Keywords

Search

Navigation