Skip to main content
SpringerLink
Log in

Road surface detection and differentiation considering surface damages

  • Original Research
  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

A challenge still to be overcome in the field of visual perception for vehicle and robotic navigation on heavily damaged and unpaved roads is the task of reliable path and obstacle detection. The vast majority of the researches have scenario roads in good condition, from developed countries. These works cope with few situations of variation on the road surface and even fewer situations presenting surface damages. In this paper we present an approach for road detection considering variation in surface types, identifying paved and unpaved surfaces and also detecting damage and other information on other road surfaces that may be relevant to driving safety. Our approach makes use of Convolutional Neural Networks (CNN) to perform semantic segmentation, we use the U-NET architecture with ResNet34, in addition we use the technique known as Transfer Learning, where we first train a CNN model without using weights in the classes as a basis for a second CNN model where we use weights for each class. We also present a new Ground Truth with image segmentation, used in our approach and that allowed us to evaluate our results. Our results show that it is possible to use passive vision for these purposes, even using images captured with low cost cameras.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://www.cityscapes-dataset.com/.

  2. http://www.cvlibs.net/datasets/kitti/raw_data.php.

  3. http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/.

  4. http://tev.fbk.eu/databases/diplodoc-road-stereo-sequence.

  5. https://github.com/sekilab/RoadDamageDetector/.

  6. http://www.lrm.icmc.usp.br/dataset.

  7. http://www.lapix.ufsc.br/pesquisas/projeto-veiculo-autonomo/datasets/?lang=en.

References

  • Abu Alhaija, H., Mustikovela, S. K., Mescheder, L., Geiger, A., & Rother, C. (2018). Augmented reality meets computer vision: Efficient data generation for urban driving scenes. International Journal of Computer Vision, 126(9), 961–972. https://doi.org/10.1007/s11263-018-1070-x.

    Article  Google Scholar 

  • Ardiyanto, I., & Adji, T. B. (2017). Deep residual coalesced convolutional network for efficient semantic road segmentation. IPSJ Transactions on Computer Vision and Applications, 9(1), 6. https://doi.org/10.1186/s41074-017-0020-9.

    Article  Google Scholar 

  • Banharnsakun, A. (2017). Hybrid abc-ann for pavement surface distress detection and classification. International Journal of Machine Learning and Cybernetics, 8(2), 699–710. https://doi.org/10.1007/s13042-015-0471-1.

    Article  Google Scholar 

  • Brostow, G. J., Fauqueur, J., & Cipolla, R. (2009). Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters, 30(2), 88–97. https://doi.org/10.1016/j.patrec.2008.04.005.

    Article  Google Scholar 

  • Cabral, F. S., Pinto, M., Mouzinho, FALN., Fukai, H., & Tamura, S. (2018) An automatic survey system for paved and unpaved road classification and road anomaly detection using smartphone sensor. In: 2018 IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI) (pp. 65–70). https://doi.org/10.1109/SOLI.2018.8476788.

  • CNT (2018) Pesquisa CNT de Rodovias 2018. Relatório Gerencial. Confederação Nacional do Transporte (CNT). Serviço Social do Transporte (SEST). Serviço Nacional de Aprendizagem do Transporte (SENAT). https://pesquisarodovias.cnt.org.br/Home.

  • Commission IE (2001) Safety of laser products. part 1: Equipment classification, requirements and user’s guide.

  • Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016). The cityscapes dataset for semantic urban scene understanding. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 3213–3223). https://doi.org/10.1109/CVPR.2016.350.

  • Cristóforis, P. D., Nitsche, M. A., Krajník, T., & Mejail, M. (2016). Real-time monocular image-based path detection. Journal of Real-Time Image Processing, 11(2), 335–348. https://doi.org/10.1007/s11554-013-0356-z.

    Article  Google Scholar 

  • Dairi, A., Harrou, F., Senouci, M., & Sun, Y. (2018). Unsupervised obstacle detection in driving environments using deep-learning-based stereovision. Robotics and Autonomous Systems, 100, 287–301. https://doi.org/10.1016/j.robot.2017.11.014.

    Article  Google Scholar 

  • Damian, C., Grigorescu, D., Ghindă, I., & Robu, M. (2019). Using mono and stereo camera system for static and moving objects detection. In: 2019 International Conference on Electromechanical and Energy Systems (SIELMEN), (pp. 1–5). https://doi.org/10.1109/SIELMEN.2019.8905820.

  • Eriksson, J., Girod, L., Hull, B., Newton, R., Madden, S., & Balakrishnan, H. (2008). The pothole patrol: Using a mobile sensor network for road surface monitoring. In: Proceedings of the 6th International Conference on Mobile Systems, Applications, and Services, ACM, New York, NY, USA, MobiSys ’08 (pp. 29–39). https://doi.org/10.1145/1378600.1378605.

  • Frisoni, R., Dionori, F., Casullo, L., Vollath, C., Devenish, L., Spano, F., Sawicki, T., Carl, S., Lidia, R., Neri, J., Silaghi, R., Stanghellini, A., & Gleave, SD. (2014). EU Road Surfaces: Economic and Safety Impact of the Lack of Regular Road Maintenance. European Parliament—Policy Department Structural and Cohesion Policies. http://www.europarl.europa.eu/thinktank/en/document.html?reference=IPOL_STU(2014)529059.

  • Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11), 1231–1237. https://doi.org/10.1177/0278364913491297.

    Article  Google Scholar 

  • Guo, C., Mita, S., & McAllester, D. (2011). Adaptive non-planar road detection and tracking in challenging environments using segmentation-based markov random field. In: 2011 IEEE International Conference on Robotics and Automation (pp. 1172–1179). https://doi.org/10.1109/ICRA.2011.5979693.

  • Guo, C., Mita, S., & McAllester, D. (2012). Robust road detection and tracking in challenging scenarios based on markov random fields with unsupervised learning. IEEE Transactions on Intelligent Transportation Systems, 13(3), 1338–1354. https://doi.org/10.1109/TITS.2012.2187896.

    Article  Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770–778). https://doi.org/10.1109/CVPR.2016.90.

  • Herghelegiu, P., Burlacu, A., Caraiman, S. (2017). Negative obstacle detection for wearable assistive devices for visually impaired. In: 2017 21st International Conference on System Theory, Control and Computing (ICSTCC), (pp. 564–570). https://doi.org/10.1109/ICSTCC.2017.8107095.

  • Howard, J. et al. (2018). fastai. https://github.com/fastai/fastai.

  • Huidrom, L., Das, L. K., & Sud, S. (2013). Method for automated assessment of potholes, cracks and patches from road surface video clips. Procedia—Social and Behavioral Sciences 104:312–321. In 2nd Conference of Transportation Research Group of India (2nd CTRG). https://doi.org/10.1016/j.sbspro.2013.11.124.

  • Jia, B., Chen, J., & Zhang, K. (2017). Recursive drivable road detection with shadows based on two-camera systems. Machine Vision and Applications, 28(5), 509–523. https://doi.org/10.1007/s00138-017-0858-y.

    Article  Google Scholar 

  • Kang, B., & Choi, S. (2017). Pothole detection system using 2d lidar and camera. In: 2017 Ninth International Conference on Ubiquitous and Future Networks (ICUFN) (pp. 744–746). https://doi.org/10.1109/ICUFN.2017.7993890.

  • Karunasekera, H., Zhang, H., Xi, T., & Wang, H. (2017). Stereo vision based negative obstacle detection. In: 2017 13th IEEE International Conference on Control Automation (ICCA) (pp. 834–838). https://doi.org/10.1109/ICCA.2017.8003168.

  • Koch, C., & Brilakis, I. (2011). Pothole detection in asphalt pavement images. Advanced Engineering Informatics 25(3):507–515, special Section: Engineering informatics in port operations and logistics. https://doi.org/10.1016/j.aei.2011.01.002.

  • Li, Y., Ding, W., Zhang, X., & Ju, Z. (2016). Road detection algorithm for autonomous navigation systems based on dark channel prior and vanishing point in complex road scenes. Robotics and Autonomous Systems, 85, 1–11. https://doi.org/10.1016/j.robot.2016.08.003.

    Article  Google Scholar 

  • Lian, J., Kong, L., Li, L., Zheng, W., Zhou, Y., Fang, S., & Qian, B. (2019). Study on obstacle detection and recognition method based on stereo vision and convolutional neural network. In: 2019 Chinese Control Conference (CCC) (pp. 8766–8771). https://doi.org/10.23919/ChiCC.2019.8866348.

  • Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., et al. (2014). Microsoft coco: Common objects in context. In D. Fleet, T. Pajdla, B. Schiele, & T. Tuytelaars (Eds.), Computer Vision: ECCV 2014 (pp. 740–755). Cham: Springer.

  • Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T., & Omata, H. (2018). Road damage detection and classification using deep neural networks with smartphone images. Computer-Aided Civil and Infrastructure Engineering, 33(12), 1127–1141. https://doi.org/10.1111/mice.12387.

    Article  Google Scholar 

  • Nguyen, L., Phung, S. L., & Bouzerdoum, A. (2017). Enhanced pixel-wise voting for image vanishing point detection in road scenes. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1852–1856). https://doi.org/10.1109/ICASSP.2017.7952477.

  • Ososinski, M., & Labrosse, F. (2012). Real-time autonomous colour-based following of Ill-defined roads (pp. 366–376). Berlin: Springer. https://doi.org/10.1007/978-3-642-32527-4_33.

    Book  Google Scholar 

  • Ososinski, M., & Labrosse, F. (2013). Automatic driving on ill-defined roads: An adaptive, shape-constrained, color-based method. Journal of Field Robotics,. https://doi.org/10.1002/rob.21494.

    Article  Google Scholar 

  • Ramos, S., Gehrig, S., Pinggera, P., Franke, U., & Rother, C. (2017). Detecting unexpected obstacles for self-driving cars: Fusing deep learning and geometric modeling. In: 2017 IEEE Intelligent Vehicles Symposium (IV) (pp. 1025–1032). https://doi.org/10.1109/IVS.2017.7995849.

  • Rankin, A., Ivanov, T., & Brennan, S. (2010). Evaluating the performance of unmanned ground vehicle water detection. In: Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop, ACM, New York, NY, USA, PerMIS ’10 (pp. 305–311), https://doi.org/10.1145/2377576.2377632.

  • Rankin, A. L., & Matthies, L. H. (2008). Daytime mud detection for unmanned ground vehicle autonomous navigation.

  • Rateke, T., & von Wangenheim, A. (2020). Passive vision road obstacle detection: A literature mapping. International Journal of Computers and Applications,. https://doi.org/10.1080/1206212X.2020.1758877.

    Article  Google Scholar 

  • Rateke, T., Justen, K. A., Chiarella, V. F., Sobieranski, A. C., Comunello, E., & Wangenheim, A. V. (2019a). Passive vision region-based road detection: A literature review. ACM Computing Survey, 52(2), 31:1–31:34. https://doi.org/10.1145/3311951.

    Article  Google Scholar 

  • Rateke, T., Justen, K. A., & von Wangenheim, A. (2019b). Road surface classification with images captured from low-cost cameras-road traversing knowledge (rtk) dataset. Revista de Informática Teórica e Aplicada (RITA).

  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In N. Navab, J. Hornegger, W. M. Wells, & A. F. Frangi (Eds.), Medical image computing and computer-assisted intervention—MICCAI 2015 (pp. 234–241). Cham: Springer.

    Chapter  Google Scholar 

  • Sengar, S. S., & Mukhopadhyay, S. (2017a). Motion detection using block based bi-directional optical flow method. Journal of Visual Communication and Image Representation, 49(C), 89–103.

    Article  Google Scholar 

  • Sengar, S. S., & Mukhopadhyay, S. (2017b). Moving object detection based on frame difference and w4. Signal, Image and Video Processing, 11(7), 1357–1364. https://doi.org/10.1007/s11760-017-1093-8.

    Article  Google Scholar 

  • Sengar, S. S., & Mukhopadhyay, S. (2020a). Motion segmentation-based surveillance video compression using adaptive particle swarm optimization. Neural Computing and Applications, 32(15), 11443–11457. https://doi.org/10.1007/s00521-019-04635-6.

    Article  Google Scholar 

  • Sengar, S. S., & Mukhopadhyay, S. (2020b). Moving object detection using statistical background subtraction in wavelet compressed domain. Multimedia Tools and Applications, 79(9), 5919–5940. https://doi.org/10.1007/s11042-019-08506-z.

    Article  Google Scholar 

  • Shi, J., Fu, F., Wang, Y., & Wang, J. (2016). A novel path segmentation method for autonomous road following. In: 2016 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), (pp. 1–6). https://doi.org/10.1109/ICSPCC.2016.7753701.

  • Shinzato, P. Y., dos Santos, T. C., Rosero, L. A., Ridel, D. A., Massera, C. M., Alencar, F., Batista, M. P., Hata, A. Y., Osório, F. S., & Wolf, D. F. (2016) Carina dataset: An emerging-country urban scenario benchmark for road detection systems. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), (pp. 41–46). https://doi.org/10.1109/ITSC.2016.7795529.

  • Smith, L. N. (2015). No more pesky learning rate guessing games. arXiv:1506.01186.

  • Smith, L. N. (2018). A disciplined approach to neural network hyper-parameters: Part 1—learning rate, batch size, momentum, and weight decay. arXiv:1803.09820

  • Sövény, B., Kovács, G., & Kardkovács, Z. T. (2015). Blind guide. Journal on Multimodal User Interfaces, 9(4), 287–297. https://doi.org/10.1007/s12193-015-0191-6.

    Article  Google Scholar 

  • STANDARD AN. (2005). American national standard for safe use of lasers outdoors. FL: Orlando.

    Google Scholar 

  • Sun, Y., Zhang, L., Leng, J., Luo, T., & Wu, Y. (2018). An obstacle detection method based on binocular stereovision. In B. Zeng, Q. Huang, A. El Saddik, H. Li, S. Jiang, & X. Fan (Eds.), Advances in multimedia information processing—PCM 2017 (pp. 571–580). Cham: Springer.

    Chapter  Google Scholar 

  • Tedeschi, A., & Benedetto, F. (2017). A real-time automatic pavement crack and pothole recognition system for mobile android-based devices. Advanced Engineering Informatics, 32, 11–25. https://doi.org/10.1016/j.aei.2016.12.004.

    Article  Google Scholar 

  • Tsai, Y., Chen, K., Chen, Y., & Cheng, J. (2018). Accurate and fast obstacle detection method for automotive applications based on stereo vision. In: 2018 International Symposium on VLSI Design, Automation and Test (VLSI-DAT), (pp. 1–4). https://doi.org/10.1109/VLSI-DAT.2018.8373249.

  • Valente, M., & Stanciulescu, B. (2017). Real-time method for general road segmentation. In: 2017 IEEE Intelligent Vehicles Symposium (IV), (pp. 443–447). https://doi.org/10.1109/IVS.2017.7995758.

  • Wang, H., Ren, M., & Yang, J. (2016). Capitalizing on the boundary ratio prior for road detection. Multimedia Tools and Applications, 75(19), 11999–12019. https://doi.org/10.1007/s11042-016-3280-y.

    Article  Google Scholar 

  • Wang, J., Sun, S., & Zhao, X. (2015). Unstructured road detection and path tracking for tracked mobile robot. In: 2015 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER), (pp. 535–539). https://doi.org/10.1109/CYBER.2015.7287996.

  • Xiao, L., Dai, B., Hu, T., & Wu, T. (2015). Fast unstructured road detection and tracking from monocular video. In: The 27th Chinese Control and Decision Conference (2015 CCDC), (pp. 3974–3980). https://doi.org/10.1109/CCDC.2015.7162618.

  • Yu, X., & Salari, E. (2011). Pavement pothole detection and severity measurement using laser imaging. In: 2011 IEEE INTERNATIONAL CONFERENCE ON ELECTRO/INFORMATION TECHNOLOGY, (pp. 1–5). https://doi.org/10.1109/EIT.2011.5978573.

  • Yuan, Y., Jiang, Z., & Wang, Q. (2015). Video-based road detection via online structural learning. Neurocomputing, 168, 336–347. https://doi.org/10.1016/j.neucom.2015.05.092.

    Article  Google Scholar 

  • Zanin, M., Messelodi, S., Modena, C. M., & Kessler, F. B. (2013). diplodoc road stereo sequence. https://tev.fbk.eu/databases/diplodoc-road-stereo-sequence

  • Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017). Scene parsing through ade20k dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 5122–5130).

  • Zu, Z., Hou, Y., Cui, D., & Xue, J. (2015). Real-time road detection with image texture analysis-based vanishing point estimation. In: 2015 IEEE International Conference on Progress in Informatics and Computing (PIC), (pp. 454–457). https://doi.org/10.1109/PIC.2015.7489888.

Download references

Author information

Authors and Affiliations

  1. Department of Informatics and Statistics, Graduate Program in Computer Science, Federal University of Santa Catarina (PPGCC), Florianópolis, SC, Brazil

    Thiago Rateke & Aldo von Wangenheim

  2. Image Processing and Computer Graphics Lab (LAPIX) at National Institute for Digital Convergence (INCoD), Florianópolis, SC, Brazil

    Thiago Rateke & Aldo von Wangenheim

Authors
  1. Thiago Rateke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aldo von Wangenheim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thiago Rateke.

Ethics declarations

Conflict of interest

The authors declare that there are no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001. CAPES (Brazilian Federal Agency for Support and Evaluation of Graduate Education). It was also supported by the Brazilian National Institute for Digital Convergence (INCoD), a research unit of the Brazilian National Institutes for Science and Technology Program (INCT) of the Brazilian National Council for Science and Technology (CNPq)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rateke, T., von Wangenheim, A. Road surface detection and differentiation considering surface damages. Auton Robot 45, 299–312 (2021). https://doi.org/10.1007/s10514-020-09964-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-020-09964-3

Keywords

Search

Navigation