Skip to main content

Advertisement

SpringerLink
Log in

Pothole detection using location-aware convolutional neural networks

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Poor road conditions, such as potholes, are a nuisance to society, which would annoy passengers, damage vehicles, and even cause accidents. Thus, detecting potholes is an important step toward pavement maintenance and rehabilitation to improve road conditions. Potholes have different shapes, scales, shadows, and illumination effects, and highly complicated backgrounds can be involved. Therefore, detection of potholes in road images is still a challenging task. In this study, we focus on pothole detection in 2D vision and present a new method to detect potholes based on location-aware convolutional neural networks, which focuses on the discriminative regions in the road instead of the global context. It consists of two main subnetworks: the first localization subnetwork employs a high recall network model to find as many candidate regions as possible, and the second part-based subnetwork performs classification on the candidates on which the network is expected to focus. The experiments using the public pothole dataset show that the proposed method could achieve high precision (95.2%), recall (92.0%) simultaneously, and outperform the most existing methods. The results also demonstrate that accurate part localization considerably increases classification performance while maintains high computational efficiency. The source code is available at https://github.com/hanshenchen/pothole-detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. National Safety Council (2018) Safety on the road. https://www.nsc.org/road-safey. Accessed 7 Jan 2019

  2. European Commission (2018) Road safety in the European Union trends, statistics and main challenges. https://ec.europa.eu/transport/road_safety/sites/roadsafety/files/vademecum_2018.pdf. Accessed 7 Jan 2019

  3. European Commission (2017) Public consultation on road infrastructure and tunnel safety. https://ec.europa.eu/transport/modes/road/consultations/2017-road-infrastructure-safety_en. Accessed 7 Jan 2019

  4. Federal Highway Administration (2018) Asset management overview. https://www.fhwa.dot.gov/asset/if08008/amo_06.cfm. Accessed 7 Jan 2019

  5. Kim T, Ryu S (2014) Review and analysis of pothole detection methods. J Emerg Trends Comput Inf Sci 5(8):603–608

    Google Scholar 

  6. Fan R, Ai X, Dahnoun N (2018) Road surface 3d reconstruction based on dense subpixel disparity map estimation. IEEE Trans Image Process 27(6):3025–3035. https://doi.org/10.1109/TIP.2018.2808770

    Article  MathSciNet  MATH  Google Scholar 

  7. Tsai YC, Chatterjee A (2017) Pothole detection and classification using 3D technology and watershed method. J Comput Civ Eng 32(2):04017078. https://doi.org/10.1061/(asce)cp.1943-5487.0000726

    Article  Google Scholar 

  8. Zhang A, Wang KCP, Ai C (2017) 3D shadow modeling for detection of descended patterns on 3D pavement surface. J Comput Civ Eng 31(4):04017019. https://doi.org/10.1061/(asce)cp.1943-5487.0000661

    Article  Google Scholar 

  9. Li S, Yuan C, Liu D, Cai H (2016) Integrated processing of image and GPR data for automated pothole detection. J Comput Civ Eng 30(6):04016015. https://doi.org/10.1061/(asce)cp.1943-5487.0000582

    Article  Google Scholar 

  10. Eriksson J, Girod L, Hull B et al (2008) The pothole patrol: using a mobile sensor network for road surface monitoring. In: Proceedings of 6th international conference on. mobile systems, applications, and services, pp 29–39. https://doi.org/10.1145/1378600.1378605

  11. Wang HW, Chen CH, Cheng DY et al (2015) A real-time pothole detection approach for intelligent transportation system. Math Probl Eng. https://doi.org/10.1155/2015/869627

    Article  Google Scholar 

  12. Xue G, Zhu H, Hu Z et al (2017) Pothole in the dark: perceiving pothole profiles with participatory urban vehicles. IEEE Trans Mobile Comput 16(5):1408–1419. https://doi.org/10.1109/tmc.2016.2597839

    Article  Google Scholar 

  13. Buza E, Omanovic S, Huseinovic A (2013) Pothole detection with image processing and spectral clustering. In: Proceedings of 2nd international conference on information technology and computer networks, pp 48–53

  14. Koch C, Brilakis I (2011) Pothole detection in asphalt pavement images. Adv Eng Inf 25(3):507–515. https://doi.org/10.1016/j.aei.2011.01.002

    Article  Google Scholar 

  15. Pereira V, Tamura S, Hayamizu S, Fukai H (2018) A deep learning-based approach for road pothole detection in timor leste. In: Proc 2018 IEEE international conference on service operations and logistics, and informatics, pp 279–284. https://doi.org/10.1109/soli.2018.8476795

  16. Anand S, Gupta S, Darbari V, Kohli S (2018) Crack-pot: autonomous road crack and pothole detection. In: Proceedings of 2017 international conference on digital image computing: techniques and applications, pp 1–6. https://doi.org/10.1109/DICTA.2018.8615819

  17. Zhang L, Yang F, Zhang YD, Zhu YJ (2016) Road crack detection using deep convolutional neural network. In: Proceedings of IEEE international conference on image processing, pp 3708–3712. https://doi.org/10.1109/icip.2016.7533052

  18. Eisenbach M, Stricker R, Seichter D et al (2017) How to get pavement distress detection ready for deep learning? A systematic approach. In: Proceedings of 2017 international joint conference on neural networks, pp 2039–2047. https://doi.org/10.1109/ijcnn.2017.7966101

  19. Fu J, Zheng H, Mei T (2017) Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 4476–4484. https://doi.org/10.1109/cvpr.2017.476

  20. Liu H, Feng J, Qi M et al (2017) End-to-end comparative attention networks for person re-identification. IEEE Trans Image Process 26(7):3492–3506. https://doi.org/10.1109/tip.2017.2700762

    Article  MathSciNet  MATH  Google Scholar 

  21. Zhang N, Donahue J, Girshick R, Darrell T (2014) Part-based R-CNNs for fine-grained category detection. In: Proceedings of European conference on computer vision. Springer, Cham, pp 834–849. https://doi.org/10.1007/978-3-319-10590-1_54

    Chapter  Google Scholar 

  22. Huang S, Xu Z, Tao D, Zhang Y (2016) Part-stacked CNN for fine-grained visual categorization. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 1173–1182. https://doi.org/10.1109/cvpr.2016.132

  23. Yang Z, Luo T, Wang D, Hu Z, Gao J, Wang L (2018) Learning to navigate for fine-grained classification. In: Proceeding of European conference on computer vision. Springer, Cham, pp 420–435. https://doi.org/10.1007/978-3-030-01264-9_26

    Chapter  Google Scholar 

  24. Machine Intelligence Institute of Africa (2017) MIIA deep learning Hackathon. http://machineintelligenceafrica.org/activities/hackathon/. Accessed 7 Jan 2019

  25. Tedeschi A, Benedetto F (2017) A real-time automatic pavement crack and pothole recognition system for mobile Android-based devices. Adv Eng Inf 32:11–25. https://doi.org/10.1061/(asce)cp.1943-5487.0000726

    Article  Google Scholar 

  26. Radopoulou SC, Brilakis I (2016) Automated detection of multiple pavement defects. J Comput Civ Eng 31(2):04016057. https://doi.org/10.1061/(asce)cp.1943-5487.0000623

    Article  Google Scholar 

  27. Nienaber S, Booysen MJ, Kroon RS (2015) Detecting potholes using simple image processing techniques and real-world footage. In: Proceedings of 34th South African transport conference, Southern African transport conference

  28. Akagic A, Buza E, Omanovic S (2017) Pothole detection: an efficient vision based method using RGB color space image segmentation. In: Proceedings of 40th international convention on information and communication technology, electronics and microelectronics, pp 1104–1109. https://doi.org/10.23919/MIPRO.2017.7973589

  29. Koch C, Jog GM, Brilakis I (2012) Automated pothole distress assessment using asphalt pavement video data. J Comput Civ Eng 27(4):370–378. https://doi.org/10.1061/(asce)cp.1943-5487.0000232

    Article  Google Scholar 

  30. Feng C, Liu MY, Kao CC, Lee TY (2017) Deep active learning for civil infrastructure defect detection and classification. In: Proceedings of ASCE international workshop on computing in civil engineering, pp 298–306. https://doi.org/10.1061/9780784480823.036

  31. Maeda H, Sekimoto Y, Seto T, Kashiyama T, Omata H (2018) Road damage detection using deep neural networks with images captured through a smartphone. Comput Aided Civ Infrastruct Eng 33:1127–1141. https://doi.org/10.1111/mice.12387

    Article  Google Scholar 

  32. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 3431–3440. https://doi.org/10.1109/cvpr.2015.7298965. 

  33. Xiao B, Wu H, Wei Y (2018) Simple baselines for human pose estimation and tracking. In: Proceedings of European conference on computer vision, pp 472–487. https://doi.org/10.1007/978-3-030-01231-1_29

    Chapter  Google Scholar 

  34. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/cvpr.2016.90

  35. Ioffe S, Szegedy C (2015). Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of 32td international conference on machine learning, international machine learning society (IMLS), vol 37, pp 448–456

  36. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Proceedings of 26th Conference on neural information processing systems, neural information processing systems foundation, pp 1097–1105

  37. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 815–823. https://doi.org/10.1109/cvpr.2015.7298682

  38. Scherer D, Muller A, Behnke S (2010) Evaluation of pooling operations in convolutional architectures for object recognition. In: Proceedings of international conference on artificial neural networks, pp 92–101. https://doi.org/10.1007/978-3-642-15825-4_10

    Chapter  Google Scholar 

  39. Nienaber S, Kroon RS, Booysen MJ (2015) A comparison of low-cost monocular vision techniques for pothole distance estimation. In: Proceedings of 2015 IEEE symposium series on computational intelligence, pp 419–426. https://doi.org/10.1109/ssci.2015.69

  40. Chen HS (2019) pothole-detection. https://github.com/hanshenChen/pothole-detection/tree/master/preprocessing. Accessed 7 Jan 2019

  41. Yang Y, Ramanan D (2013) Articulated human detection with flexible mixtures of parts. IEEE Trans Pattern Anal Mach Intell 35(12):2878–2890. https://doi.org/10.1109/tpami.2012.261

    Article  Google Scholar 

  42. Chollet F et al (2015) Keras. https://github.com/fchollet/keras. Accessed 7 Jan 2019

  43. Abadi M, Agarwal A, Barham P et al (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. Preprint, submitted Mar 14, 2016. Software available from tensorflow.org. http://arxiv.org/abs/1603.04467

  44. Russakovsky O, Deng J, Su H et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vision 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  45. Kingma D, Ba J (2015) Adam: A method for stochastic optimization. In: Proceedings of international conference on learning representations

  46. Marais J et al (2017) PotholeDetection. https://github.com/jandremarais/PotholeDetection. Accessed 7 Jan 2019

  47. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 4700–4708. https://doi.org/10.1109/CVPR.2017.243

Download references

Acknowledgements

This work was supported by the National Science Foundation of China (No. 61871350) and the Department of Communication of Zhejiang Province, China (No. 2017JY04).

Author information

Authors and Affiliations

  1. College of Information Engineering, Zhejiang University of Technology, Hangzhou, 310023, China

    Hanshen Chen, Minghai Yao & Qinlong Gu

  2. Zhejiang Institute of Communications, Hangzhou, 311112, China

    Hanshen Chen

Authors
  1. Hanshen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minghai Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinlong Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Minghai Yao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, H., Yao, M. & Gu, Q. Pothole detection using location-aware convolutional neural networks. Int. J. Mach. Learn. & Cyber. 11, 899–911 (2020). https://doi.org/10.1007/s13042-020-01078-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-020-01078-7

Keywords

Search

Navigation