Skip to main content
SpringerLink
Log in

ADBench: benchmarking autonomous driving systems

  • Special Issue Article
  • Published:
Computing Aims and scope Submit manuscript

Abstract

Driven by the improvements in a variety of domains, autonomous driving is becoming a reality and today, industry aims at moving toward fully autonomous vehicles. High-tech chip manufacturers are designing high-performance and energy-efficient platforms in accordance with safety standard requirements. However, the software used to implement advanced functionalities in autonomous vehicles challenges real-time constraints on those platforms. Hence, there is a clear need for industry-level autonomous driving benchmarks to evaluate platforms and systems. In this paper, we propose ADBench, a benchmarking approach and benchmark suite for state-of-the-art autonomous driving platforms, in accordance with the key modules, structural design and functions of AD systems, building on several industry-level autonomous driving systems. The use of standard benchmarks facilitates the design, verification and validation process of autonomous systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. A bag is a file format in ROS for storing ROS message data. They are typically created by a tool like rosbag, which subscribes to one or more ROS topics, and stores the serialized message data in a file as it is received. These bag files can also be played back in ROS to the same topics they were recorded from, or even remapped to new topics.

References

  1. Alcon M, Tabani H, Kosmidis L, Mezzetti E, Abella J, Cazorla FJ (2020) Timing of autonomous driving software: problem analysis and prospects for future solutions. IEEE Real-Time Embed Technol Appl Symp (RTAS) IEEE 2020:267–280

    Google Scholar 

  2. Alcon M, Tabani H, Abella J, Kosmidis L, Cazorla FJ (2020) En-route: on enabling resource usage testing for autonomous driving frameworks. In: Proceedings of the 35th annual ACM symposium on applied computing, pp 1953–1962

  3. Alcaide S, Kosmidis L, Tabani H, Hernandez C, Abella J, Cazorla FJ (2018) Safety-related challenges and opportunities for GPUS in the automotive domain. IEEE Micro 38(6):46–55

    Article  Google Scholar 

  4. ApolloAuto, Apollo 3.0 software architecture (2018). https://github.com/ApolloAuto/apollo/blob/master/docs/specs/Apollo_3.0_Software_Architecture.md

  5. ApolloAuto, Perception (2018). https://github.com/ApolloAuto/apollo/blob/r3.0.0/docs/specs/perception_apollo_3.0.md

  6. ApolloAuto (2018) 3D obstacle perception. https://github.com/ApolloAuto/apollo/blob/r3.0.0/docs/specs/3d_obstacle_perception.md

  7. ApolloAuto (2018) Traffic light perception. https://github.com/ApolloAuto/apollo/blob/master/docs/specs/traffic_light.md

  8. ApolloAuto (2018) Multi-sensor fusion localization. https://github.com/ApolloAuto/apollo/blob/r3.0.0/modules/localization/msf/README.md

  9. ApolloAuto (2018) Planning. https://github.com/ApolloAuto/apollo/tree/r3.5.0/modules/planning/README.md

  10. ApolloAuto (2018) Prediction. https://github.com/ApolloAuto/apollo/blob/r3.0.0/modules/prediction/README.md

  11. Clint Whaley R, Aberdeen D, Brett M, Coult N, Castaldo T, Dittrich M, Gaudet D, Goto K, Horner J, Maguire C, Mattox T, Deitz H, Nguyen V, Strazdins P, Ruhe J, Soendergaard P, Staelin C (2018) Automatically Tuned Linear Algebra Software (ATLAS). http://math-atlas.sourceforge.net/

  12. Xianyi Z, Qian W, Saar W, Chothia Z, Shaohu C, Wen L et al (2020) An optimized BLAS library (OpenBLAS). http://www.openblas.net/

  13. Baidu Apollo, an open autonomous driving platform (2018). http://apollo.auto/

  14. Blackford L et al (2002) An updated set of basic linear algebra subprograms (BLAS). ACM Trans Math Softw 28(2):135–151

    Article  MathSciNet  Google Scholar 

  15. Chetlur S et al., CUDNN: Efficient primitives for deep learning. ArXiv preprint arXiv:1410.0759

  16. Caesar H, Bankiti V, Lang AH, Vora S, Liong VE, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O. Nuscenes: a multimodal dataset for autonomous driving. ArXiv preprint arXiv:1903.11027

  17. Corp T (2018) Tesla autopilot. https://www.tesla.com/autopilot

  18. NVIDIA (2021) cuBLAS. http://docs.nvidia.com/cuda/cublas/

  19. Demler M (2017) Xavier simplifies self-driving cars. In: Microprocessors Report, The Linly Group, June

  20. Dolgov D, Thrun S, Montemerlo M, Diebel J (2010) Path planning for autonomous vehicles in unknown semi-structured environments. Int J Robot Res 29(5):485–501

    Article  Google Scholar 

  21. EEMBC (2019) Introducing the EEMBC MLMark Benchmark. https://www.eembc.org/mlmark/

  22. EEMBC (2019) The \({\rm ADASMark}^{\rm TM}\) Benchmark: A Performance Measurement and Optimization Tool for Automotive Companies Building Next-Generation Advanced Driver-Assistance Systems (ADAS). https://www.eembc.org/adasmark/

  23. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? The kitti vision benchmark suite. In: Conference on computer vision and pattern recognition (CVPR)

  24. Intel (2020) Intel oneAPI Math Kernel Library: The fastest and most-used math library for Intel-based systems. https://software.intel.com/en-us/intel-mkl

  25. Koopman P, Wagner M (2016) Challenges in autonomous vehicle testing and validation. SAE Int J Trans Saf 4:15–24. https://doi.org/10.4271/2016-01-0128

    Article  Google Scholar 

  26. Merkel D (2014) Docker: lightweight Linux containers for consistent development and deployment. Linux J 2014(239):2

    Google Scholar 

  27. Paz D, Lai P-J, Chan N, Jiang Y, Christensen HI, Autonomous vehicle benchmarking using unbiased metrics. ArXiv preprint arXiv:2006.02518

  28. Pujol R, Tabani H, Kosmidis L, Mezzetti E, Abella J, Cazorla FJ (2019) Generating and exploiting deep learning variants to increase heterogeneous resource utilization in the Nvidia Xavier. In: 31st euromicro conference on real-time systems (ECRTS 2019), Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik

  29. Quigley M et al (2009) ROS: an open-source robot operating system. ICRA workshop on open source software, vol 3. Japan, Kobe, p 5

  30. Redmon J, Farhadi A (2018) YOLOv3: An Incremental Improvement. CoRR. arXiv:1804.02767

  31. Reddi VJ, Cheng C, Kanter D, Mattson P, Schmuelling G, Wu C-J, Anderson B, Breughe M, Charlebois M, Chou W, et al (2020) Mlperf inference benchmark. In: 2020 ACM/IEEE 47th annual international symposium on computer architecture (ISCA). IEEE, pp 446–459

  32. Rong G, Shin BH, Tabatabaee H, Lu Q, Lemke S, Možeiko M, Boise E, Uhm G, Gerow M, Mehta S, et al. Lgsvl simulator: a high fidelity simulator for autonomous driving. ArXiv preprint arXiv:2005.03778

  33. Tabani H, Kosmidis L, Abella J, Cazorla FJ, Bernat G (2019) Assessing the adherence of an industrial autonomous driving framework to iso 26262 software guidelines. In: Proceedings of the 56th annual design automation conference 2019. ACM, p 9

  34. Tabani H, Mazzocchetti F, Benedicte P, Abella J, Cazorla FJ (2021) Performance analysis and optimization opportunities for Nvidia automotive GPUS. J Parallel Distrib Comput 152:21–32

    Article  Google Scholar 

  35. Tabani H, Pujol R, Abella J, Cazorla FJ (2020) A cross-layer review of deep learning frameworks to ease their optimization and reuse. In: IEEE 23rd international symposium on real-time distributed computing (ISORC). IEEE, pp 144–145

  36. NVIDIA (2021) TensorRT: A platform for high-performance deep learning inference. https://developer.nvidia.com/tensorrt

  37. The autoware foundation, autoware: an open autonomous driving platform https://github.com/CPFL/Autoware/ (2016)

  38. NVIDIA (2018) Self-driving Safety Report. https://www.nvidia.com/en-us/self-driving-cars/safety-report/

  39. Wan G, Yang X, Cai R, Li H, Zhou Y, Wang H, Song S (2018) Robust and precise vehicle localization based on multi-sensor fusion in diverse city scenes. In: IEEE international conference on robotics and automation (ICRA), pp 4670–4677

Download references

Acknowledgements

This work has been partially supported by the Spanish Ministry of Economy and Competitiveness (MINECO) under Grant TIN2015-65316-P, the SuPerCom European Research Council (ERC) project under the European Union’s Horizon 2020 research and innovation programme (Grant Agreement No. 772773), and the HiPEAC Network of Excellence.

Author information

Authors and Affiliations

  1. Barcelona Supercomputing Center (BSC), Plaça Eusebi Güell, 1-3, 08034, Barcelona, Spain

    Hamid Tabani, Joan Moya, Jaume Abella & Francisco J. Cazorla

  2. Universitat Politécnica de Catalunya, Barcelona Supercomputing Center (BSC), Plaça Eusebi Güell, 1-3, 08034, Barcelona, Spain

    Roger Pujol & Miguel Alcon

Authors
  1. Hamid Tabani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roger Pujol
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Miguel Alcon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joan Moya
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jaume Abella
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Francisco J. Cazorla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hamid Tabani.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tabani, H., Pujol, R., Alcon, M. et al. ADBench: benchmarking autonomous driving systems. Computing 104, 481–502 (2022). https://doi.org/10.1007/s00607-021-00975-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-021-00975-1

Keywords

Mathematics Subject Classification

Search

Navigation