Skip to main content
SpringerLink
Log in

Variant center-symmetric census transform for real-time stereo vision architecture on chip

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Stereo vision is one of the most crucial operations in many computer vision applications, and stereo matching is its most important step. In recent years, stereo matching has developed in the direction of increasing accuracy and higher computational speed, and the real-time stereo matching algorithm based on hardware architecture has been increasingly emphasized due to its applicability in embedded systems. The most frequently used method on FPGA is the census transform (CT) because of its simple structure and easy parallelization and the high quality of the generated disparity maps, but CT has the drawbacks of mismatches in some regions and dependence on a central pixel. In this paper, an improved CT-based semi-global stereo matching algorithm with pipeline and parallel operation based on FPGA is proposed in order to increase matching accuracy in specific regions while satisfying real-time constraints. In the matching cost step, pixels are divided into two parts, with the two methods calculated to generate the bit-vector feature. A four-path semi-global matching algorithm with twice aggregate is proposed in the cost aggregation stage. The left–right check, mismatching point filling, and the median filter to enhance the final disparity map are also required. The novel algorithm is evaluated on the Middleburry benchmark and implemented on Xilinx ZYNQ-7000 SoCs, which results in a throughput of 640 × 480/60 fps, with 64 disparity levels at 100 MHz. Compared with the related work, we improve the average accuracy by 1.61%, making our approach suitable for real-time embedded systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Kriegman, D.J., Triendl, E., Binford, T.O.: Stereo vision and navigation in buildings for mobile robots. IEEE Trans. Robot. Autom. 5(6), 792–803 (1989)

    Article  Google Scholar 

  2. Do, P.N.B., Pguyen, Q.C.: A review of stereo-photogrammetry method for 3-D reconstruction in computer vision. In: International Symposium on Communications and Information Technologies (ISCIT), Ho Chi Minh City, Vietnam, pp. 138–143 (2019)

  3. Sui, J., Jin, W.Q.: Therealization and development of binocular stereo vision technology. Appl. Electron. Tech. 30(10), 4–6 (2004)

    Google Scholar 

  4. Hirschmuller, H.: Accurate and efficient stereo processing by semi-global matching and mutual information. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, pp. 807–814 (2005)

  5. Zhang, X., Dai, H., Sun, H., et al.: Algorithm and VLSI architecture co-design on efficient semi-global stereo matching. IEEE Trans. Circuits Syst. Video Technol. 30(99), 4390–4403 (2019)

    Google Scholar 

  6. Wang, E., Zhu, Y., Peng, L., et al.: Stereo matching algorithm based on the combination of matching costs. In: Proceedings of the7th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent Systems, Honolulu, pp. 1001–1004 (2017)

  7. Chai, Y., Yang, F.: Semi-global stereo matching algorithm based on minimum spanning tree. In: Proceedings of the 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference, Xi’an, pp. 2181–2185 (2018)

  8. Oussama, Z., Mohammed, R., Aouatif, A., et al.: A hierarchical stereo matching algorithm based on adaptive support region aggregation method. Pattern Recognit. Lett. 112(9), 205–211 (2018)

    Google Scholar 

  9. Hong, G.S., Kim, B.G.: A local stereo matching algorithm based on weighted guided image filtering for improving the generation of depth range images. Displays Technol. Appl. 49, 80–87 (2017)

    Article  Google Scholar 

  10. Kitagawa, M., Shimizu, I.: High accuracy local stereo matching using DoG scale map. In: Proceedings of the 15th IAPR International Conference on Machine Vision Applications, pp. 258–261 (2017)

  11. Han, P., Zhao, M., Chen, S.: Fusion of texture, color and gradient information for stereo matching cost computation. In: Proceedings of the 2nd International Conference on Image, Vision and Computing (ICIVC), Chengdu, pp. 118–121 (2017)

  12. Hamzah, R.A., Kadmin, A.F., Hamid, M.S., et al.: Improvement of stereo matching algorithm for 3D surface reconstruction. Signal Process. Image Commun. 65, 165–172 (2018)

    Article  Google Scholar 

  13. Ma, N., Men, Y., Men, C., et al.: Segmentation-based stereo matching using combinatorial similarity measurement and adaptive support region. Int. J. Light Electron Opt. 137, 124–134 (2017)

    Article  Google Scholar 

  14. Shahbazi, M., Sohn, G., Théau, J.: High-density stereo image matching using intrinsic curves. ISPRS J. Photogramm. Remote. Sens. 146, 373–388 (2018)

    Article  Google Scholar 

  15. Cheng, F., Zhang, H., Yuan, D., et al.: Stereo matching by using the global edge constraint. Neurocomputing 131, 217–226 (2014)

    Article  Google Scholar 

  16. Tan, X., Sun, C., Pham, T.D.: Stereo matching based on multi-direction polynomial model. Signal Process. Image Commun. 44, 44–56 (2016)

    Article  Google Scholar 

  17. San, T., War, N.: Stereo matching algorithm by hill-climbing segmentation. In: Proceedings of the 6th IEEE Global Conference on Consumer Electronics, Nagoya, Oct, pp. 24–27 (2017)

  18. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision 47(1–3), 7–42 (2002)

    Article  Google Scholar 

  19. Banz, C., Hesselbarth, S., Flatt, H., et al.: Real-time stereo vision system using semi-global matching disparity estimation: architecture and FPGA-implementation. In: International Conference on Embedded Computer Systems, Samos, Greece, pp. 93–101 (2010)

  20. Hirschmuller, H.: Stereo processing by semi-global matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008)

    Article  Google Scholar 

  21. Rahnama, O., Cavallari, T., Golodetz, S., et al.: R3SGM: real-time raster-respecting semi-global matching for power-constrained systems. In: International Conference on Field Programmable Technology (FPT), Naha, Okinawa, Japan, pp. 105–112 (2018)

  22. Perri, S., Corsonello, P., Cocorullo, G.: Adaptive census transform: a novel hardware-oriented stereo vision algorithm. Comput. Vis. Image Underst. 117(1), 29–41 (2013)

    Article  Google Scholar 

  23. Hernandez-Juarez, D., Chacón, A., Espinosa, A., et al.: Embedded real-time stereo estimation via semi-global matching on the GPU. Proc. Comput. Sci. 80(C), 143–153 (2016)

    Article  Google Scholar 

  24. Mei, X., Sun, X., Zhou, M., et al.: On building an accurate stereo matching system on graphics hardware. In: IEEE International Conference on Computer Vision Workshops. IEEE (2012)

  25. Ojala, T., Pietikainen, M., Maenpaa, T.: Multi-resolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)

    Article  Google Scholar 

  26. Spangenberg, R., Langner, T., Raúl, R.: Weighted semi-global matching and center-symmetric census transform for robust driver assistance. 34–41 (2013)

  27. Vivado design suite user guide-high-level synthesis, UG902 (v2017.1), April 2017. https://www.xilinx.com/support/Documentation/sw_manuals/xilinx2017_1/ug-902-vivado-high-level-synthesis.pdf. Accessed 30 Apr 2018

  28. Middlebury Stereo Vision page. http://vision.middlebury.edu/stereo

  29. Li, Y., Li, Z., Yang, C., et al.: High throughput hardware architecture for accurate semi-global matching. Integr. VLSI J. 65, 417–427 (2019)

    Article  Google Scholar 

  30. Perez-Patricio, M., Aguilar-Gonzalez, A.: FPGA implementation of an effificient similarity-based adaptive window algorithm for real-time stereo matching. J. Real-Time Image Process. 16(2), 271–287 (2019)

    Article  Google Scholar 

Download references

Funding

This work was supported by [The National Natural Science Foundation of China] and [Major Projects of China's Manned Space Engineering] (Grant numbers [no. 61125101] and [no. RWZY640601]).

Author information

Authors and Affiliations

  1. Lanzhou Institute of Physics, CAST, Lanzhou, 730000, China

    Chenyuan Zhao, Wenxin Li & Qingxi Zhang

Authors
  1. Chenyuan Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenxin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qingxi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenxin Li.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Ethical approval

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript. The authors have no financial or proprietary interests in any material discussed in this article. Authors are responsible for correctness of the statements provided in the manuscript. See also Authorship Principles. The Editor-in-Chief reserves the right to reject submissions that do not meet the guidelines described in this section.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, C., Li, W. & Zhang, Q. Variant center-symmetric census transform for real-time stereo vision architecture on chip. J Real-Time Image Proc 18, 2073–2083 (2021). https://doi.org/10.1007/s11554-021-01087-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01087-x

Keywords

Search

Navigation