Skip to main content

Advertisement

SpringerLink
Log in

Robotic target following with slow and delayed visual feedback

  • Regular Paper
  • Published:
International Journal of Intelligent Robotics and Applications Aims and scope Submit manuscript

Abstract

Following rapidly and precisely a moving target has become the core functionality in robotic systems for transportation, manufacturing, and medical devices. Among existing targeting following methods, vision-based tracking continues to thrive as one of the most popular, and is the closest method to human perception. However, the low sampling rate and the time delays of visual outputs fundamentally hinder real-time applications. In this paper, we show the potential of significant performance gain in vision-based target following when partial knowledge of the target dynamics is available. Specifically, we propose a new framework with Kalman filters and multi-rate model-based prediction (1) to reconstruct fast-sampled 3D target position and velocity data, and (2) to compensate the time delay. Along the path, we study the impact of slow sampling and the delay duration, and we experimentally verify different algorithms with a robot manipulator equipped with an eye-in-hand camera. The results show that our approach can achieve 95% error reduction rate compared with the commonly used visual servoing technique, when the target is moving with high speed and the visual measurements are slow and incapable of providing timely information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Abbreviations

\(\{A\}\) :

A 3D coordinate system with origin at porint A.

\({}^{A}{}{\xi _{B}}\) :

The 3D pose of frame \(\{B\}\) with respect to frame \(\{A\}\).

\({}^{A}{}{\varvec{v}_{B}}\) :

The 3D velocity of frame \(\{B\}\) with respect to frame \(\{A\}\).

\(\oplus\) :

The composition operator of relative poses, e.g., \({}^{A}{\xi _{C}}={}^{A}{}{\xi _{B}}\oplus ^{B}{}{\xi _C}\).

\({}^{A}{\varvec{R}_B}\) :

The rotation matrix corresponding to the relative pose \({}^{A}{}{\xi _B}\).

\({}^{A}{}{\varvec{t}_B}\) :

The translation vector corresponding to the relative pose \({}^{A}{}{\xi _B}\).

\(\varvec{R}_{x}(\theta )\) :

The rotation matrix that corresponds to the 3D rotating of \(\theta\) degrees about x axis.

\(A'\) :

The transpose of matrix A.

References

  • Aldoma, A., Marton, Z.C., Tombari, F., Wohlkinger, W., Potthast, C., Zeisl, B., Rusu, R.B., Gedikli, S., Vincze, M.: Tutorial: Point cloud library: Three-dimensional object recognition and 6 dof pose estimation. IEEE Robot. Autom. Mag. 19(3), 80–91 (2012)

    Article  Google Scholar 

  • Barrois, B., Wöhler, C.: 3d pose estimation of vehicles using stereo camera. Encyclopedia of Sustainability Science and Technology pp. 10589–10612 (2012)

  • Bar-Shalom, Y., Li, X.R., Kirubarajan, T.: Estimation with applications to tracking and navigation: theory algorithms and software. Wiley, Hoboken (2004)

    Google Scholar 

  • Bateux, Q., Marchand, E., Leitner, J., Chaumette, F., Corke, P.: Training deep neural networks for visual servoing. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3307–3314 (2018)

  • Bensalah, F., Chaumette, F.: Compensation of abrupt motion changes in target tracking by visual servoing. In: Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots, vol. 1, pp. 181–187. IEEE (1995)

  • Castaño, A., Hutchinson, S.: Visual compliance: Task-directed visual servo control. IEEE Trans. Robot. Autom. 10(3), 334–342 (1994)

    Article  Google Scholar 

  • Chalimbaud, P., Berry, F.: Embedded active vision system based on an fpga architecture. EURASIP J. Embed. Syst. 2007(1), 26–26 (2007)

    Article  Google Scholar 

  • Chaumette, F., Hutchinson, S.: Visual servo control. I. Basic approaches. IEEE Robot. Autom. Magaz. 13(4), 82–90 (2006)

    Article  Google Scholar 

  • Chaumette, F., Hutchinson, S.: Visual servo control. ii. Advanced approaches [tutorial]. IEEE Robot. Autom. Magaz. 14(1), 109–118 (2007)

    Article  Google Scholar 

  • Chaumette, F., Rives, P., Espiau, B.: Classification and realization of the different vision-based tasks. In: Visual Servoing: Real-Time Control of Robot Manipulators Based on Visual Sensory Feedback, pp. 199–228. World Scientific (1993)

  • Chen, J., Dawson, D.M., Dixon, W.E., Behal, A.: Adaptive homography-based visual servo tracking for a fixed camera configuration with a camera-in-hand extension. IEEE Trans. Control Syst. Technol. 13(5), 814–825 (2005)

    Article  Google Scholar 

  • Chen, X., Xiao, H.: Multirate forward-model disturbance observer for feedback regulation beyond nyquist frequency. Syst. Control Lett. 94, 181–188 (2016)

    Article  MathSciNet  Google Scholar 

  • Delbrück, T., Linares-Barranco, B., Culurciello, E., Posch, C.: Activity-driven, event-based vision sensors. In: Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pp. 2426–2429. IEEE (2010)

  • Firouzi, H., Najjaran, H.: Real-time monocular vision-based object tracking with object distance and motion estimation. In: 2010 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pp. 987–992. IEEE (2010)

  • Garrido-Jurado, S., Munoz-Salinas, R., Madrid-Cuevas, F.J., Medina-Carnicer, R.: Generation of fiducial marker dictionaries using mixed integer linear programming. Pattern Recogn. 51, 481–491 (2016)

    Article  Google Scholar 

  • Janabi-Sharifi, F., Marey, M.: A kalman-filter-based method for pose estimation in visual servoing. IEEE Trans. Robot. 26(5), 939–947 (2010)

    Article  Google Scholar 

  • Lepetit, V., Moreno-Noguer, F., Fua, P.: Epnp: An accurate O(n) solution to the PnP problem. Int. J. Comput. Vis. 81(2), 155 (2009)

    Article  Google Scholar 

  • Lin, C.Y., Wang, C., Tomizuka, M.: Pose estimation in industrial machine vision systems under sensing dynamics: A statistical learning approach. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 4436–4442. IEEE (2014)

  • Luo, R.C., Chou, S.C., Yang, X.Y., Peng, N.: Hybrid eye-to-hand and eye-in-hand visual servo system for parallel robot conveyor object tracking and fetching. In: IECON 2014-40th Annual Conference of the IEEE Industrial Electronics Society, pp. 2558–2563. IEEE (2014)

  • Malis, E., Chaumette, F., Boudet, S.: 2 1/2 d visual servoing. IEEE Trans. Robot. Autom. 15(2), 238–250 (1999)

    Article  Google Scholar 

  • Mohebbi, A., Keshmiri, M., Xie, W.F.: An eye-in-hand stereo visual servoing for tracking and catching moving objects. In: Proceedings of the 33rd Chinese Control Conference, pp. 8570–8577. IEEE (2014)

  • Morel, G., Liebezeit, T., Szewczyk, J., Boudet, S., Pot, J.: Explicit incorporation of 2d constraints in vision based control of robot manipulators. In: Experimental Robotics VI, pp. 99–108. Springer (2000)

  • Nakabo, Y., Ishikawa, M., Toyoda, H., Mizuno, S.: 1 ms column parallel vision system and its application of high speed target tracking. In: Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), vol. 1, pp. 650–655. IEEE (2000)

  • Sadeghi, F.: Divis: Domain invariant visual servoing for collision-free goal reaching. In: Robotics: Science and Systems (RSS) (2019)

  • Siradjuddin, I., Behera, L., McGinnity, T.M., Coleman, S.: A position based visual tracking system for a 7 dof robot manipulator using a kinect camera. In: The 2012 international joint conference on neural networks (IJCNN), pp. 1–7. IEEE (2012)

  • Wang, C., Lin, C.Y., Tomizuka, M.: Visual servoing considering sensing dynamics and robot dynamics. IFAC Proceedings 6th IFAC Symposium on Mechatronic Systems, vol. 46(5), pp. 45–52 (2013)

  • Wang, C., Lin, C.Y., Tomizuka, M.: Design of kinematic controller for real-time vision guided robot manipulators. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 4141–4146. IEEE (2014)

  • Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes. In: Robotics: Science and Systems (RSS) (2018)

  • Xiao, H., Chen, X.: Following fast-dynamic targets with only slow and delayed visual feedback: A kalman filter and model-based prediction approach. In: Dynamic Systems and Control Conference (2019)

  • Zhang, M., Liu, X., Xu, D., Cao, Z., Yu, J.: Vision-based target-following guider for mobile robot. IEEE Transactions on Industrial Electronics (2019)

  • Zheng, Y., Kuang, Y., Sugimoto, S., Astrom, K., Okutomi, M.: Revisiting the PnP problem: A fast, general and optimal solution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2344–2351 (2013)

Download references

Author information

Authors and Affiliations

  1. Mechanical Engineering, University of Washington, 3900 E Stevens Way NE, Seattle, WA, 98195, USA

    Hui Xiao & Xu Chen

Authors
  1. Hui Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xu Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiao, H., Chen, X. Robotic target following with slow and delayed visual feedback. Int J Intell Robot Appl 4, 378–389 (2020). https://doi.org/10.1007/s41315-020-00151-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41315-020-00151-2

Keywords

Search

Navigation