Skip to main content
Log in

Robust Localization of Unmanned Surface Vehicle Using DDQN-AM

  • Robot and Applications
  • Published:
International Journal of Control, Automation and Systems Aims and scope Submit manuscript

Abstract

A robust localization algorithm has been proposed for an Unmanned Surface Vehicle (USV) using DDQN-AM (Double Deep Q-Network with Action Memory). Inertial Navigation System (INS) and Global Positioning System (GPS) are used for collecting location information and for controlling the USV with DDQN-AM algorithm. Conventional probability-based filters such as Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF) are suitable for estimating behaviors of nonlinear systems and have been widely applied for localizing various kinds of nonlinear vehicles. However, the algorithms diverge or linearization position error increases in localizing USVs that are required to follow a desired trajectory accurately under high wave and wind conditions. Also, computational complexity is very high, which causes the instability of the vehicle system. To address these problems, DDQN with Adam optimization for error back propagation have been adopted in this research. However, if learning is performed using the training data set of the current time (current state, reward, action) in the replay buffer of DDQN, the learning performance (accuracy for states and running time) may deteriorate and learning load may occur. therefore, to improve the learning performance of DDQN, DDQN-AM has been newly proposed by adding AM in this research. To verify the effectiveness and to show the superior learning performance of the proposed DDQN-AM against the conventional approaches, a simulation using Matlab and field experiments with a prototype USV have been performed at the seashore. As a result, the position error of DDQN-AM was improved by 57.6% compared to the existing approach, and the average running time was also shortened by 1 minute, 30 seconds.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. J. W. Han and J. W. Kim, “Navigation of an unmanned surface vessel under bridges,” Proc. of the 10th International Conference on Ubiquitous Robots and Ambient Intelligence URAI, pp. 206–210, October 2013.

  2. A. Sinisterra, M. Dhanak, and N. Kouvaras, “A USV Platform for Surface Autonomy,” Oceans, pp.1–8, September 2017.

  3. B. Xu, Y. P. Xiao, W. Gao, Y. G. Zhang, Y. L. Liu, and Y. Liu, “Dual-model reverse CKF algorithm in cooperative navigation for USV,” Hindawi Publishing Corporation Mathematical Problems in Engineering, vol. 2014, pp.1–17, November 2014.

    Google Scholar 

  4. B. Xu, J. Bai, G. Wang, Z. Zhang, and W. Huang, “Cooperative Navigation and Localization for Unmanned Surface Vessel with Low-cost sensors,” DGON Inertial Sensors and Systems, pp. 19–32, September 2014.

  5. S. Y. Kim and J. M. Lee, “Estimation of attitude and position of moving objects using multi-filtered inertial navigation system,” Transanctions of the Korean Institute of Electrical Engineers, vol. 60, no. 12, pp. 2339–2345, December 2011.

    Article  Google Scholar 

  6. Z. M. Li, X. H. Chang, and L. Yu, “Robust quantized H filtering for discrete-time uncertain systems with packet droputs,” Applied Mathmatics and Computation, vol. 275, pp. 361–371, February 2016.

    Article  Google Scholar 

  7. X. H. Chang, J. Xiong, and J. H. Park, “Estimation for a class of parameter-controlled tunnel diode circuits,” IEEE Transanctios on Systems, Man and Cybernetics: Systems, vol. 50, no. 11, pp. 4697–4707, August 2018.

    Article  Google Scholar 

  8. H. Xu, N. Wang, H. Zhao, and Z. Zheng, “Deep reinforcement learning-based path planning of under actuated surface vessels,” Cyber-Physical Systems, vol. 5, pp. 1–17, November 2018.

    Article  Google Scholar 

  9. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, pp. 436–444, May 2015.

    Article  Google Scholar 

  10. V. R. Konda and J. N. Tsitsiklis, “Actor-critic algorithms,” Advances in Neural Information Processing Systems, pp. 1008–1014, 2000.

  11. C. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, pp. 279–292, 1992.

    MATH  Google Scholar 

  12. K. S. Kim, J. H. Moon, D. K. Kim, S. B. Jo, and J. M. Lee, “Natural behavior learning based on deep reinforcement learning for autonomous navigation of mobile robots,” Institute of Control, Robotics and Systems, vol. 24, no. 3, pp. 256–262, March 2018.

    Article  Google Scholar 

  13. Z. Peimin, Z. Huajun, and T. Xinchi, “The design of gain scheduling PID controller of the USV course control system,” Proc. of Chinese Automation Congress (CAC), pp. 1408–1413, Dec. 2018.

  14. D. H. Choi, Y. J. Lee, S. M. Hong, and J. Y. Kim, “Development of hovering-type AUV test-bed Octagon,” Journal of the Korean Society of Marine Engineering, vol. 40, no. 6, pp. 516–526, July 2016.

    Google Scholar 

  15. M. Shabani, A. Gholami, and N. Davari, “Asynchronous direct Kalman filtering approach for underwater integrated navigation system,” Nonlinear Dynamics, vol. 80, pp. 71–85, 2015.

    Article  MathSciNet  Google Scholar 

  16. Y. S. Park, W. S. Choi, S. I. Han and J. M. Lee, “Navigation system of UUV using multi-sensor fusion-based EKF,” Journal of Institute of Control, Robotics and Systems (in Korean), vol. 22, no. 7, pp.562–569, July 2016.

    Article  Google Scholar 

  17. K. J. Kim, C. G. Park, M. J. Yu, and Y. B. Park, “A performance comparison of extended and unscented Kalman filters for INS/GPS tightly coupled approach,” Institute of Control, Robotics and Systems, vol. 12, no. 8, pp. 780–788, January 2006.

    Google Scholar 

  18. Z. Mahmoudi, N. K. Poulsen, H. Madsen, and J. B. Jørgensen, “Adaptive unscented Kalman filter using maximum likelihood estimation,” IFAC Papers On Line, vol. 50, no. 1, pp. 3859–3864, July 2017.

    Article  Google Scholar 

  19. P. Isbi, S. A. Jones, and P. Siddarth, “Analysis of different activation functions using back propagation neural networks,” Journal of Theoretical and Applied Information Technology, vol. 47, no. 3, pp. 1344–1348, January 2013.

    Google Scholar 

  20. P. A. Rao, B. N. Kumar, S. Cadabam, and T. Praveena, “Distributed Deep Reinforcement Learning using Tensor flow,” Proc. of International Conference on Current Trends in Computer, Electrical, Electronics and Communication ICCTCEEC, pp. 171–174, December 2018.

  21. X. Han, H. Liu, F. Sun, and X. Zhang, “Active object detection with multistep action prediction using deep Q-network,” IEEE Transactions on Industrial Informatics, vol. 15, no. 6, pp. 3723–3731, January 2019.

    Article  Google Scholar 

  22. L. Tai, G. Paolo, and M. Liu, “Virtual-to-real deep reinforcement learning: continuous control of mobile robots for mapless navigation,” Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 31–36, September 2017.

  23. N. Shone, T. N. Ngoc, V. D. Phai, and Q. Shi, “A Deep Learning Approach to Network Intrusion Detection,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 2, no. 1, pp. 41–50, Februray 2018.

    Article  Google Scholar 

  24. I. Carlucho, M. D. Paula, S. Wang, B. V. Menna, Y. R. Petillot and G. G. Acosta, “AUV position tracking control using end-to-end deep reinforcement learning,” Oceans MTS/IEEE Charleston, pp. 1–8, October 2018.

  25. A. Kim and J. Lee, “Data analysis platform using open source based deep learning engine,” Proc. of International Conference on Platform Technology and Service (Plat-Con), pp. 1–5, January 2018.

  26. H. Rizk, M. Torki, and M. Youssef, “CellinDeep: Robust and accurate cellular-based indoor localization via deep learning,” IEEE Sensors Journal, vol. 19, no. 6, pp. 2305–2312, March 2019.

    Article  Google Scholar 

  27. L. Zhao, M. Roh, and S. J. Lee, “Control method for path following and collision avoidance of autonomous ship based on deep reinforcement learning,” Journal of Marine Science and Technology, vol. 27, no. 4, pp. 293–310, May 2019.

    Google Scholar 

  28. A. Khan, “Deep reinforcement learning based tracking behavior for Underwater vehicles,” Norwegian University of Science and Technology, pp. 1–113, June 2018.

  29. X. H. Chang, Y. Liu, and M. Shen, “Resilident control design for lateral motion regulation of intelligent vehicle,” IEEE/ASME Transactions on Mechatronics, vol. 24, no. 6, pp. 2488–2497, October 2019.

    Article  Google Scholar 

  30. J. Wang and C. Liu, “Stability analysis for a class of nonlinear networked control systems,” Proceedings of the 37th Chinese Control Conference, pp. 1287–1290, July 2018.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jangmyung Lee.

Additional information

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Recommended by Associate Editor Yang Tang under the direction of Editor Myo Taeg Lim.

This research was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2019R1A2C2088859).

Wonseok Choi received his M.S. degree in Electricity and Electronic Engineering from Pusan National University, Korea, in 2016 and a Ph.D. candidate in Division of robotics convergence from Pusan National University, Korea in current. His research interests include deep learning, robotic control and system IoT.

Hosun Kang received his M.S. degrees in Electricity and Electronic Engineering from Pusan National University, Korea, in 2019 and He is current Ph.D candidate in Electricity and Electronic Engineering from Pusan National Univerisity, Korea. His current research interests include artificial intelligent and image processing, localization estimation.

Jangmyung Lee received his B.S. and M.S. degrees in electronics engineering from Seoul National University, Seoul, Korea, in 1980 and 1982, respectively, and his Ph.D. in Computer Engineering from the University of Southern California(USC), Los Angeles, in 1990. Since 1992, he has been a professor with the Intelligent Robot Laboratory, Pusan National University, Busan, Korea. His current research interests include intelligent robotic systems, ubiquitous ports and intelligent sensor. Prof. Lee is a past president of the Korean Robotics Society, and a Vice president of ICROS. He is also the head National Robotics Research Center, SPENALO.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Choi, W., Kang, H. & Lee, J. Robust Localization of Unmanned Surface Vehicle Using DDQN-AM. Int. J. Control Autom. Syst. 19, 1920–1930 (2021). https://doi.org/10.1007/s12555-020-0157-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12555-020-0157-7

Keywords

Navigation