Elsevier

Applied Soft Computing

Volume 110, October 2021, 107601
Applied Soft Computing

Hierarchical deep reinforcement learning to drag heavy objects by adult-sized humanoid robot

https://doi.org/10.1016/j.asoc.2021.107601Get rights and content

Highlights

  • Proposing a hierarchical DQL algorithm for object dragging with adult-sized robot.

  • Proposing a Three-layered Convolution Volumetric Network (TCVN) model for 3D object detection.

  • Proposing a lightweight real-time instance segmentation algorithm named Tiny-YOLACT.

  • Simulating and implementing on a real robot with extensive experimental results.

Abstract

Most research on robot manipulation focuses on objects that are light enough for the robot to pick them up. However, in our daily life, some objects are too big or too heavy to be picked up or carried, so that dragging them is necessary. Although bipedal humanoid robots have nowadays good mobility on level ground, dragging unfamiliar objects including large and heavy objects on various surfaces is an interesting research area with many applications, which will provide insights into human manipulation and will encourage the development of novel algorithms for robot motion planning and control. This is a challenging problem, not only because of the unknown and potentially variable friction of the foot, but also because the feet of the robot may slip during unbalanced poses. In this paper, we propose a novel hierarchical deep learning algorithm that learns how to drag heavy objects with an adult-sized humanoid robot for the first time. First, we present a Three-layered Convolution Volumetric Network (TCVN) for 3D object classification with point clouds volumetric occupancy grid integration. Second, we propose a lightweight real-time instance segmentation method named Tiny-YOLACT for the detection and classification of the floor surface. Third, we propose a deep Q-learning algorithm to learn the policy control of the Center of Mass of the robot (DQL-COM). The DQL-COM algorithm learning is bootstrapped using the ROS Gazebo simulator. After initial training, we complete training on the THORMANG-Wolf, a 1.4 m tall adult-sized humanoid robot with 27 degrees of freedom and weighing 48 kg, on three distinct types of surfaces. We evaluate the performance of our approach by dragging eight different types of objects (e.g., a small suitcase, a large suitcase, a chair). The extensive experiments (480 times on the real robot) included dragging a heavy object with a mass of 84.6 kg (two times greater than the robot’s weight) and showed remarkable success rates of 92.92% when combined with the force–torque sensors, and 83.75% without force–torque sensors.

Introduction

Humanoid robots are one of the important types of robots that researchers are developing rapidly with real-life applications [1], [2], [3], [4], [5]. The last decade’s influence by humanoid robots in the social, healthcare, manipulation, and education domains is considerable [6], [7]. By looking at state-of-the-art developments of robots it becomes clear that the humanoid robot’s development offers significant potential in alleviating tedious and tough tasks that are currently performed by humans. In this vein, the important question is “Why using humanoid robots?”. The answer can be described in three main fundamental functionalities of humanoid robots including (i) humanoid robots’ high adaptability to work in the human-made environment as it is, (ii) humanoid robots ability to use tools for humans as it is, and (iii) humanoid robots human-like shape that makes it easier to adopt with people [8]. Generally, humanoid robots are expected to work alongside humans or work as an alternative to humans in hazardous environments, such as rescue applications [9]. Therefore, dragging large or heavy objects in many environments for example evacuation processes in rescue situations is an important task. Some objects are very big and heavy that normal people cannot pick, carry, and place. So, dragging objects in different conditions is inevitable. Although bipedal humanoid robots have high mobility for dragging objects, the relative disturbance in the Center of Mass (COM) makes the humanoid robot’s balance control very challenging. The relative disturbance can be because of uncertainties in floor friction and also foot slippage along with the unbalanced situation of the robot. So far, just a few studies with a focus on the dragging large or heavy objects problem’s characteristics have been presented [10], [11], [12], [13], [14], [15], [16], [17], [18].

Deep Learning (DL) is a machine learning technique that can learn the optimal policy of highly complex data on a machine decision algorithm through the training process [19]. Therefore, with the development of DL algorithms, Deep Reinforcement Learning algorithms (DRL) with different variants for different applications have been presented [20]. The most well-known DRL algorithm is the Deep Q-Network (DQN), which has even dominated human-level control through playing Atari games [21]. DQN is specified for an unknown and complex dynamic system that uses a discrete control strategy that can be generalized on more features on large-scale datasets. Likewise, in the robotic domain, DQN has a great capacity to apply to different types of humanoid robot control applications [22]. On the vision problems, the DL algorithms have been applied on unified approaches over the last decade and have been shown promising results in the object detection system [23]. There are many challenges that must be overcome before DL algorithms can be applied directly in complex humanoid robot applications. In [24], the authors indicate the challenges of using two-stage detection systems based on various Convolutional Neural Networks (CNN) on a humanoid robot and they highlighted their speed-accuracy tradeoff. Moreover, there are few notable DL works based on a one-staged object detector that can achieve advantages in object detection system’s speed-accuracy [25], [26], [27].

In this paper, we propose a robust hierarchical deep learning method on an adult-sized bipedal humanoid robot in order to drag different objects including a heavy object on various surfaces as shown in Fig. 1.

The proposed algorithm breaks and solves the dragging problem with three hierarchical deep learning-based algorithms. In this regard, first, we present a Three-layered Convolution Volumetric Network (TCVN) for 3D object classification with point clouds volumetric occupancy grid integrating. Input data of the proposed TCVN model used a voxel grid representation from point clouds data acquired from the robot’s LiDAR scanner. Second, we propose a lightweight real-time instance segmentation network called Tiny You Only Look at Coefficients (Tiny-YOLACT) for floor surface detection. The Tiny-YOLACT model is adopted from the original YOLACT [25] model and utilized the ResNet-18 model [28] as a backbone network that segments the floor from the robot’s camera. Third, we propose a deep Q-learning algorithm to learn the control policy of offsetting the Center of Mass (DQL-COM) of the robot while the robot drags different objects, which keep tracking with the robot’s ZMP in the support polygon.

The main contributions of the present paper are as follows. (i) Proposing a Three-layered Convolution Volumetric Network (TCVN) model for 3D object detection as a classification task that outperforms state-of-the-art algorithms. (ii) Proposing a lightweight real-time instance segmentation algorithm named Tiny-YOLACT for floor surface type detection. (iii) Proposing a deep Q-learning algorithm for control policy of object dragging for an adult-sized humanoid robot named DQL-COM for the first time.

To avoid costly experiments in terms of time and real environment constraints, the proposed DQL-COM algorithm is trained on the Robot Operating System (ROS) Gazebo [29] simulator first, then it is adopted with a real robot on three different types of surfaces. Afterward, we implemented and tested the proposed method on the THORMANG-Wolf adult-sized humanoid robot which was a modified version of the commercially available THORMANG3 robot [30]. We evaluated the proposed DQL-COM with two types of experiments on three types of surfaces including eight different objects. In these experiments, we defined two scenarios as using IMU along with a foot Pressure (F/T) sensor and using the single IMU data as input.

The rest of the paper is organized as follows. Section 2 discusses literature reviews and recently proposed algorithms for similar problems. Section 3 describes the problem definition. Section 4 discusses the proposed hierarchical deep learning method in three algorithms in detail. Section 5 provides the experimental result of the proposed method on a THORMANG-Wolf robot. Finally, Section 6 concludes the paper.

Section snippets

Literature review

The bipedal humanoid robot balance control system is a challenging research problem and has been addressed in a variety of studies [31]. Although many state-of-the-art stabilize walking approaches in biped robot have been extensively studied [31], [32], but walking with disturbance such as pushing [10], [11], [12], [13], [14], carrying [15], [16], or lifting [16], [17], [18] large or heavy objects is still an open problem. On the other hand, in the robotic field, some research studies covered

Problem definitions and background

Proper walking in humanoid robots is facing with balance control challenge. By implementing the desired walking pattern to realize a stable walking on a different real-time environmental condition, the control process still can be influenced by uncertainties. In dragging large or heavy objects problem, we assume a robot needs to drag different objects that are heavy to lift or the objects’ shapes or sizes make them difficult to carry on different kinds of surfaces. Therefore, to deal with this

Proposed algorithm

In this section, the design of the proposed algorithm to solve the dragging large or heavy objects problem with the THORMANG-Wolf robot is described. In Fig. 7, we demonstrated a block diagram of the proposed hierarchical deep learning algorithm. We present three proposed deep learning algorithms in a hierarchical platform to solve the dragging large or heavy objects problem as follows. (i) Deep learning algorithm on 3D object classification. (ii) Deep learning algorithm on real-time instance

Experimental results

In this section, to evaluate the accuracy of the proposed method, first, we describe the experimental setup. Then we conduct and evaluate the proposed hierarchical method in four separate sections. First, we show experiments on 3D object classification, where the accuracy of the proposed TCVN model is compared to classified different types of 3D objects. Second, we show experiments of the floor detection on three different surfaces to show the performance of the proposed lightweight Tiny-YOLACT

Conclusion

In this paper, we presented a hierarchical deep learning method in which we used adult-sized biped humanoid robot THORMANG-Wolf to drag different objects including large size and heavy objects. The main objective of the proposed method was to drag different objects on various surfaces through a deep learning algorithm. In general, the proposed approach was to solve the dragging problem, presented in three different deep learning approaches. First, for object detection, we proposed a deep

CRediT authorship contribution statement

Saeed Saeedvand: Conceptualization, Methodology, Writing - review & editing. Hanjaya Mandala: Implementation, Visualization, Data collecting, Writing - original draft. Jacky Baltes: Investigation, Supervision, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was financially supported by the ‘Chinese Language and Technology Center’ of National Taiwan Normal University (NTNU) from The Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan, and Ministry of Science and Technology , Taiwan, under Grant Nos. MOST 108-2634-F-003-002, MOST 108-2634-F-003-003, and MOST 108-2634-F-003-004 (administered through Pervasive Artificial Intelligence Research (PAIR)

References (70)

  • ChoudhuryA. et al.

    Humanoid robot-application and influence

    (2018)
  • MandalaH. et al.

    Synchronous dual-arm manipulation by adult-sized humanoid robot

  • KajitaS. et al.

    Introduction To Humanoid Robotics

    (2014)
  • SaeedvandS. et al.

    Robust multi-objective multi-humanoid robots task allocation based on novel hybrid metaheuristic algorithm

    Appl. Intell.

    (2019)
  • T. Takubo, K. Inoue, T. Arai, Pushing an object considering the hand reflect forces by humanoid robot in dynamic...
  • MotoiN. et al.

    Real-time gait planning for pushing motion of humanoid robot

    IEEE Trans. Ind. Inf.

    (2007)
  • D. Omrčen, C. Böge, T. Asfour, A. Ude, R. Dillmann, Autonomous acquisition of pushing actions to support object...
  • S. Nozawa, Y. Kakiuchi, K. Okada, M. Inaba, Controlling the planar motion of a heavy object by pushing with a humanoid...
  • M. Murooka, S. Nozawa, Y. Kakiuchi, K. Okada, M. Inaba, Whole-body pushing manipulation with contact posture planning...
  • K. Harada, S. Kajita, H. Saito, M. Morisawa, F. Kanehiro, K. Fujiwara, K. Kaneko, H. Hirukawa, A humanoid robot...
  • J.C. Vaz, H. Lee, Y. Jun, P. Oh, Towards tasking humanoids for lift-and-carry non-rigid material, in: 2017 14th...
  • Y. Ohmura, Y. Kuniyoshi, Humanoid robot which can lift a 30kg box by whole body contact and tactile feedback, in: 2007...
  • A. Laurenzi, D. Kanoulas, E.M. Hoffman, L. Muratore, N.G. Tsagarakis, Whole-body stabilization for visual-based box...
  • LeCunY. et al.

    Deep learning

    Nature

    (2015)
  • MnihV. et al.

    Human-level control through deep reinforcement learning

    Nature

    (2015)
  • ArulkumaranK. et al.

    Deep reinforcement learning: A brief survey

    IEEE Signal Process. Mag.

    (2017)
  • Ruiz-del SolarJ. et al.

    A survey on deep learning methods for robot vision

    (2018)
  • JavadiM. et al.

    Humanoid robot detection using deep learning: A speed-accuracy tradeoff

  • D. Bolya, C. Zhou, F. Xiao, Y.J. Lee, YOLACT: Real-time instance segmentation, in: 2019 IEEE/CVF International...
  • MaturanaD. et al.

    Voxnet: A 3d convolutional neural network for real-time object recognition

  • M. Simon, K. Amende, A. Kraus, J. Honer, T. Samann, H. Kaulbersch, S. Milz, H. Michael Gross, Complexer-yolo: Real-time...
  • HeK. et al.

    Deep residual learning for image recognition

    (2015)
  • . Gazebo,...
  • R. Inc, THORMANG3 Full size open platform humanoid,...
  • XieY. et al.

    A review: Robust locomotion for biped humanoid robots

    J. Phys. Conf. Ser.

    (2020)
  • Cited by (14)

    • Implementation of intelligent navigational techniques for inter-collision avoidance of multiple humanoid robots in complex environment

      2022, Applied Soft Computing
      Citation Excerpt :

      Their hybrid method involves a group of diverse humanoid robots attempting to maximize five goals (assignment rewards, assignment completion span, total power, maximal energy usage for a single robot, and ailed timeline penalties). Authors have presented a unique hierarchical deep learning system that teaches an adult-sized humanoid robot learning to transport bulky items [21]. For humanoids in complex surroundings, a unique navigational control system has been conceived and installed in humanoid NAO based on the regression technique [22].

    • Lightweight mechatronic system for humanoid robot

      2023, Knowledge Engineering Review
    • Autonomous Navigation of Robots: Optimization with DQN

      2023, Applied Sciences (Switzerland)
    View all citing articles on Scopus
    View full text