Collision-free human-robot collaboration based on context awareness

https://doi.org/10.1016/j.rcim.2020.101997Get rights and content

Highlights

  • The paper introduced a collision-free human-robot collaboration system based on context awareness concept that can improve the overall efficiency of the system.

  • An efficient transfer learning based human poses recognition module is proposed to recognise human operators’ assembly poses.

  • An overall human-robot collaboration system architecture is designed and integrated.

  • Different modules and algorithms introduced in the paper are tested with case studies.

Abstract

Recent advancements in human-robot collaboration have enabled human operators and robots to work together in a shared manufacturing environment. However, current distance-based collision-free human-robot collaboration system can only ensure human safety but not assembly efficiency. In this paper, the authors present a context awareness-based collision-free human-robot collaboration system that can provide human safety and assembly efficiency at the same time. The system can plan robotic paths that avoid colliding with human operators while still reach target positions in time. Human operators’ poses can also be recognised with low computational expenses to further improve assembly efficiency. To support the context-aware collision-free system, a complete collision sensing module with sensor calibration algorithms is proposed and implemented. An efficient transfer learning-based human pose recognition algorithm is also adapted and tested. Two experiments are designed to test the performance of the proposed human pose recognition algorithm and the overall system. The results indicate an efficiency improvement of the overall system.

Introduction

Recently, the new developments in the research field of human-robot collaboration (HRC) have generated huge interests. Compared with traditional robotic manufacturing systems [1], HRC manufacturing systems can allow human operators to work together with robots in the same team without time or space separation [2], [3], [4]. In an HRC team, human operators can provide better problem-solving skills, whereas robots can provide better strength and accuracy. The efficiency of the manufacturing system can be further improved with the utilisation of advantages from both human operators and robots. Consequently, human operators will have the opportunity to focus on creative tasks, while robots are commanded to support human operators during assembly.

The most important criteria for an applicable HRC manufacturing system is the safety of human operators. To provide a collision-free environment in all circumstances is the basic requirement for any HRC manufacturing system. However, the current industrial robot control systems are mostly designed with model-based control technologies [5]. Traditional industrial robots do not consider human operators and other moving obstacles during assembly operations. Thus, current industrial robot systems cannot provide a collision-free working environment for human operators in HRC manufacturing systems. To ensure the safety of HRC manufacturing systems, many researchers proposed sensor-based safety systems [6], [7], [8]. By utilising the depth sensors’ real-time monitoring capability, the distance between human operators and robots can be actively monitored. Robots can also be controlled to stop or move away if the distance between human operators and robots is too close. However, the approach, on the other hand, will decrease the efficiency of collaborative assembly, as the robots will frequently move away and stop during the assembly process.

Recently, one of the emerging direction of HRC research is context awareness [9,10]. The situation where human operators and robots collaboratively work together in the same workstation, creates a unique environmental context. Such a context can include the information of general assembly environment, assembly tools, assembly parts, human operators’ behaviours, and assembly sequences, etc. In HRC manufacturing systems, human operators can naturally perceive the context by common sense and observation, whereas robots cannot understand the context easily without sensors and reasoning. If robots can understand and be aware of the environmental context similarly like human operators, a safer and more efficient HRC environment will be possible.

Following the idea of context awareness, in this paper, the authors present a collision-free HRC system that considers the context of HRC assembly, which will guarantee the safety and efficiency of the HRC manufacturing system. Compared with the distance-based safety approach, the proposed system can not only plan robot paths that avoid collisions with human operators but also perceive information from human operators’ assembly poses. By recognising human operators’ poses during assembly, robots can be controlled to different tasks accordingly. To support the above-mentioned requirements, the authors proposed a context-aware collision-free HRC system design that includes a collision sensing module, a path planning module, a real-time HRC interface and a context-aware human pose recognition module. A complete collision sensing module with sensor calibration algorithms is designed and implemented. An efficient transfer learning-based human operator pose recognition algorithm is also adapted and tested.

The remainder of the paper is organised as follows: in Section 2, the related literature is reviewed. In Section 3, the overall system design is introduced. In Section 4, some of the key algorithms and methods of collision sensing are discussed. Section 5 explains the proposed human pose recognition algorithms that can provide efficient recognition of human operators’ assembly poses. Section 6 provides two designed experiments to showcase the capability of the proposed system. Section 7 collects a final overall discussion, before concluding the paper in Section 8.

Section snippets

Literature review

There have been many different studies focused on the development and implementation of HRC safety systems. Some reported studies approached the HRC safety issue from the direction of tracking a human operator and computing the distance between the human operator and the robot [7,[11], [12], [13], [14], [15], [16], [17], [18], [19]]. One of the popular safety designs is the space separation (safety zones) approach. By applying different safety strategies in different predefined safety zones,

Overall system design

The system design of the context-aware collision-free HRC system is shown in Fig. 1. There are four major system components: path planning module, collision sensing module, real-time HRC interface, and context-aware human pose recognition module. The real-time HRC interface provides data connectivity between sensors, robots, and other modules. The collision sensing module collects and tracks human operators, moving objects and robots. Collision event can be triggered if a potential collision is

Calibration

Calibration is the bases for the HRC collision sensing algorithm. There are two different types of calibration in the collision sensing system. Depth sensor intrinsic parameters calibration is the first type of calibration, which will provide quality image for the collision sensing system. However, the intrinsic parameters calibration is not the focus of this study. In this section, the authors will only focus on the other type of calibration: robot hand-eye calibration. Robot hand-eye

Context-aware human pose recognition

As we discussed in previous sections, if the context of HRC assembly environment can be captured and understood by robots, the safety and efficiency of the HRC manufacturing system can be further improved. Some of the contextual information for instance, assembly sequence, can be easily predefined and understood, whereas the majority part of the contextual information, for instance, human operator-related context, is still missing. Some of the recent publications also revealed that the

Experiments

In this section, the authors will test the implemented collision sensing module, path planning module, real-time HRC interface, and context-aware human pose recognition module. Two experiments are designed to test the performance of the implemented context-aware collision-free HRC system. The first experiment is a test on the context-aware human pose recognition module. The second experiment is an overall system performance test.

Discussions

The authors presented a collision-free HRC manufacturing system based on context awareness. The overall design target is to improve the efficiency of the distance-based collision-free approach. With the introduction of context awareness, human operators’ assembly poses can be recognised. Naturally, the robot can be controlled more efficiently, and the efficiency of the system can be improved.

The majority of the system is implemented and integrated based on ROS and MoveIt! related software

Conclusions and future work

In this paper, the authors proposed a collision-free HRC manufacturing system based on context awareness. The proposed system includes collision sensing module, path planning module and context-aware human pose recognition module. The collision sensing module and path planning module are devised to detect potential collision between human operators and robots, and at the same time, plan a path to reach target location. The context-aware human pose recognition module utilises a transfer

References (44)

  • H. Liu et al.

    Human motion prediction for human-robot collaboration

    J. Manuf. Syst.

    (2017)
  • B. Schmidt et al.

    Automatic work objects calibration via a global–local camera system

    Robot. Comput. Integr. Manuf.

    (2014)
  • H. Liu et al.

    Remote human–robot collaboration: a cyber–physical system application for hazard manufacturing environment

    J. Manuf. Syst.

    (2020)
  • H. Liu et al.

    An AR-based Worker Support System for Human-Robot Collaboration

    Procedia Manuf

    (2017)
  • C. Morato et al.

    Toward Safe Human Robot Collaboration by Using Multiple Kinects Based Real-time Human Tracking

    J. Comput. Inf. Sci. Eng.

    (2014)
  • A.M. Zanchettin et al.

    Safety in Human-Robot Collaborative Manufacturing Environments: metrics and Control

    IEEE Trans. Autom. Sci. Eng.

    (2016)
  • P. Wang et al.

    Deep learning-based human motion recognition for predictive context-aware human-robot collaboration

    CIRP Ann.

    (2018)
  • L. Wang et al.

    Symbiotic human-robot collaborative assembly

    CIRP Ann.

    (2019)
  • C. Reardon et al.

    Towards safe robot-human collaboration systems using human pose detection

    IEEE Conf. Technol. Pract. Robot Appl. TePRA. 2015-Augus.

    (2015)
  • D. Kulić et al.

    Pre-collision safety strategies for human-robot interaction

    Auton. Robot.

    (2007)
  • A. Mohammed et al.

    Active collision avoidance for human–robot collaboration driven by vision sensors

    Int. J. Comput. Integr. Manuf.

    (2017)
  • M. Ragaglia et al.

    Safety-aware trajectory scaling for Human-Robot Collaboration with prediction of human occupancy

    Proc. 17th Int. Conf. Adv. Robot. ICAR

    (2015)
  • Cited by (0)

    View full text