Peg-in-hole assembly in live-line maintenance based on generative mapping and searching network

https://doi.org/10.1016/j.robot.2021.103797Get rights and content

Highlights

  • Dividing the peg-in-hole assembly into two phases.

  • Targets detected by YOLOv3 in the phase of rough alignment.

  • Establishing a hole-searching model by fusing the signals of vision and force.

  • Generative Mapping and Searching Network for optimizing the hole-searching routine.

  • The network is trained in the virtual environment and applied in the actual scene.

Abstract

Replacement of lightning arrester is one of the common tasks in live-line maintenance, and peg-in-hole assembly is a very difficult operation for a robot, because there are visual inaccuracy and force model uncertainty in the process of assembly. This paper presents a new implementation approach fusing signals of vision detection and fuzzy force to realize the high efficiency peg-in-hole assembly by a manipulator autonomously. YOLOv3 is applied as the visual detection network for rough alignment. In the phase of precise hole-searching, we establish a two-dimensional hole-searching model by fusing signal of vision detection and fuzzy force as the condition of state transitions, and propose a new semi-supervised learning network to optimize the hole-searching routine. The performance of the approach is verified by experiments in the simulation environment and the laboratory environment.

Introduction

Uninterrupted power supply plays an important role in people’s daily life. In order to ensure the continuous and stable power supply, it is necessary for workers to do dangerous live-line maintenance work at high altitudes. Obviously, developing an intelligent robot to do the maintenance work autonomously can avoid operation risks and improve the work efficiency, which is of great significance [1], [2], [3].

However, the scene of live-line maintenance is unstructured, and the typical parts such as the cable is flexible and prone to deformation. The task which involves peg-in-hole assembly, requires high accuracy of control [4]. Therefore, robots should have not only the ability to detect objects and scenes similar to human eyes, but also the dexterous operation ability similar to human hands, as well as the learning, adaptation and decision-making ability similar to human brain. It is a very challenging task to develop a live-line maintenance robot which can complete such complex task.

Lightning arrester replacement is one of the common tasks in live-line maintenance. The lower screw of the arrester is inserted into the hole of the cross arm and fixed by the nut, and the upper screw is used as the terminal to fix end of the cable. The most difficult problem for replacing the lightning arrester is to install the crimp terminal on the upper screw of the arrester, which can be regarded as a peg-in-hole assembly, as the upper screw of the arrester can be regarded as a “peg”, and the hole of the crimp terminal is regarded as a “hole”. Fig. 1(a) shows the typical parts of the peg-in-hole assembly. In the installation process, when the crimp terminal is close to the screw, it will inevitably block the screw, and it is difficult for the manipulator to distinguish the peg and the hole under the visual detection, as shown in Fig. 1(b). Furthermore, when the manipulator grasps the flexible cable, it is difficult to build an accurate force model for the cable due to its deformation, as shown in Fig. 1(c).

In this paper, the peg-in-hole assembly strategy based on vision and force are studied, taking robotic lightning arrester replacement as an example, and aiming at the difficult problem of installing the crimp terminal on the screw. The main contribution of this paper is to propose an approach for the manipulator to autonomously complete the assembly of the crimp terminal and the screw. Contribution details are as follows:

  • We divide the task of installing the crimp terminal into two phases, the rough alignment by visual servo and the precise hole-searching by visual/force fusion. The visual servo in the phase of rough alignment is designed to make the crimp terminal close to the screw rapidly when the distance between them is far away.

  • We establish a two-dimensional hole-searching model. The positional relationship between the crimp terminal and the screw can be judged by the visual and force sensor.

  • We propose a hole-searching policy-Generative Mapping and Searching Network (GMSN), which estimates the optimal routine from a known two-dimensional state map. The training result of GMSN is applied in the actual environment, as it was trained only in the virtual environment. Through this network, the time consuming of hole searching can be effectively reduced and the success rate is improved.

The rest of this paper is organized as follows. In Section 2, the related work in this area is presented. The proposed approach is given in detail in Section 3. In Section 4, we perform the experimental analysis, which is followed by a summary and prospects for future research in Section 5.

Section snippets

Related work

The peg-in-hole assembly with manipulator has been widely used in the operation of production lines, which is a very difficult task [5], [6]. Strategies of the solution can be divided into four types, including visual servo, force/torque servo, control without additional sensors, and hole searching policy.

The peg-in-hole assembly can be accomplished by using the position control under structured and sample scenes. The position difference between the target hole and the peg is obtained as the

Methodology

In order to automatically accomplish the peg-in-hole assembly, the manipulator needs to have the ability to identify and locate the crimp terminal and the screw spatially. The location information can be obtained by depth-camera-based target detection and ranging. Nevertheless, the information is not accurate enough to complete high-precision hole alignment. In addition, when the crimp terminal is close to the screw, it will block the screw, resulting in signal loss. As a result, it is

Experiments

We set up the experimental platform by referring to the actual distribution lines, and took pictures of the actual scene to make the background panel. The experimental platform was equipped with a manipulator (UR5), a depth camera (Intel Realsense D435) behind the manipulator, a force/torque sensor(Robotiq FT300), the lightning arrester and desktop computer, as shown in Fig. 7. The proposed network is trained in Titan XP, and experiments are conducted in an indoor environment. In this paper,

Conclusion

This paper presents a novel framework for the peg-in-hole assembly in live-line maintenance. YOLOv3 network and visual servo are used to realize the rough alignment of the crimp terminal and the screw. A two-dimensional hole-searching model is established by fusing signals of vision and force. Through this model, the state of each waypoint on the search routine can be obtained. Furthermore, we propose a new semi-supervised learning network-GMSN, which contains the G-net for estimating the true

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by National Natural Science Foundation of China (61973167), and Key Research and Development Project of Jiangsu Province (BE2017161).

Wei Wu received the B.Sc. degrees in 2015 and is currently pursuing the Ph.D. degree in Control Science and Engineering at Nanjing University of Science and Technology. His research interests focus on the visual servo and autonomous control of robots.

References (33)

  • YanY. et al.

    Research on mechanism configuration and coordinated control for power distribution network live working robot

    Ind. Robot: Int. J. Robot. Res. Appl.

    (2020)
  • NagarajanP. et al.

    Vision based pose estimation of multiple peg-in-hole for robotic assembly

  • KimY.-L. et al.

    Hole detection algorithm for chamferless square peg-in-hole based on shape recognition using f/t sensor

    Int. J. Precis. Eng. Manuf.

    (2014)
  • DietrichF. et al.

    On contact models for assembly tasks: Experimental investigation beyond the peg-in-hole problem on the example of force-torque maps

  • ShirinzadehB. et al.

    A hybrid contact state analysis methodology for robotic-based adjustment of cylindrical pair

    Int. J. Adv. Manuf. Technol.

    (2011)
  • LiuS. et al.

    Sensing and control for simultaneous precision peg-in-hole assembly of multiple objects

    IEEE Trans. Autom. Sci. Eng.

    (2020)
  • Wei Wu received the B.Sc. degrees in 2015 and is currently pursuing the Ph.D. degree in Control Science and Engineering at Nanjing University of Science and Technology. His research interests focus on the visual servo and autonomous control of robots.

    Hui Zhou received the B.S. degree in control science and engineering, the M.S. degree in instrumentation and automation, and the Ph.D. degree in control theory and engineering from Huazhong University of Science and Technology, Wuhan, China, in 2005, 2008, and 2012, respectively. Since 2012, he has been an Assistant Professor in the Research Center for Neural Engineering, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. Since 2018, he has been an Associate Professor in the School of Automation, Nanjing University of Science and Technology, Nanjing, China. His research interests include rehabilitation robots, motion control and modeling, and human machine interaction.

    Yu Guo received the B.Sc. and M.Sc. degrees in Automation, both from Huazhong University of Science and Technology, Wuhan, China, in 1984 and 1987, respectively, and her Ph.D. degree in Control Science and Engineering from Nanjing University of Science and Technology. In 1987, she joined the faculty of the School of Automation, Nanjing University of Science and Technology, and is currently a Professor of Automatic Control there. Her main research interests include intelligent robot control, optimization for complicated systems and so forth.

    Yifei Wu received the Ph.D. degree in Automation at Nanjing University of Science and Technology and is currently an Associate Professor of Nanjing University of Science and Technology. His research interests focus on servo motor control and intelligent robot.

    Jian Guo received the Ph.D. degree in Automation at Nanjing University of Science and Technology in 2002. He was a visiting scholar at Purdue University from 2008 to 2009 and is currently a Professor of Automatic Control at Nanjing University of Science and Technology. His research interests focus on robot system design and automatic control.

    This work was supported by National Natural Science Foundation of China (61973167), and Key Research and Development Project of Jiangsu Province (BE2017161).

    View full text