Environment Perception Technologies for Power Transmission Line Inspection Robots

Chen, Minghao; Tian, Yunong; Xing, Shiyu; Li, Zhishuo; Li, En; Liang, Zize; Guo, Rui

doi:https://doi.org/10.1155/2021/5559231

Journal of Sensors

On this page

Abstract Introduction Conclusion Conflicts of Interest Acknowledgments References Copyright Related Articles

Special Issue

Sensors, Signal, and Artificial Intelligent Processing

View this Special Issue

Review Article | Open Access

Volume 2021 | Article ID 5559231 | https://doi.org/10.1155/2021/5559231

Environment Perception Technologies for Power Transmission Line Inspection Robots

Minghao Chen,^1,2Yunong Tian,^1,2Shiyu Xing,^1,2Zhishuo Li,^1,2En Li ,^1,2Zize Liang,^1,2and Rui Guo³

Academic Editor: Bin Gao

Received07 Jan 2021

Revised27 Jan 2021

Accepted11 Mar 2021

Published31 Mar 2021

Abstract

With the fast development of the power system, traditional manual inspection methods of a power transmission line (PTL) cannot supply the demand for high quality and dependability for power grid maintenance. Consequently, the automatic PTL inspection technology becomes one of the key research focuses. For the purpose of summarizing related studies on environment perception and control technologies of PTL inspection, technologies of three-dimensional (3D) reconstruction, object detection, and visual servo of PTL inspection are reviewed, respectively. Firstly, 3D reconstruction of PTL inspection is reviewed and analyzed, especially for the technology of LiDAR-based reconstruction of power lines. Secondly, the technology of typical object detection, including pylons, insulators, and power line accessories, is classified as traditional and deep learning-based methods. After that, their merits and demerits are considered. Thirdly, the progress and issues of visual servo control of inspection robots are also concisely addressed. For improving the automation degree of PTL robots, current problems of key techniques, such as multisensor fusion and the establishment of datasets, are discussed and the prospect of inspection robots is presented.

1. Introduction

Traditional PTL inspection methods include line crawling inspection, ground-based inspection, and manual inspection with telescopes, as shown in Figure 1. Their defects are clearer with the progress of the power system. These methods are slow and dangerous and may not be conducted sometimes. The reasons are as follows: (a) intricate and diverse workspace. The arrangement of PTL corridors includes several scenarios, such as overhead ground wire and multibundled conductors. The slope of conductors is different. What is more, a great variety of obstacles are on the PTL, as shown in Figure 2 [1]. (b) Geographical conditions of PTL are various. Part of the PTL is located in some complex areas such as swamps, lakes, and mountains. Although the speedy and maneuverable helicopter inspection method can overcome this difficulty, its detection precision of small-scale objects is affected by the long working distance. Consequently, it is also not the best inspection method. Automatic PTL inspection technology is eager to be developed in these cases, which is also a challenging task. Apart from mechanical design, PTL environment perception and control technologies are the foundation of automatic PTL inspection. And they are research hotspots in PTL inspection.

(a)

(b)

(c)

In the whole process of automatic PTL inspection, the main tasks completed by inspection robots now are as follows: (a) 3D reconstruction of PTL corridors [2, 3]. 3D reconstruction can be employed for the visualization of PTL, which is helpful for inspectors to analyze the excursion trend of PTL corridors, interference of surrounding environment, broken point detection, and snow loading. (b) Fine inspection of pylons [4] and components [5, 6]. Object detection is exploited to recognize and locate them and their defects, such as cracked nut, bolt looseness, and fitting corrosion, which are difficult to be found by manual inspection. Besides, object detection can be interfered with by rain, snow, and wind. Therefore, 3D reconstruction is also applied to a local map to acquire a more precise target position. (c) Obstacle crossing and fault maintenance [7, 8]. Obstacle crossing and fault maintenance can be realized by remote teleoperation or autonomous control. Remote teleoperation is easy to realize, which depends on the manual operation of a ground control unit. However, it is difficult to achieve autonomous operation. Due to much interference existing in the PTL environment, the autonomous operation of inspection robots requires a more robust control system. In contrast to sensor-based control, visual servo control is more robust and flexible. It is suitable for the unpredictable environment of PTL corridors. Aimed at these tasks, 3D reconstruction, object detection, and visual servo of PTL inspection are summarized and analyzed for providing a widely comprehensible review.

The rest of this paper is organized as follows: Section 2 describes the development of the transmission line corridor in 3D reconstruction. Section 3 introduces the progress of traditional and deep learning-based detection algorithms in the field of PTL inspection. Then, their merits and demerits are considered. Section 4 concerns the visual servo of different inspection robots. Section 5 analyzes key technical problems and presents future development directions of inspection robots. Finally, the conclusion is presented in Section 6.

2. 3D Reconstruction of PTL Inspection

3D reconstruction is a means to obtain 3D data of measured objects. It is mainly used to acquire object texture, structure, and scale. There are two approaches to 3D reconstruction: the contact and noncontact methods. In contrast to the former way, noncontact methods are extensively used. Based on whether or not to actively transmit measurement signals, they are divided into active and passive visual means. According to various principles, active visual methods are classified as laser methods [9–13], structured light methods [14], and interferometry [15–17]. On the basis of the number of vision sensors used, passive visual methods can be classified as monocular vision [18, 19], binocular vision [20, 21], and multieye vision [22, 23].

In the field of PTL inspection, active visual methods are chiefly used to construct the 3D model of the PTL corridor. The characteristics of objects can be intuitively expressed by the 3D model. It is helpful for inspectors to discover existing or potential hazards in the transmission line corridor.

2.1. The 3D Reconstruction of Power Lines

The power line plays a key role in the entire PTL corridor as the main carrier of electricity supply. 3D reconstruction of power lines is the foundation for the analysis of wire sag, icing, wind deviation, and distance measurement. Therefore, many studies of it are presented.

2.1.1. Active Visual Methods

Now, laser scanning is normally adopted to complete power line reconstruction. It contains three sections, which are point cloud extraction, point cloud segmentation, and model fitting. Point cloud extraction is divided into simple extraction and fine extraction. The raw data contains point clouds of various objects. The simple extraction is utilized to eliminate point clouds of ground. In addition, fine extraction is employed to further eliminate point clouds that do not belong to lines. Point cloud segmentation falls into determining point clouds of one span and a single conductor. Firstly, the data of different spans are divided according to the highest suspension point. And then, the data of a single power line in one span are confirmed. Finally, the model is selected for fitting.

The extraction of point clouds is generally based on filtering methods. Firstly, DTM [24] is generated by filtering. Then, the points of the ground and objects are differentiated. After that, features such as elevation are employed to obtain the point clouds of power lines. Methods based on TIN such as PTD [25] are used commonly. Yu et al. [26] utilized PTD to eliminate ground points. Furthermore, an angle filter was adopted to acquire point clouds of power lines. In addition to PTD, slope filtering [27] and morphological filtering [28] are also used for eliminating ground points. Nevertheless, they are less applied in the extraction of power lines compared to PTD. The elevation is used in conjunction with DTM to gain the point clouds of power lines. Based on nDSM, Liu et al. [29] used the elevation histogram statistical method to get point clouds of power lines. Besides, there is also direct use of elevation to complete extraction. Shen et al. [30] proposed an elevation threshold segmentation algorithm based on the subspace feature and an elevation density segmentation algorithm. Some other methods for point cloud extraction are also employed. Mclaughlin [31] used a GMM and the EM algorithm for extraction. What is more, Jwa and Sohn [32] adopted Hough transform, eigenvalue analysis, point density analysis, and the one-outlier testing technique for extraction.

In the process of segmentation, point clouds of lines in one span are decided by the position of pylons. The position of pylons can be obtained by the height [26, 33] or the density of point clouds [29]. Moreover, the segmentation of point clouds can be also completed by the minimum linkage hierarchical clustering [34] and the second derivative [35]. There are three types of techniques to determine point clouds of a single conductor. (a) Clustering. To distinguish each power line, point clouds that belong to the same power line in one span are clustered. Lai et al. [36] proposed a cluster analysis method based on spatial distance restriction. What is more, Lin et al. [35] presented the 3D connected component analysis based on fixed-radius near neighbors (FNN) and the -means algorithm [37] of normalized projection. (b) Hough transform (HT). Yu et al. [26] and Wu et al. [33] employed HT for the segmentation of the single power line. Melzer et al. [34] used an iterative version of the HT, but this method was unstable. (c) Local growth. The means gradually merges or grows to form the entire power line model, such as the local affine model [31] and the piecewise model growing (PMG) [32].

Four types of curve fitting models of power lines are typically employed: (a) the polynomial model [33, 38], (b) the model that combines a line segment and a parabola [36, 37], (c) the model that combines a line segment and the quadratic polynomial with two variables [29, 39], and (d) the model that combines a line segment and a catenary [26, 31, 34, 35]. The first model directly fits the entire power line. However, the other three models split the power line into a line segment and a curved part for fitting. The difference between the latter three models is the expressions of curve fitting. The second model is the approximate expression of the fourth model. And the performance of the second model is much better because of its simpler formula. Besides, Zhang et al. [40] improved the second model and the fourth model. Overall, the second model is better.

In the above literature, the methods used in each step of the paper are shown in Table 1, which completed the whole process of 3D reconstruction.

Now, 3D reconstruction of power lines is focused on areas with simple structures. The background of PTL, areas of different structures, and various types of power lines are considered much less. It is difficult to identify and reconstruct sleeve nodes and spacer nodes. Moreover, the density of LiDAR point clouds influences firsthand 3D reconstruction.

2.1.2. Passive Visual Methods

Passive visual methods are less employed in the 3D reconstruction of power lines, and their accuracy is lower than that of active visual methods. Zhang et al. [41] developed a multiangle imaging power line detection system to reconstruct 3D power lines, but the method’s accuracy was low. Ganovelli et al. [42] presented a means of using a handful of images to reconstruct power lines. This means employed SFM. Because the method was based on specific assumptions, it could not be used in inspection tasks actually. Maurer et al. [43] presented a means that used semantic segmentation based on FCN and multiview geometry. The flowchart is shown in Figure 3. The method was effective but had high requirements on hardware. It is hard to match different images with the lack of effective point features of power lines by passive visual methods. Therefore, active visual methods are mostly employed in the power line reconstruction.

2.2. The 3D Reconstruction of Ground Surface and Pylons

In addition to the reconstruction of power lines, the PTL corridors also need to reconstruct the ground surface and pylons. Nonetheless, their 3D reconstruction is in the preliminary stage. With regard to the ground surface, Maurer et al. [43] completed the 3D reconstruction of power lines and the ground surface simultaneously. In light of the reconstruction of pylons, Chen et al. [44] raised a model-driven approach to reconstruct pylons. The pylon could be divided into head, body, and foot. The three segments were reconstructed by different strategies. And they were assembled to the whole pylon by utilizing the direction and position. The result showed that the method’s accuracy could reach the centimeter level. Guo et al. [45] presented a means based on model library and stochastic geometry for pylon reconstruction. This method built an energy model of the correlation between point clouds and pylons. It also used simulated annealing and an MCMC sampler to reconstruct pylons. But this means required plenty of calculations.

Recently, oblique photography has been applied to 3D reconstruction of PTL corridors. Xi et al. [46] designed a UAV power inspection system based on oblique photography. The system used the UAV to obtain multiview images of ground objects and accelerated the 3D reconstruction of the PTL corridors through data processing and model refinement. Pei et al. [47] designed a data acquisition device for 3D reconstruction of PTL corridors, which could obtain the structural information of the pylons’ four facades. Oblique photography has some defects in the field of PTL inspection. Oblique photography requires 60%-80% of the air belt overlap. Hence, it is necessary to collect data by multiple flights and the collection efficiency is low. Oblique photography is better for the 3D reconstruction of larger objects but ordinary for small and hollow objects such as power lines and pylons. Consequently, it is imperative to combine oblique photography with other means to gain good reconstruction results. It will be one of the research directions for the 3D reconstruction of PTL corridors in the future.

3. Object Detection of PTL Inspection

Object detection refers to a method of identifying and positioning targets from the background. In nearly two decades, the development of object detection is separated into the traditional detection period and deep learning-based detection period [48]. In the early stage of research on PTL inspection robots, traditional detection algorithms were used to identify and locate power components. With the performance of handcrafted features that tended to be stable, the research on traditional detection algorithms was almost stagnant. The development of inspection robots’ object detection fell into a bottleneck. On account of the emergence of R-CNN [49], the progress of object detection algorithms had taken a quantum leap. Neural networks are now used in object detection. They are able to boost detection performance and acquire robust characteristics. As a result, the research on object detection for PTL inspection robots has gradually changed from traditional detection algorithms to deep learning-based detection algorithms.

3.1. Traditional Object Detection Algorithms

Traditional object detection algorithms used on inspection robots were mostly foreground modeling-based methods over the last decades. They are classified as region selection, feature extraction, and classification. Sliding windows are used to acquire many candidate bounding boxes of the entire picture. And then, features of candidate bounding boxes are extracted. Classifiers, such as SVM [50], AdaBoost [51], and multilayer perceptron (MLP), are used to determine objects of candidate bounding boxes as the target or background. At last, NMS [52] is used to complete the detection. In the field of PTL inspection, SIFT [53], SURF [54], LBP [55], and HOG [56] are often used in traditional detection algorithms. Relevant research can be divided into two aspects: (a) pylons and (b) insulators and fittings.

3.1.1. Pylons

While conducting pylon detection, pylons and birds’ nests are mainly detected. For pylons, Sampedro et al. [57] advanced a power pylon detection method based on HOG and MLP. Two MLPs were trained in this method. They were used for background-foreground segmentation and the classification of four types of pylons. The system architecture is shown in Figure 4. Cerón et al. [58] advanced a pylon detection method based on a line detection method and a grid of two-dimensional feature descriptors. Pylons could be characterized well by the descriptors. Wang et al. [59] used HOG and SVM to complete the rapid detection of pylons, but this method’s dismissal alarm rate was higher than the false alarm rate. Wang et al. [60] presented a method based on HOG for detecting the pylon components from far to near. This method used HOG features of the pylon in different orientations to train the MLP. It was suitable for pylon detection in an open environment. Aiming at the bird’s nest, Zhang et al. [61] raised the means based on the coarse-fine search tactics. It used HOG and AdaBoost to detect pylons’ position and roughly determined the candidate bounding boxes of birds’ nests. Then, color features were used to subtly detect birds’ nests. Xu et al. [62] proposed a method based on HSV and texture features. This method could eliminate much interference caused by areas, which were similar to the color and texture of the bird’s nest under the complex background.

(a)

(b)

3.1.2. Insulators and Fittings

In terms of insulators and fittings, many of the detection algorithms are designed for a single object, while only a few algorithms are for multiobjects. For insulators, Zhao et al. [63] came up with an insulator detection means based on SIFT and RANSAC. This method used RANSAC to remove abnormal SIFT features of insulators and completed the insulator detection through affine transformation. After that, Zhao and Liu [64] presented an approach based on SURF and intuitionistic fuzzy set (IFS). The steps are shown in Figure 5. The insulator’s SURF features were divided by IFS. Then, the connected regions of all categories were calculated to obtain the smallest circumscribed rectangle of each region for positioning. Prasad et al. [65] used SVM and local binary pattern histogram Fourier (LBP-HF) to complete health status detection of insulators, with a precision of 93.33%.

For fittings, Zhang et al. [66] proposed a fitting means based on HOG. This method used PCA to reduce the dimension of HOG features. Then, these features were identified by SVM. In contrast to other traditional algorithms, this means is more accurate. Zhang et al. [67] raised a method that could detect vibration dampers under the complex background. This method used Relief-F to weight and merge aggregate channel features (ACF) and complex frequency domain features. Ultimately, AdaBoost and NMS were used to complete the detection of vibration dampers. Feng et al. [68] employed HOG and SVM to accomplish the detection of bolts, but the accuracy of the means was low. Fan et al. [69] presented a method based on an improved HT for bolt detection. The peak selection strategy of HT was ameliorated for the improvement of detection accuracy.

On the grounds of size, the targets of PTL can be classified into the pylon level, the device level, the part level, and the component level [70]. At the moment, object detection research is focused on targets of pylon level and device level. There is less research on detecting targets of the part level and component level. Traditional object detection algorithms applied to pylon level and device level generally use the framework of “feature expression” + “classifier”. Consequently, features and classifiers are vital for the precision of detecting different objects. In traditional algorithms, basic features usually cannot perfectly characterize the detected target. Features need to be improved as the detected object is replaced. This phenomenon is more obvious in the field of PTL inspection. There are nearly two thousand kinds of fittings in different specifications used in the field of PTL inspection. The features acquired from different regions cannot be unified. For the purpose of ensuring precision, it is imperative to design features for specific targets. Although the features designed for specific objects are intuitive and accessible, it has drawbacks such as low efficiency and poor generalization ability. In addition, tasks of PTL inspection require the robustness and real-time performance of algorithms. Facing the changes of light and viewing angle in the environment, handcrafted features are less robust. Besides, detection accuracy will be immensely influenced in the face of harsh operating environments. Moreover, handcrafted features are highly complex. And they are poorly real-time, which cannot satisfy needs of PTL tasks.

3.2. Deep Learning-Based Object Detection Algorithms

Powered by deep learning, R-CNN [49] broke the deadlock of stagnation of object detection. Plenty of similar approaches emerged in the following years, and they are also applied in PTL. These algorithms are classified as the “two-stage detection” and the “one-stage detection”. In “two-stage detection” algorithms, since the speed of Faster R-CNN is relatively quick [71, 72], it is often used in PTL inspection. In “one-stage detection” algorithms, the algorithms of the YOLO series [73–76] are used much more. Object detection algorithms, which are based on deep learning, can also be classified as two aspects like traditional algorithms.

3.2.1. Pylons

In terms of pylons, Guo et al. [77] presented a real-time pylon detection model based on YOLO. The method used -means to improve the anchor parameters in YOLO. Its detection rate was about twenty frames per second, and mAP was 94.09%. Wang et al. [78] verified the performance of different networks for detecting pylons. For birds’ nests, Shi et al. [79] proposed a bird’s nest detection model by employing RetinaNet [80]. This model was accurate but not real-time. No other neural network models had been tried as the backbone in this model, so there were some limitations in the experiment. Aiming at the same task, Wang et al. [81] proposed an approach, which was used to detect birds’ nests in a multiscale of high-voltage pylons. For enhancing the detection ability for birds’ nests, multiscale convolution features and region proposals were employed in this model, which was based on Faster R-CNN. The model was evaluated on a test set composed of two thousand images. And the average detection precision reached 84.55%.

3.2.2. Insulators and Fittings

For insulators, Liu et al. [82] employed Faster R-CNN to detect insulators. The precision reached 94%. Zhao et al. [83] proposed an approach based on improved Faster R-CNN. This approach achieved better detection of insulators by improving NMS and anchor generation in RPN. It also solved the problem of missed detection caused by insulator shielding. Han et al. [84] designed a new neural network to detect insulators. There was high average precision but low real-time performance in the model.

In terms of fittings, the detection based on deep learning is no longer limited to a single target. In recent years, research on multiobject detection has emerged. The targets include a variety of fittings and insulators. Lin et al. [85] advanced a means based on improved Faster R-CNN. The means could keep a good detection performance in different resolutions, angles, and positions. Dong [86] came up with a real-time multidevice detection method based on YOLOv3. Data augmentation was used to improve the model’s performance. And it had good average accuracy for insulators and vibration dampers. Yang et al. [87] presented an MSFF-KCD algorithm to detect multiscale devices. The precision of this means could maintain 86% on ARM devices, but the speed is not quick. In addition, Yang et al. [88] compared the performance of different backbone networks based on SSD [89] and used feature fusion to improve detection accuracy. For the purpose of detecting fittings, Qi et al. [90] presented an improved SSD model. This model improved the detection performance of small or dense targets in a complex background by improving Intersection over Union (IoU) and using the repulsion loss function. For the same task, Liu et al. [91] proposed a means based on R-FCN. Online hard example mining, sample adjusting, and soft NMS were employed to promote the performance of R-FCN.

The above research of multidevice detection indicates that deep learning-based detection algorithms have a good generalization ability. They do without a complicated process of handcrafted features and can choose suitable and robust features to express targets. For multiscale problems, similar to the traditional algorithms, their performance is poor. So it is necessary to consider the use of multiscale technologies, such as improved anchor generation, SNIP [92], and TridentNet [93], to solve this problem. It is a good direction for the power line inspection study.

Due to the confidentiality of power line inspection datasets, there is no one public dataset now, which hinders the progress of power line inspection. So only some typical deep learning-based detection algorithms for PTL inspection are listed in Table 2 for comparing performance. In China, there are two power grid companies, which are State Grid Corporation of China and China Southern Power Grid. They have massive datasets of the power grid. Consequently, if they cooperatively create a public dataset, which combines data from power grids across the country, researchers could first use public datasets to testify the effectiveness of the means or the model proposed by themselves and show their results based on private datasets.

Although it is impossible to accurately compare the performance between these models, it can be seen that the performance is improved due to new modified models. However, there do not exist neural network models, which achieve simultaneously high precision and speed, in object detection. Just as these two series, the precision and the speed are focused on severally. The improved algorithms based on these two series often sacrifice real time to improve detection accuracy. Now researchers always use YOLO series to detect objects because of its excellent real-time performance. Nevertheless, PTL inspection should be able to fulfill the high requirements in both aspects. Thus, many improved algorithms are only in the research stage. Moreover, limited by computational capabilities of embedded platforms, they cannot be actually used in engineering. Besides, the deep learning model used in engineering needs to be capable of training with new data, but little research is ongoing in this area.

4. Visual Servo of PTL Inspection

The visual servo system takes visual information as feedback. After the relative pose between the robot and the target is computed by visual information, it can realize the robot’s pose control. Three classification standards including types of feedback [95–97], camera location [98], and calibration requirements [99] are commonly used in visual servo systems. Among them, PBVS and IBVS are usually employed in the tasks of PTL inspection. The former is based on position, and the latter is based on images. According to operation modes, inspection robots can be divided into climbing robots, UAVs, and hybrid platforms [94], as shown in Figure 6. Research on hybrid platforms is not yet mature, and visual servo control solutions are only used in the first two types of inspection robots.

(a)

(b)

(c)

4.1. Servo Control of Climbing Robots

The visual servo control of climbing robots is most applied to autonomous line-grasping control in the process of bypassing obstacles. And visual servo is less used in other aspects of control.

In terms of autonomous line grabbing, Wang et al. [100] designed a single-camera-based visual servo control method for line grasping. Although this method could meet the line-grasping requirements, it mainly depended on the mechanical structure of the robot. Zhang et al. [101] designed an IBVS controller for getting across obstacles. The image processing of this scheme was complicated, and it was sensitive to light changes. After that, Zhao et al. [102] proposed a means of combining remote control and IBVS. This method was effectively carried out in the laboratory environment. Guo et al. [103] raised an IBVS method for autonomous line-grasping control. This method used a 2D fuzzy controller. The final deflection error of a grasping line was within two degrees, and the time was less than forty seconds. Wang et al. [104] presented a line-grasping control based on hand-eye-vision. This method used a 2D fuzzy controller as shown in Figure 7. The impacts of changes in lighting, background, and other factors on the line-grasping accuracy were considered comprehensively. And it consumed twenty seconds. Aiming at a new dual-arm inspection robot designed by themselves, He et al. [105] proposed an adaptive visual servo method. This method could provide good characteristics of power lines without hand-eye calibration, and it improved the robustness of autonomously grasping the line.

The above-mentioned studies are aimed at the line-grasping problem of dual-arm PTL inspection robots, and many of them use IBVS. But the inspection robots only complete the autonomous line-grasping function in the laboratory environment, and they are not used in actual application scenarios. The problems of image noise, model errors, control delay, and environmental changes are not considered, which may be met in the actual scenarios. Besides, it takes too long for the inspection robots to grasp the line and the efficiency is low. Compared with traditional visual servo, uncalibrated visual servo has higher flexibility and adaptability, which is more suitable for inspection robots. Therefore, uncalibrated visual servo of inspection robots will be the future research direction.

In other ways, aiming at the problem of autodocking charging control for inspection robots, Wu et al. [107] used IBVS for precise positioning control. A variable universe fuzzy control was employed to overcome the difficulty of insufficient control precision. Besides, Jiang et al. [106] presented a double closed-loop autonomous localization control means. It originated from the BP network and visual servo. The BP network and visual servo were, respectively, used to solve the coarse and fine positioning of the manipulator. This method could effectually boost the automation level of inspection robots, but it took too long to complete the bolt alignment. The control architecture diagram of the method is shown in Figure 8.

Intelligent methods have begun to be used in the visual servo of inspection robots. However, they are not fully used in visual servo because of calculation drawbacks of intelligent means. Besides, many of the above-mentioned visual servo studies only concentrate on one task. However, there were many maintenance tasks on the PTL. The visual servo scheme based on single-task control has poor applicability, so it is necessary to focus on the research of multitask visual servo control.

4.2. Servo Control of UAVs

Compared with the visual servo control of climbing robots, the research scope of UAVs’ visual servo control is more limited. Currently, only visual servo control of tracking power lines is available.

Aiming at the problem of tracking power lines, Araar and Aouf [108] of Cranfield University proposed two solutions, which, respectively, were an IBVS method combined with the LQ-Servo and a partial PBVS (PPBVS) method. There was a better effect of PPBVS by contrast. However, this method was greatly affected by calibration errors. Xie et al. [109] raised a novel IBVS method. It could be robust to camera calibration errors without depth estimation. However, it was only in the simulation stage. Aiming at the problem of fixed-wing UAV tracking targets in the presence of wind, Mills et al. [110] designed an IBVS method. It considered the wind correction angle as the desired line angle. The result showed that it could boost the tracking response of UAVs. Rafique and Lynch [111] used IBVS combined with output feedback for the motion control of UAVs. Taking into account uncertain factors such as linear acceleration disturbance and quality, it adopted the inner-outer loop structure. The result showed that the stability of the structure is good. However, it was not tested in the outdoor PTL environment.

Many studies of visual servo control of UAVs are in the simulation stage. UAVs are notably influenced by environmental factors. The influence of wind on the motion control of UAVs needs to be considered in tracking power lines. Besides, compared to climbing robots, the inspection speed of UAVs is faster. It is necessary to boost response speed of servo during tracking. On the basis of tracking power lines, UAVs also need to perform path planning. So path planning for visual servo of UAVs is a good research direction.

5. Problems and Prospects

Many research institutions have started to study inspection robots globally. However, the research of inspection robots is only in the preliminary stage. Some key problems remain unsolved in terms of 3D reconstruction, object detection, and visual servo control. At the same time, the research of inspection robots in other areas still needs to develop.

5.1. Pivotal Technical Problems

It is effective to use inspection robots to replace manual inspection. And there are clear application requirements of PTL inspection. Hence, there is a very important engineering value to develop an inspection robot system that can perform efficient operations in a complex PTL environment. At present, studies concentrates on mechanical design, environmental perception, and visual servo control in the field of PTL inspection. Nevertheless, many of the results are mechanical design-related, and few results are given in other areas. Therefore, for boosting the efficiency and operation precision of inspection robots, the following problems need to be solved.

5.1.1. The Problem of Environmental Perception

Both the high-altitude environment and natural interference factors pose challenges to inspection robots. In this scenario, highly accurate environmental information is imperative for task conduction. The use of a single sensor will inevitably have perceptual limitations. For example, illumination variations affect the camera and fog affects the LiDAR. When power devices are aging or damaged, fault areas on the power devices can be hardly recognized with naked eyes, but heat or discharge will occur in the fault area. These phenomena can be detected by infrared thermal devices and UV cameras. Therefore, multisensor information fusion is indispensable for PTL inspection. As shown in Figure 9, it can be carried out from the following steps:

3D fusion reconstruction of transmission line corridors is used. Firstly, a rough 3D model of the transmission line corridor is constructed by oblique photography. The data registration is completed by the ICP algorithm to form a detailed 3D model. The fault areas are approximately found by the detailed 3D model.

The data fusion of infrared, visible light, and ultraviolet images is employed. After determining approximate fault areas and time synchronization, these sensors are used to shoot for image fusion. At present, image fusion still rests on the pixel stage. Feature fusion should be further considered for it.

The accurate coordinate of discharge, hotspot, and visible defects in fault areas are gained by object detection. The spatial conversion relations are gained by the calibration of the visible-light camera and LiDAR. For getting 3D information, the precise coordinate of the defect area is projected into point clouds according to spatial conversion relations. Then, the 3D information is used for local 3D reconstruction.

Visual servo control is applied. The current pose of the robot and objects can be obtained by local 3D reconstruction. After that, it can be provided to a 3D visual servo controller to complete the control.

5.1.2. The Problem of Datasets Required for Deep Learning

With the development of AI, a growing number of fields commence research by employing deep learning. Deep learning algorithms rely on data, but there is no standard power inspection dataset for researchers to conduct algorithm research. Therefore, the establishment of a standard power inspection dataset for deep learning development promotion is urgently needed. It can be constructed from the following steps:

Standards of inspection images and power devices are made. Aiming at different defects of power devices, a unified defect description and classification rules should be established. The size and sharpness of inspection images should be normalized. Standards provide convenience when instructing personnel in data collection and highly improve the dataset construction.

Datasets of PTL inspection should be divided according to various regions. The geographical environment and weather conditions of each region are different. PTL will be affected by a distinct climate, which makes the type of the same power device be disparate. Different types of power devices may vary greatly in appearance and shape, which results in regional characteristics. Therefore, there are regional characteristics in PTL corridors. In addition, the background of various regions differs greatly. Hence, it is necessary to divide the whole dataset first according to the territories.

According to different kinds of power devices and degrees of defects, datasets of regional PTL inspection should be further divided into different subset datasets. According to specific power devices in PTL, it is helpful to form the corresponding PTL dataset by subset datasets, which can represent the characteristics greatly and be helpful for training models. The steps are shown in Figure 10.

Now, a good deep learning model is always based on high-quality annotated data. However, it is expensive to collect data, especially annotated data. In the field of PTL inspection, this phenomenon is more obvious. Consequently, researchers try to employ weakly supervised data to train models [112, 113]. Furthermore, they employ unsupervised learning to annotate data. These methods can be also used in object detection of PTL inspection [114] for reducing the cost of data annotation. It will be a good research direction of PTL inspection.

5.1.3. The Problem of Applying a Light Network Model Suitable for Embedded Platforms

The training of models usually runs on servers. The computational ability should be considered when deep learning-based object algorithms are used. Therefore, setting up a light-weight network in advance is significant. Then the model, transferred from the server to the embedded platform, can be additionally trained in actual scenarios. At present, different backbones such as SSD and YOLOv4 can be used in MobileNetv3. Furthermore, fine-tuning can also be employed.

5.1.4. The Problem of Control Efficiency and Robustness of the Visual Servo System

Now, the whole control process of grasping the line or tracking lines takes much time. And the efficiency is low. In this regard, improving visual sampling frequency and image processing speed is able to ameliorate its real-time capability. In addition, subtle structural changes of inspection robots may occur due to wind and drag of vibration dampers, which leads to errors. The uncalibrated visual servo has a high tolerance for these errors. Therefore, it can be adopted for improving the robustness.

5.2. Prospects

In the application scenarios of PTL inspection, not only the foundation capabilities of inspection robots but also further expansion should be realized according to requirements. It is helpful to realize the vital engineering value of inspection robots. The facets of future directions are as follows:

5.2.1. The Development of Mechanical Design

The core of inspection robots is the mechanical design. And a good one can greatly improve the efficiency. For example, compared to LineScout [115], LineRanger [116] takes much less time to cross obstacles because of the ingenious and passive obstacle-crossing mechanism. However, the mechanical design of PTL inspection robots is not refined now, and robots cannot balance inspection and operation. So far, in contrast to other inspection robots, climbing robots are more practical due to their in-contact measurements. The engineering value of climbing robots is greater when they can succeed in crossing pylons. So the mechanical design of inspection robots still needs to be improved.

5.2.2. The Development of Energy Harvesting

Battery capacities directly influence the duration of PTL inspection. Regarding a certain load capacity of robots, a high-capacity battery, such as lithium battery, can increase maximum endurance of robots, but it limits the weight of other mechanical components. So online charging is important. For increasing the inspection endurance, inspection robots can acquire electric energy from power line autonomously. So an induction power supply system should be developed.

5.2.3. The Development of Electromagnetic Protection

EMI on inspection robots is more intense with increasing voltage, especially the transient discharge process of the equipotential working. However, the present inspection robots are weak to EMI. Therefore, it is an inevitable problem during the inspection. The entire robot should conduct electricity for avoiding electrical discharge within components of robots. Shielding is an effective protection measurement aimed at EMI, and high-quality shields should be designed. However, inspection robots cannot be fully shielded, which means that other optimization technologies, such as routing, filtering, and grounding techniques, should be considered. Consequently, how to design electromagnetic protection is a crucial challenge.

5.2.4. The Development of Modularity

It is inevitable that inspection robots will be damaged in a bad working environment. And the modularity development of inspection robots is significant for rapid maintenance. The modularity of inspection robot is convenient for the maintainer to find the fault location and replace the fault module of robots. Moreover, it is easy to install and disassemble inspection robots by modularity, which also improves the efficiency of online and offline switchover. Furthermore, it helps to switch robot configurations in different inspection scenarios, such as overhead ground wire and multibundled conductors.

5.2.5. The Development of Reliability

The working environment of inspection robots is harsh, such as the power line area with long span and steep slope. There is a requirement for good robustness to accurately complete the task. Consequently, inspection robots should have the ability to cross the most complex area in PTL. For the purpose of satisfying the needs of multiple and complicated tasks, effective load capacity and operation control technology should be improved. What is more, before online operation, robots should be tested for endurance, such as wear of the mechanical system, battery performance, and EMI robustness.

5.2.6. The Development of Cooperation

This cooperation not only refers to the cooperation among a variety of inspection robots. It also includes the cooperation between inspection robots and the ground control system (GCS). The multirobot system can promote the performance of PTL inspection. In addition, in the case of teleoperation, the efficiency of inspection can be also affected by the cooperation between inspection robots and GCS. Appropriate deployments of GCS points in a long PTL corridor are the key to the cooperation between inspection robots and GCS. The cooperation includes real-time bidirectional data transmission, the energy consumption of disassembly, installation, and teleoperation.

5.2.7. The Development of Friendly Operability and Autonomy

The final goal is to achieve autonomous patrol of inspection robots. However, autonomous control and teleoperation need to coexist in the process of development. Although inspection robots could automatically get across obstacles such as spacers and suspension clamps, there is a lack of autonomous ability in specific operations. The specific operations include dealing with damaged power lines and removing foreign bodies. For fulfilling tasks securely and briskly, teleoperation should be used. Meanwhile, the friendly man-machine interface of inspection robots should be developed for simple teleoperation.

6. Conclusion

For promoting researchers to develop advanced research of PTL inspection, studies of 3D reconstruction, object detection, and visual servo of PTL inspection are summarized. The extant problems, such as perception limitation and lack of datasets, are correspondingly discussed. What is more, several aspects are posed, which are promising research directions in the field of PTL inspection. In a word, the present perception technologies and control technologies cannot meet the requirement of automatic and accurate power line inspection. With the emergence of these research results, the automation degree of inspection robots will be highly boosted.

Conflicts of Interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Acknowledgments

This project was supported by the National Key Research and Development Program of China (Grant 2018YFB1307400) and the National Natural Science Foundation of China (Grant 61873267).

References

N. Pouliot, P. L. Richard, and S. Montambault, “LineScout technology opens the way to robotic inspection and maintenance of high-voltage power lines,” IEEE Power and Energy Technology Systems Journal, vol. 2, no. 1, pp. 1–11, 2015.
View at: Publisher Site | Google Scholar
C. Zhang, Z. Liu, S. Yang, and B. Xu, “Key technologies of laser point cloud data processing in power line corridor,” in Conference on LIDAR Imaging Detection and Target Recognition, Changchun, China, 2017.
View at: Google Scholar
X. Mai, C. Chen, and X. Peng, “3D visualization technique of transmission line corridors: system design and implementation,” Electric Power, vol. 48, no. 2, pp. 98–103, 2015.
View at: Google Scholar
B. Chen and X. Miao, “Distribution line pole detection and counting based on YOLO using UAV inspection line video,” Journal of Electrical Engineering and Technology, vol. 15, no. 1, pp. 441–448, 2020.
View at: Google Scholar
F. Gao, J. Wang, Z. Kong et al., “Recognition of insulator explosion based on deep learning,” in 14th IEEE International Computer Conference on Wavelet Active Media Technology and Information Processing, pp. 79–82, Chengdu, China, 2017.
View at: Google Scholar
Z. Zhao, A. Jiang, Y. Qi, W. Zhang, and W. Zhao, “Fittings detection in transmission line images with SSD model embedded occlusion relation module,” CAAI Transactions on Intelligent Systems, vol. 15, no. 4, pp. 1–7, 2020.
View at: Google Scholar
K. Toussaint, N. Pouliot, and S. Montambault, “Transmission line maintenance robots capable of crossing obstacles: state-of-the-art review and challenges ahead,” Journal of Field Robotics, vol. 26, no. 5, pp. 477–499, 2009.
View at: Google Scholar
S. Montambault and N. Pouliot, “About the future of power line robotics,” in 1st International Conference on Applied Robotics for the Power Industry, pp. 1–6, Montreal, QC, Canada, 2010.
View at: Google Scholar
K. Kraus and N. Pfeifer, “Determination of terrain models in wooded areas with airborne laser scanner data,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 53, no. 4, pp. 193–203, 1998.
View at: Publisher Site | Google Scholar
W. Boehler, M. Vicent, and A. Marbs, “Investigating laser scanner accuracy,” The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 34, no. 5, pp. 696–701, 2003.
View at: Google Scholar
W. Goebel, B. M. Kampa, and F. Helmchen, “Imaging cellular network dynamics in three dimensions using fast 3d laser scanning,” Nature Methods, vol. 4, no. 1, pp. 73–79, 2007.
View at: Publisher Site | Google Scholar
M. C. Amann, T. Bosch, and M. Lescure, “Laser ranging: a critical review of usual techniques for distance measurement,” Optical Engineering, vol. 40, no. 1, pp. 10–19, 2001.
View at: Google Scholar
S. May, D. Droeschel, D. Holz, C. Wiesen, and S. Fuchs, “3d pose estimation and mapping with time-of-flight cameras,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, France, 2008.
View at: Google Scholar
S. Zhu and X. Qiang, “Analysis of 3-d coordinate vision measuring methods with feature points on workpiece,” Optics and Precision Engineering, vol. 8, no. 2, pp. 192–197, 2000.
View at: Google Scholar
D. Geng, Y. He, and H. Su, “Study on the measurement of transparent step by white-light interferometer,” Optical Instruments, vol. 35, no. 6, pp. 74–77, 2013.
View at: Google Scholar
L. W. Bingleman and G. S. Schajer, “DIC-based surface motion correction for ESPI measurements,” Experimental Mechanics, vol. 51, no. 7, pp. 1207–1216, 2011.
View at: Publisher Site | Google Scholar
L. Sun and Y. Yu, “Transient 3D deformation measurement method by color splitting based on phase shift and ESPI,” Optical Instruments, vol. 38, no. 1, pp. 20–26, 2016.
View at: Google Scholar
M. R. Oswald, E. Töppe, C. Nieuwenhuis, and D. Cremers, “A review of geometry recovery from a single image focusing on curved object reconstruction,” in Innovations for Shape Analysis, Springer, Heidelberg, Berlin, 2013.
View at: Google Scholar
J. Chen, Y. Zhang, P. Song, Y. Wei, and Y. Wang, “Application of deep learning to 3d object reconstruction from a single image,” Acta Automatica Sinica, vol. 45, no. 4, pp. 657–668, 2019.
View at: Google Scholar
M. Sizintsev and R. P. Wildes, “Spacetime stereo and 3d flow via binocular spatiotemporal orientation analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 11, pp. 2241–2254, 2014.
View at: Publisher Site | Google Scholar
F. Qi, D. Zhao, and W. Gao, “Reduced reference stereoscopic image quality assessment based on binocular perceptual information,” IEEE Transactions on Multimedia, vol. 17, no. 12, pp. 2338–2344, 2015.
View at: Publisher Site | Google Scholar
C. Stewart and C. Dyer, “The trinocular general support algorithm: a three-camera stereo algorithm for overcoming binocular matching errors,” in 1988 Second International Conference on Computer Vision, pp. 134–138, Los Alamitos, USA, 1988.
View at: Google Scholar
M. Goesele, N. Snavely, B. Curless, H. Hoppe, and S. M. Seitz, “Multi-view stereo for community photo collections,” in 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8, Rio de Janeiro, Brazil, 2007.
View at: Google Scholar
D. Yang, “Digital terrain model,” Bulletin of Surveying and Mapping, vol. 3, pp. 3–5, 1998.
View at: Google Scholar
P. Axelsson, “Dem generation from laser scanner data using adaptive tin models,” International Archives of Photogrammetry and Remote Sensing, vol. 33, pp. 110–117, 2000.
View at: Google Scholar
J. Yu, C. Mu, Y. Feng, and Y. Dou, “Powerlines extraction techniques from airborne LiDAR data,” Geomatics and Information Science of Wuhan University, vol. 36, no. 11, pp. 1275–1279, 2011.
View at: Google Scholar
G. Vosselman, “Slope based filtering of laser altimetry data,” International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 33, pp. 935–942, 2000.
View at: Google Scholar
K. Zhang, S. C. Chen, D. Whitman, M. L. SHyu, J. Yan, and C. Zhang, “A progressive morphological filter for removing nonground measurements from airborne LiDAR data,” IEEE Transactions on Geoscience and Remote Sensing, vol. 41, no. 4, pp. 872–882, 2003.
View at: Publisher Site | Google Scholar
Z. Liu, J. Liang, and J. Zhang, “Power lines extraction from airborne LiDAR data using spatial domain segmentation,” Journal of Remote Sensing, vol. 18, no. 1, pp. 61–76, 2014.
View at: Google Scholar
X. Shen, C. Qin, Y. Du, and X. Yu, “An automatic power line extraction method from airborne light detection and ranging point cloud in complex terrain,” Journal of Tongji University. Natural Science, vol. 46, no. 7, pp. 982–987, 2018.
View at: Google Scholar
R. A. McLaughlin, “Extracting transmission lines from airborne LiDAR data,” IEEE Geoscience and Remote Sensing Letters, vol. 3, no. 2, pp. 222–226, 2006.
View at: Publisher Site | Google Scholar
Y. Jwa and G. Sohn, “A piecewise catenary curve model growing for 3d power line reconstruction,” Photogrammetric Engineering and Remote Sensing, vol. 78, no. 12, pp. 1227–1240, 2012.
View at: Publisher Site | Google Scholar
J. Wu, L. Chen, L. Li, J. Yang, and X. Liang, “Power line extraction and reconstruction from airborne LiDAR point cloud,” Laser Technology, vol. 43, no. 4, pp. 500–505, 2019.
View at: Google Scholar
T. Melzer and C. Briese, “Extraction and modeling of power lines from ALS abstract: point clouds,” 28th Workshop of Austrian Association for Pattern Recognition, pp. 47–54, 2004.
View at: Google Scholar
X. Lin, M. Duan, J. Zhang, and Y. Zang, “A method of reconstructing 3d powerlines from airborne LiDAR point clouds,” Science of Surveying and Mapping, vol. 41, no. 1, pp. 109–114,64, 2016.
View at: Google Scholar
X. Lai, D. Dai, M. Zheng, and Y. Du, “Powerline three-dimensional reconstruction for LiDAR point cloud data,” Journal of Remote Sensing, vol. 18, no. 6, pp. 1223–1229, 2014.
View at: Google Scholar
X. Lin and J. Zhang, “3d power line reconstruction from airborne LiDAR point cloud of overhead electric power transmission corridors,” Acta Geodetica et Cartographica Sinica, vol. 45, no. 3, pp. 347–353, 2016.
View at: Google Scholar
L. Cheng, L. Tong, Y. Wang, and M. Li, “Extraction of urban power lines from vehicle-borne LiDAR data,” Remote Sensing, vol. 6, no. 4, pp. 3302–3320, 2014.
View at: Publisher Site | Google Scholar
J. Liang, J. Zhang, K. Deng, Z. Liu, and Q. Shi, “A new power-line extraction method based on airborne LiDAR point cloud data,” in 2011 International Symposium on Image and Data Fusion, pp. 1–4, Tengchong, China, 2011.
View at: Google Scholar
J. Zhang, M. Duan, X. Lin, and Y. Zang, “Comparison and analysis of models for 3d power line reconstruction using LiDAR point cloud,” Geomatics and Information Science of Wuhan University, vol. 42, no. 11, pp. 1565–1572, 2017.
View at: Google Scholar
W. Zhang, G. Yan, N. Wang, Q. Li, and W. Zhao, “Automatic 3d power line reconstruction of multi-angular imaging power line inspection system,” Proceedings of SPIE - The International Society for Optical Engineering, vol. 6752, no. 1, pp. 10–19, 2007.
View at: Google Scholar
F. Ganovelli, L. Malomo, and R. Scopigno, “Reconstructing power lines from images,” in 2018 International Conference on Image and Vision Computing New Zealand, pp. 1–6, Auckland, New Zealand, 2018.
View at: Google Scholar
M. Maurer, M. Hofer, F. Fraundorfer, and H. Bischof, “Automated inspection of power line corridors to measure vegetation undercut using UAV-based images,” in International Conference on Unmanned Aerial Vehicles in Geomatics, pp. 33–40, Bonn, Germany, 2017.
View at: Google Scholar
Z. Chen, Z. Lan, H. Long, and Q. Hu, “3d modeling of pylon from airborne lidar data,” in Remote Sensing of the Environment: 18th National Symposium on Remote Sensing of China, vol. 9158, International Society for Optics and Photonics.
View at: Google Scholar
B. Guo, X. Huang, Q. Li, F. Zhang, J. Zhu, and C. Wang, “A stochastic geometry method for pylon reconstruction from airborne LiDAR data,” Remote Sensing, vol. 8, no. 3, p. 243, 2016.
View at: Publisher Site | Google Scholar
L. Xi, H. Zhao, S. Wang, X. Zhang, and M. Ma, “The design of unmanned aerial vehicle power patrol base on oblique photography technology,” Electronic Science and Technology, vol. 32, no. 5, pp. 89–92, 2019.
View at: Google Scholar
H. Pei, S. Jiang, G. Lin, H. Huang, W. Jiang, and C. Yang, “3d reconstruction of transmission route based on UAV oblique photogrammetry,” Science of Surveying and Mapping, vol. 41, no. 12, pp. 292–296, 2016.
View at: Google Scholar
Z. Zou, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: a survey,” 2019.
View at: Google Scholar
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587, Columbus, OH, 2014.
View at: Google Scholar
J. Platt, “Sequential minimal optimization: a fast algorithm for training support vector machines,” Advances in Kernel Methods-Support Vector Learning, vol. 208, 1998.
View at: Google Scholar
Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” Journal of Computer and System Sciences, vol. 55, no. 1, pp. 119–139, 1997.
View at: Publisher Site | Google Scholar
A. Neubeck and L. Van Gool, “Efficient non-maximum suppression,” in 18th International Conference on Pattern Recognition, vol. 3, pp. 850–855, Hong Kong, China, 2006.
View at: Google Scholar
D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004.
View at: Publisher Site | Google Scholar
H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (surf),” Computer Vision and Image Understanding, vol. 110, no. 3, pp. 346–359, 2008.
View at: Publisher Site | Google Scholar
T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 7, pp. 971–987, 2002.
View at: Publisher Site | Google Scholar
N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893, San Diego, CA, 2005.
View at: Google Scholar
C. Sampedro, C. Martinez, A. Chauhan, and P. Campoy, “A supervised approach to electric tower detection and classification for power line inspection,” in 2014 International Joint Conference on Neural Networks, pp. 1970–1977, Beijing, China, 2014.
View at: Google Scholar
A. Cerón, I. Mondragón, and F. Prieto, “Real-time transmission tower detection from video based on a feature descriptor,” IET Computer Vision, vol. 11, no. 1, pp. 33–42, 2017.
View at: Google Scholar
X. Wang, D. Li, and L. Zhang, “A supervised approach to electric tower detection for power line inspection,” Northeast Electric Power Technology, vol. 38, no. 11, pp. 12–14,19, 2017.
View at: Google Scholar
Z. Wang, J. Han, X. Sun, and B. Yang, “Method for orientation determination of transmission line tower based on visual navigation,” Laser and Optoelectronics Progress, vol. 56, no. 8, 2019.
View at: Google Scholar
Y. Zhang, Y. Chen, D. Wang, Z. Qian, and C. Ma, “Coarse-to-fine detection for nests on pylon,” Information Technology, vol. 3, pp. 104–109, 2017.
View at: Google Scholar
J. Xu, J. Han, Z. Tong, and Y. Wang, “Method for detecting bird's nest on tower based on UAV image,” Computer Engineering and Application, vol. 53, no. 6, pp. 231–235, 2017.
View at: Google Scholar
Z. Zhao, N. Liu, and Y. Yuan, “The recognition and localization of insulators based on SIFT and RANSAC,” in Proceedings of 3rd International Conference on Multimedia Technology, pp. 692–699, Guangzhou, China, 2013.
View at: Google Scholar
Z. Zhao and N. Liu, “The recognition and localization of insulators adopting SURF and IFS based on correlation coefficient,” Optik, vol. 125, no. 20, pp. 6049–6052, 2014.
View at: Publisher Site | Google Scholar
P. S. Prasad and B. P. Rao, “LBP-HF features and machine learning applied for automated monitoring of insulators for overhead power distribution lines,” in 2016 International Conference on Wireless Communications, Signal Processing and Networking, pp. 808–812, Chennai, India, 2016.
View at: Google Scholar
F. Zhang, R. Guo, Z. Cheng et al., “Detection for transmission line obstacles based on principal component gradient histogram,” Computer Engineering and Application, vol. 52, no. 15, pp. 254–259,265, 2016.
View at: Google Scholar
D. Zhang, X. Qiu, C. Cao, and J. Zhu, “An vibration damper detection algorithm combined with aggregation channel and complex frequency domain features,” Computer Technology and Development, vol. 30, no. 3, pp. 147–151, 2020.
View at: Google Scholar
M. Feng, W. Luo, L. Yu et al., “A bolt detection method for pictures captured from an unmanned aerial vehicle in power transmission line inspection,” Journal of Electric Power Science and Technology, vol. 33, no. 4, pp. 135–140, 2018.
View at: Google Scholar
S. Fan, D. Yang, D. Zou, and Y. Yan, “Vision-based tracing, recognition and positioning strategy for bolt tightening live working robot on power transmission line,” Journal of Electronic Measurement and Instrumentation, vol. 31, no. 9, pp. 1514–1523, 2017.
View at: Google Scholar
Z. Zhao, H. Qi, and L. Nie, “Research overview on visual detection of transmission lines based on deep learning,” Guangdong Electric Power, vol. 32, no. 9, pp. 11–23, 2019.
View at: Google Scholar
R. Girshick, “Fast R-CNN,” in 2015 IEEE International Conference on Computer Vision, pp. 1440–1448, Santiago, Chile, 2015.
View at: Google Scholar
S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137–1149, 2017.
View at: Publisher Site | Google Scholar
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: unified, real-time object detection,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788, Seattle, WA, USA, 2016.
View at: Google Scholar
J. Redmon and A. Farhadi, “YOLO9000: better, faster, stronger,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 6517–6525, Honolulu, HI, USA, 2017.
View at: Google Scholar
J. Redmon and A. Farhadi, “YOLOv3: an incremental improvement,” 2018.
View at: Google Scholar
A. Bochkovskiy, C. Y. Wang, and H. Y. Liao, “YOLOv4: optimal speed and accuracy of object detection,” 2020.
View at: Google Scholar
J. Guo, B. Chen, R. Wang, J. Wang, and L. Zhong, “YOLO-based real-time detection of power line poles from unmanned aerial vehicle inspection vision,” Electric Power, vol. 52, no. 7, pp. 17–23, 2019.
View at: Google Scholar
H. Wang, G. Yang, E. Li, Y. Tian, M. Zhao, and Z. Liang, “High-voltage power transmission tower detection based on Faster R-CNN and YOLO-V3,” in 2019 Chinese Control Conference, pp. 8750–8755, Guangzhou, China, 2019.
View at: Google Scholar
L. Shi, H. Yang, Z. Zhou, L. Yang, H. Zhang, and H. Du, “Intelligent detection of bird's nest based on RetinaNet model,” Power Systems and Big Data, vol. 23, no. 2, pp. 53–58, 2020.
View at: Google Scholar
T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 318–327, 2020.
View at: Publisher Site | Google Scholar
J. Wang, H. Luo, P. Yu, L. Zheng, and F. Hu, “Bird's nest detection in multi-scale of high-voltage tower based on Faster R-CNN,” Journal of Beijing Jiaotong University, vol. 43, no. 5, pp. 37–43, 2019.
View at: Google Scholar
X. Liu, H. Jiang, J. Chen, J. Chen, S. Zhuang, and X. Miao, “Insulator detection in aerial images based on faster regions with convolutional neural network,” in 14th International Conference on Control and Automation, pp. 1082–1086, Anchorage, AK, USA, 2018.
View at: Google Scholar
Z. Zhao, Z. Zhen, L. Zhang, Y. Qi, Y. Kong, and K. Zhang, “Insulator detection method in inspection image based on improved Faster R-CNN,” Energies, vol. 12, no. 7, 2019.
View at: Google Scholar
J. Han, Z. Yang, Q. Zhang et al., “A method of insulator faults detection in aerial images for high-voltage transmission lines inspection,” Applied Sciences, vol. 9, no. 10, 2019.
View at: Google Scholar
G. Lin, B. Wang, H. Peng, X. Wang, S. Chen, and L. Zhang, “Multi-target detection and location of transmission line inspection image based on improved faster-RCNN,” Electric Power Automation Equipment, vol. 39, no. 5, pp. 213–218, 2019.
View at: Google Scholar
S. Dong, “Real-time detection of power transmission line key components based on YOLOv3,” Electronic Measurement Technology, vol. 42, no. 23, pp. 173–178, 2019.
View at: Google Scholar
G. Yang, C. Sun, N. Zhang et al., “Detection of key components of transmission lines based on multi-scale feature fusion,” Electric Measurement and Instrumentation, vol. 57, no. 3, pp. 54–59, 2020.
View at: Google Scholar
G. Yang, C. Sun, D. Wang et al., “Comparative study of transmission line component detection models based on UAV front end and SSD algorithm,” Journal of Taiyuan University of Technology, vol. 51, no. 2, pp. 212–219, 2020.
View at: Google Scholar
W. Liu, D. Anguelov, D. Erhan et al., “SSD: single shot multibox detector,” in 14th European Conference on Computer Vision, pp. 21–37, Cham, 2016.
View at: Google Scholar
Y. Qi, A. Jiang, Z. Zhao, J. Lang, and L. Nie, “Fittings detection method in patrol images of transmission line based on improved SSD,” Electric Measurement and Instrumentation, vol. 56, no. 22, pp. 7–12,43, 2019.
View at: Google Scholar
S. Liu, B. Wang, K. Gao, Y. Wang, C. Gao, and J. Chen, “Object detection method for aerial inspection image based on region-based fully convolutional network,” Automation of Electric Power Systems, vol. 43, no. 13, pp. 162–168, 2019.
View at: Google Scholar
B. Singh and L. S. Davis, “An analysis of scale invariance in object detection - SNIP,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3578–3587, Salt Lake City, UT, USA, 2018.
View at: Google Scholar
Y. Li, Y. Chen, N. Wang, and Z. Zhang, “Scale-aware trident networks for object detection,” in 2019 IEEE/CVF International Conference on Computer Vision, pp. 6053–6062, Seoul, South Korea, 2019.
View at: Google Scholar
A. B. Alhassan, X. Zhang, H. Shen, and H. Xu, “Power transmission line inspection robots: a review, trends and challenges for future research,” International Journal of Electrical Power and Energy Systems, vol. 118, 2020.
View at: Google Scholar
S. Hutchinson, G. D. Hager, and P. I. Corke, “A tutorial on visual servo control,” IEEE Transactions on Robotics and Automation, vol. 12, no. 5, pp. 651–670, 1996.
View at: Publisher Site | Google Scholar
E. Malis, F. Chaumette, and S. Boudet, “2 1/2 D visual servoing,” IEEE Transactions on Robotics and Automation, vol. 15, no. 2, pp. 238–250, 1999.
View at: Publisher Site | Google Scholar
F. Chaumette and E. Malis, “2-1/2-D visual servoing: a possible solution to improve image-based and position-based visual servoings,” in IEEE International Conference on Robotics and Automation, pp. 630–635, San Francisco, CA, USA, 2000.
View at: Google Scholar
Q. Zhao, G. Lian, and Z. Sun, “Survey of robot visual servoing,” Control and Decision, vol. 16, no. 6, pp. 849–853, 2001.
View at: Google Scholar
B. Tao, Z. Gong, and H. Ding, “Survey on uncalibrated robot visual servoing control,” Chinese Journal of Theoretical and Applied Mechanics, vol. 48, no. 4, pp. 767–783, 2016.
View at: Google Scholar
L. Wang, H. Wang, L. Fang, and M. Zhao, “Visual-servo-based line-grasping control for power transmission line inspection robot,” Robot, vol. 29, no. 5, pp. 451–455, 2007.
View at: Google Scholar
Y. Zhang, Z. Liang, M. Tan, W. Ye, and B. Lian, “Visual servo control of obstacle negotiation for overhead power line inspection robot,” Robot, vol. 29, no. 2, pp. 111–116, 2007.
View at: Google Scholar
D. Zhao, G. Yang, E. Li, and Z. Liang, “Design and its visual servoing control of an inspection robot for power transmission lines,” in 2013 IEEE International Conference on Robotics and Biomimetics, pp. 546–551, Shenzhen, China, 2013.
View at: Google Scholar
W. Guo, H. Wang, Y. Jiang, and P. Sun, “Visual servo control for automatic line-grasping of a power transmission line inspection robot,” Robot, vol. 34, no. 5, pp. 620–627, 2012.
View at: Publisher Site | Google Scholar
W. Wang, G. Wu, Y. Bai et al., “Hand-eye-vision based control for an inspection robot's autonomous line grasping,” Journal of Central South University, vol. 21, no. 6, pp. 2216–2227, 2014.
View at: Publisher Site | Google Scholar
T. He, H. Wang, W. Chen, and W. Wang, “Visual servoing of a new designed inspection robot for autonomous transmission line grasping,” in International Conference on Wearable Sensors and Robots, pp. 553–569, Singapore, 2017.
View at: Google Scholar
W. Jiang, Z. Zhou, W. Chen, L. Yu, H. Li, and G. Wu, “Manipulator double close loop autonomous localization control of high-voltage cable mobile operation robot,” Transactions of Beijing Institute of Technology, vol. 39, no. 6, pp. 589–596, 2019.
View at: Google Scholar
G. Wu, Z. Yang, W. Wang, L. Guo, J. Hu, and P. Zhou, “On auto-docking charging control method for the inspection robot,” Journal of Harbin Institute of Technology, vol. 48, no. 7, pp. 123–129, 2016.
View at: Google Scholar
O. Araar and N. Aouf, “Visual servoing of a quadrotor UAV for autonomous power lines inspection,” in 22nd Mediterranean Conference on Control and Automation, pp. 1418–1424, Palermo, Italy, 2014.
View at: Google Scholar
H. Xie, A. Lynch, and M. Jagersand, “IBVS of a rotary wing UAV using line features,” in IEEE 27th Canadian Conference on Electrical and Computer Engineering, pp. 1–6, Toronto, Canada, 2014.
View at: Google Scholar
S. Mills, N. Aouf, and L. Mejias, “Image based visual servo control for fixed wing UAVs tracking linear infrastructure in wind,” in 2013 IEEE International Conference on Robotics and Automation, pp. 5769–5774, Karlsruhe, Germany, 2013.
View at: Google Scholar
M. A. Rafique and A. F. Lynch, “Output-feedback image-based visual servoing for multirotor unmanned aerial vehicle line following,” IEEE Transactions on Aerospace and Electronic Systems, vol. 56, no. 4, pp. 3182–3196, 2020.
View at: Google Scholar
H. Bilen and A. Vedaldi, “Weakly supervised deep detection networks,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2846–2854, Seattle, WA, 2016.
View at: Google Scholar
P. Tang, X. Wang, S. Bai et al., “PCL: proposal cluster learning for weakly supervised object detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 1, pp. 176–191, 2020.
View at: Publisher Site | Google Scholar
C. Ge, J. Wang, J. Y. Wang, Q. Qi, H. F. Sun, and J. X. Liao, “Towards automatic visual inspection: a weakly supervised learning method for industrial applicable object detection,” Computers in Industry, vol. 121, p. 103232, 2020.
View at: Publisher Site | Google Scholar
S. Montambault and N. Pouliot, “Design and validation of a mobile robot for power line inspection and maintenance,” in 6th International Conference on Field and Service Robotics, Chamonix, France, 2007.
View at: Google Scholar
P. L. Richard, N. Pouliot, F. Morin et al., “LineRanger: analysis and field testing of an innovative robot for efficient assessment of bundled high-voltage powerlines,” in 2019 International Conference on Robotics and Automation, pp. 9130–9136, Montreal, QC, Canada, 2019.
View at: Google Scholar

Copyright

Copyright © 2021 Minghao Chen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

1182

Downloads

1240

Citations