Abstract
Real-time load detection method for belt conveyors based on computer vision is the research topic of this paper. A belt conveyor system equipped with cameras and a laser generator is used as the test apparatus. As the basis for conveyor intelligent speed regulation, two methods from different angles to perceive the load of conveyor belt were proposed, applied, and compared in this paper. Method 1 is based on the area proportion and method 2 is the detection based on laser-based computer vision technology. Laboratory experiments show that both methods can well detect the load on the conveyor belt. Method 2 is more economical and practical under the background of existing technology, also compared to the method 1, which provides a new idea and theoretical basis for the energy-saving control and intelligent development of the conveyor.
1. Introduction
Belt conveyor is developing towards intelligence, long distance, and large capacity, while its energy consumption has attracted more and more attention, especially the problem of unnecessary energy waste. The causes of the energy waste of conveyors can be summarized into two aspects: one is caused by the preliminary selection design of the belt conveyor, and the other is caused by the control factors.
In the preliminary selection of the belt conveyor design stage, the equipment production capacity of a belt conveyor is usually determined according to a certain multiple M (M > 1) of the maximum production capacity [1]. The redundancy of such capacity not only ensures the production capacity but also causes energy waste to a certain extent. The unnecessary energy loss caused by complex driving devices is also an important source, such as the loss from inefficient motors and gearbox. Last but not least, the compression resistance of the conveyor belt is an important source of energy waste of conveyor belt, which is closely related to the type of conveyor belt, the designed speed, and the spacing of the rollers. Once the conveyor is designed and installed, it is difficult to adjust these to achieve energy saving.
In the production process, the energy waste of conveyor mainly comes from the contradiction between transport continuity and production discontinuity, which means the conveyor is always running at full speed, regardless of the amount of load or no load on the belt, especially in the field of the coal mine.
Some scholars have proposed using the new transmission device to achieve the purpose of energy saving, such as the application of permanent magnet synchronous motor (PMSM) [2–4] and low resistance roller [5].
Many scholars have pointed out that the speed regulation of conveyor is one of the important means to realize energy saving. Reference [6] presented a fuzzy control method to optimize the speed control of belt conveyors with respect to reducing the power required, when effectively avoiding the interference with speed control caused by extreme material loading situations and inappropriate belt speed changes. The power consumption model of the belt conveyor system including motor, frequency converter, conveyor operating resistance, and other factors was derived in [7], and an energy-saving control strategy for the belt conveyor system based on material flow was proposed. A three-step speed adjustment strategy called Estimation-Computation-Optimization (ECO) was proposed in [8] to determine the minimum speed adjustment time, which improves the applicability of conveyor speed control, taking into account the dynamic characteristics of the conveyor belt. An energy-saving optimization model of belt conveyor coal flow, belt running speed, and system power based on BP neural network was established in [9]. An active control algorithm was presented in [10] for energy savings between multistage conveyor systems.
In addition to the improvements to conveyor speed control strategies, many scholars pointed out that the basis for intelligent speed regulation of belt conveyors should be the real-time load of the conveyor [11–13]. At present, three methods are commonly applied in detecting load distribution: (1) electronic belt scale, (2) the nuclear scale, and (3) ultrasonic rangefinder. These three methods will be phased out because of their poor accuracy and/or radiation. Some new measurement methods have also emerged in recent years: a laser scanning-based conveyor belt flow measurement method is proposed in [14] for estimating the flow rate by acquiring a three-dimensional point cloud of material on the conveyor belt in real time using a 3D laser scanner. Reference [15] proposed a method to measure the pressure distribution directly on a running conveyor belt by using a tactile pressure sensor.
With the development of computer vision technology, it is increasingly used for the detection of conveyor operating conditions, especially in the field of belt tear detection, belt speed detection, and other aspects. The study in [16] proposed a general machine vision approach for online estimation of rock mixture composition and illustrated it on a very challenging nickel mineral system with very heterogeneous minerals and similar coloration, and rock fragments can be dry or wet; good results were obtained for dry minerals through a pilot plant conveyor belt application. A multispectral visual detection method for conveyor belt longitudinal tear was proposed in [17] to solve the drawbacks of the existing conveyor belt longitudinal tear detection technology, and its reliability and real time were verified to meet the requirements in the industrial field. The study in [18] developed a new method of belt edge detection based on the deep convolution network used in the industrial scene of the coal transport belt, addressing the problems that traditional mechanical antideviation treatment is not timely and the edge detection of the conveyor belt with machine vision is imprecise. An improved edge detection algorithm based on Canny operator and morphology processing and a belt positioning algorithm based on Hough line detection used for belt positioning were proposed in [19]. A belt conveyor deviation detection system based on computer vision was proposed in [20] to realize the visualization of belt deviation.
Motivated by the above observations, researchers have generally focused on speed control strategies for conveyors rather than the measurement of real-time conveyor loads. With the development of intelligence conveyor and computer vision technology, two methods based on computer vision technology from different angles in this paper to perceive the load of conveyor belt were proposed, applied, and compared, with the advantages of affordable and noncontact measurement, which can be used as a basis for intelligent speed regulation of conveyors.
Section 2 explains the experimental setup, the methodology is explained in Section 3, results are presented in Section 4, and discussion is in Section 5; then the conclusion is highlighted in Section 6.
2. Experimental Setup
Experimental data collection and image processes were carried out on the experimental setup shown in Figure 1. The main body of the experimental setup is composed of belt conveyor, Arduino, the camera with lens, lights, wire laser, and lens hood.
The belt conveyor is used as an experimental platform to transport materials. Arduino and proximity sensor are used to detect whether there are materials on the conveyor belt. When materials come, camera and lights act to collect data. Arduino, meanwhile, also is used to regulate the speed of the motor by controlling the PWM signal applied to the motor controller. Lego blocks and screws were adopted as the material to be delivered in the two kinds of experimental methods. Lego is used in method 1 and screws are used in method 2. Camera 1 is mounted vertically above the conveyor belt, and camera 2 is mounted on a bracket towards the incoming material.
The purpose of this paper is to present the basic principles of the algorithm and test the feasibility. During the experiment, we adopted a manual feeding method and qualitatively described the amount of load. The amount of load was divided into three grades: large, medium, and small. The purpose of the experiment is to achieve different speed settings depending on the load; the greater the load, the higher the conveyor belt speed, to avoid the contradiction between the small load and high speed.
3. Methodology
3.1. Method 1: Flow Detection Method Based on Area Proportion
The quantity of material is identified and determined by calculating the ratio of material area to the region of interest (ROI) area. Camera 1 was used to obtain the experimental data in method 1.
The conveyor belt is basically divided into two types in real life: flat type and slotted type. The fluting property of materials on the flat conveyor belt is poor and the materials are scattered, as shown in Figure 2. Then a judgment can be made by calculating the ratio of material area to the background area and dividing it into three grades. Meanwhile the material is well grooved on a slotted conveyor belt and usually clusters in the center of the belt groove. Based on this, we can qualitatively compare the amount of material by comparing the ratio of the width of the material to the width of the belt, as shown in Figure 3.
(a)
(b)
(c)
(a)
(b)
(c)
Due to the limitation of experimental conditions, this paper focuses on the flat conveyor belt. The background subtraction method was used in this paper to calculate the ratio of the material area to the ROI area. The material area in the image information is also called the foreground area, and the belt without any load is called background.
Background subtraction is a widely used method in the current moving object detection technology [21]. The basic idea of background subtraction is similar to the interframe difference method, which extracts the target area by using the difference operation of different images; the process of extracting the target is shown in Figure 4.
Background subtraction based on the static background was adopted in this paper. When the conveyor is running normally with no material on the conveyor belt, one of the frames was picked as the static background.
The specific principle and implementation method can be divided into seven steps as follows: Step 1: get the background image with grayscale processing, . Step 2: obtain the current image according to the video sequence with grayscale processing, . Step 3: calculate the absolute grayscale image of two frames of images, : Step 4: set the threshold and binarize the image to extract the moving target. Step 5: apply mathematical morphological filtering on the image , and then perform the connected component analysis of the region. If the area of a connected region is greater than a given threshold, it is regarded as the target (foreground), which is also the region of the material on the conveyor. Step 6: calculate the area of the target and the ratio of the area of the material to the total area. Step 7: compare the detected material area to preset values and classify the target region into small, medium, or high amounts depending on the area. Send the control signal to the microcontroller for the speed control based on the material amount.
3.2. Method 2: Laser-Based Computer Vision Technology
Method 2 is a laser-based computer vision technology. A laser was used to obtain the material profile information, especially the area of the cross section S, as shown in Figure 5. Then the volume of the material on the conveyor can be estimated combining S with the current belt speed .
(a)
(b)
(c)
Suppose that the belt conveyor transports materials at a constant speed ; then S(1), S(2), …, S(n) is the contour section area of laser material in frames 1, 2, n within the period t, as shown in Figure 6(a). Then the volume of the material V(t) during the period t is
(a)
(b)
(c)
Two parameters, the belt speed and the cross-sectional area, are needed to calculate the real-time load on the conveyor belt, and the belt speed could be obtained from the speed sensor.
Due to the different particle properties of different conveying materials, two principles are given to calculate the cross-sectional area. If the material particles are small and the surface is smooth, the laser lines on the material surface obtained by the camera are mostly smooth and continuous curves, as shown in Figure 6(b), which can be directly processed using curve fitting after laser line detection. But, in most cases, the shape and size of the transported materials are irregular, and the obtained laser lines are not smooth and continuous, sometimes maybe some discrete point instead. As in Figure 6(c), the method of curve fitting or multibreak line fitting is no longer suitable. So, the problem boils down to how to efficiently extract boundaries from a bunch of discrete points and reconstruct contour shapes. It seems like a convex hull problem.
Figure 7 shows the comparison between the results obtained by the convex hull algorithm and those obtained by the alpha-shapes algorithm. As a result of the image processing, it is known that the discrete points are not ideally convex hull problems but rather have some irregular set of points that contain a variety of concave hulls. Then alpha-shapes algorithm was used to solve this problem.
(a)
(b)
Alpha-shapes algorithm was first put out in [22]. It is a powerful way to abstract the intuitive shapes from discrete point sets to get a rough outline from a jumble of unordered points. Suppose that the alpha-shapes from a point set are a polygon that is defined and is unique by the point set S and radius parameters . Its principle is shown in Figure 7(b). We can imagine a circle of radius rolling outside the point set S. When is large enough, the circle will not roll inside the point set. Therefore, when the value of is very small, every point in the point set is the boundary S. If is large (), then the boundary is the convex hull of the set of points S.
There are n points in a finite discrete point set S; these points can be composed of line; we can use the following method to judge which is on the borderline segment: within the point set S, draw a circle with radius through any two points P1 and P2, as shown in Figure 8; if there are no other points within this circle, the points P1 and P2 are considered as boundary points, and the line connecting P1 and P2 is considered the boundary line segment.
(a)
(b)
4. Results
4.1. Experimental Results of Method 1
In method 1, the amount of material on the conveyor belt was divided into three grades: small, medium, and large. A frame image with the original image and data processing results randomly selected from three levels are shown in Figure 9. The experiment was repeated twice to better identify the criteria for the three grades.
(a)
(b)
(c)
(d)
(e)
(f)
Figures 9(a)–9(c) are the original images, which represent a small amount, a medium amount, and a large amount, respectively. Meanwhile, Figures 9(d)–9(f) correspond to the processing results of Figures 9(a)–9(c), respectively.
The green rectangle in Figures 9(a), 9(b), and 9(c) illustrates the region of interest that we defined. The region of interest (ROI) has two functions: the first one is to reduce the image processing area, reduce the computation amount, and improve the running speed. Another function is to be used as a benchmark. After the target was detected, a reference is needed to perceive the size of the target. In this paper, we judge the size of the target by measuring the ratio of the target area to the ROI area. Figure 10 shows the results of area proportion. Then two thresholds T1 and T2 were set as the standard to judge the amount of materials.
(a)
(b)
The area ratio was obtained in real time as the benchmark of conveyor speed regulation. When the proportion of material area on the conveyor belt is lower than T1, the load on the conveyor belt is considered less and the conveyor should run at a slower speed. When it is between T1 and T2, it is believed that there is a certain increase of the material on the conveyor belt, and the conveyor should speed up appropriately. Besides, the closer it is to T2, the more the conveyor speeds up. When it is larger than T2, the conveyor further increases the speed or even runs at full speed to speed up the material conveying process and prevent the occurrence of piling. The area information has been continuously returning to the microcontroller that controls the speed using the PWM (Pulse Width Modulation) signal supplied to the motor controller. More thresholds also can be set to define the conveyor’s economic speed in more detail; this setting is like the CVT (Continuously Variable Transmission) of a car gearbox.
4.2. Experimental Results of Method 2
Screws are used as the transport material in the experiment of method 2; the same volume of screws was reconfigured three times to test the algorithm. The only difference between the three experiments is that the screw is transported in a different state; that is, the cross section is different, and we can see the difference in Figures 11(a) and 11(c).
(a)
(b)
(c)
(d)
Figures 11(b) and 11(d) show the point set S and the alpha-shapes result after laser line extraction and other image processings. It shows the cross-sectional information of the material more intuitively, and the area SHP algorithm was used to calculate the area and then the real-time cross-sectional area as the results were output in Figure 12.
(a)
(b)
(c)
Figure 12 shows the cross-sectional area of materials in each frame obtained by this algorithm in three experiments.
According to (3), the real-time load on the conveyor belt can be calculated by obtaining the detailed cross-sectional area and the belt speed, while the speed of the conveyor belt can be obtained in real time through the speed sensor. The dotted line in Figure 12 shows the average cross-sectional area of the material in this period t. By comparing the average area in the three experiments, it is generally considered that the method is more feasible and has less experimental error.
Just as the result in method 1, the threshold can also be set in method 2. If the detected cross-sectional area continues to be larger than the threshold we set, the system will send a signal to the controller to control the motor to speed up. Or, on the contrary, when the detection value continues to be less than the threshold value we set, the conveyor shall enter the state of low speed operation. By judging and analyzing which threshold range the cross-sectional area of the material is on continuously, the amount of load at this moment can be estimated combining with the conveyor belt speed.
5. Discussion
Both methods can qualitatively describe the load on the conveyor belt rather than quantitatively. The reasonable thresholds were set to determine the amount of load in this paper. Each method has its own advantages.
Compared with method 2, the experimental device used in method 1 is relatively simple, and the image obtained is clear and stable, but the height information of the material is lacking in the image information. Through this method, the intelligent speed regulation of the conveyor can be achieved to a certain extent, but the reliability of the result is much lower.
Method 2 has higher accuracy in determining the load on the conveyor belt compared with method 1, but the device used in this method is also relatively complex. Then the process of image processing is also complex, involving the extraction of laser lines, edge detection, contour reconstruction, and area calculation. The core of this method is calculating the material cross-sectional area. It is necessary to adopt reasonable contour reconstruction method according to the physical properties of different materials. Besides, compared with the existing 2D or 3D laser contour scanner, this method is cheaper and more stable.
As for why the experiment was not carried out on the slot conveyors, it is because the slot conveyor has certain defects that cannot be compensated: the conveyor belt is a viscoelastic body. It is supported by idler rollers, and there will be a sag greatly related to the distance between the idlers, the type of conveyor belt, and the load on the conveyor belt between two adjacent idlers. On this basis, the method of obtaining the cross-sectional area of the material through machine vision is not feasible with a large error. Similarly, 2D laser scanners cannot solve this problem.
6. Conclusions and Future Work
The real-time load of the belt conveyor is the basis for realizing intelligent speed regulation of the conveyor. A reasonable speed adjustment of the conveyor not only saves energy but also reduces wear and tear on the equipment and then reduces production costs.
In view of the detection method of real-time load on conveyor belt and combined with the advantages of noncontact measurement unique to computer vision technology, two qualitative methods were proposed in this paper for measuring and describing the real-time load of conveyor belt.
The first method is the load detection method based on the area ratio, which uses the ratio of the material area to the ROI area as the basis to judge the load amount for the flat conveyor belt. Besides, a method based on the ratio of material width to conveyor belt width for slotted conveyor belt is also proposed. The second method is a detection based on laser-based computer vision technology. The method detects the cross-sectional area of the material at the laser line in real time, and then the real-time load volume could be estimated in combination with the belt speed. In both approaches, we perform conveyor speed regulation by setting reasonable thresholds to qualitatively describe how much load is applied. Laboratory tests have shown that both methods are good for volume detection of materials and speed adjustment of conveyors in the case of transporting Legos and screws.
In the next work, the stacking density of the material will be determined and the quantitative description of the material volume will be carried out using camera calibration techniques to make the experimental results more accurate. The error problem caused by the lumpiness of the material or surface reflection is also gradually being solved. Experiments are under way to determine the load of slot conveyors, and we have solved the previous problems by deploying a set of laser generators and cameras above and below the conveyor belt in order to effectively measure the cross-sectional area of the material.
Data Availability
All data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare no conflicts of interest.
Authors’ Contributions
Mengchao Zhang contributed to methodology, software, and original draft preparation; Hao Shi contributed to reviewing and editing of the paper. All authors have read and agreed to the published version of the manuscript.
Acknowledgments
The authors wish to thank Dr. Vedang Chauhan, Assistant Professor at Western New England University, for his help in building the experimental platform and in the implementation of the algorithm.