Abstract
The Underwater Vehicle Manipulator System (UVMS) is an essential equipment for underwater operations. However, it is difficult to control due to the constrained problems of weak illumination, multidisturbance, and large inertia in the underwater environment. After the UVMS mathematical model based on water flow disturbance is established, fusion image enhancement algorithm based on Retinex theory is proposed to achieve fine perception of the target. The control method based on redundant resolution algorithm is adopted to establish the anti-interference controller of the manipulator, which can compensate the internal and external uncertain interference. Finally, stable underwater operation is realized. The target ranging method is used to solve the angle of each joint of the manipulator to complete the tracking and grasping of the target. Underwater experiments show that the algorithm can improve the clarity of underwater images, ensure the accuracy of robot capture, and optimize the UVMS control performance.
1. Introduction
The Underwater Vehicle Manipulator System (UVMS) is an essential equipment for underwater operations, and it plays an important role in national defense construction, underwater engineering operations, underwater rescue, and marine exploration and development. At present, the UVMS has evolved from information-based operation to autonomous operation, which requires a precise control system. Underwater target detection and recognition are important prerequisites for the underwater operation of UVMS manipulators. The popular method is acoustic and optical perception. Currently, underwater optical sensing is mainly used, but underwater optical sensing faces many unfavorable factors such as the scattering and absorption of light by water, low contrast, and optical distortion. This makes the UVMS unable to achieve reliable identification of complex underwater environments. Therefore, enhancing the fine recognition capability of UVMS autonomous underwater targets is the basis for achieving accurate target information determination and manipulator capturing.
Underwater imaging technology is widely used in engineering applications and scientific experiments, such as underwater tracking and positioning and exploitation of marine resources [1]. The underwater target object is affected by water during the imaging process, and the light is partially absorbed, reflected, and scattered. The image is affected by the distance between the camera and the target object, and the intensity of the light decreases sharply as the distance increases [2]. Affected by this factor, the contrast of underwater images is reduced, and the sharpness of the target object in the image is poor. It is not conducive to further processing of underwater video images [3].
In the recent years, domestic and foreign scholars have studied a variety of underwater image enhancement methods, such as the adaptive histogram equalization based on contrast limitation, histogram method based on Rayleigh distribution stretching [4], transformation probability method [5], multiscale fusion method [6], image depth of field estimation method [7], optical property estimation method [8], and so on. These methods belong to the spatial domain enhancement method, which can remove the color cast effect and increase the difference between the original similar pixels, thereby improving the contrast and sharpness of the image. But, the disadvantage of these methods is that it is easy to produce insufficient or excessive enhancement, thereby forming a false edge or texture. In addition, underwater image enhancement methods based on color constancy theory have also been applied, such as Retinex algorithm [9] and nonlinear image enhancement algorithm based on Retinex algorithm because Retinex algorithm itself has the function of constant color, which can significantly improve the local contrast of image enhancement. The algorithm can effectively improve the details and color information of the dark area of the artificial light source image; this is unmatched by other current image enhancement algorithms, so it is particularly suitable for UVMS. UVMS fusion image enhancement algorithm based on bilateral filtering and retinal algorithm is proposed in this paper. Firstly, the initial image is bilaterally filtered to obtain a coarse image. Secondly, the initial image is divided by the coarse image to obtain a detailed image. Thirdly, the rough image is improved by Retinex enhancement algorithm. Finally, the processed rough image and details are processed. Then, the image is fused to get a clear image.
The complexity and unpredictability of the real underwater environment make the design of UVMS precise controllers more difficult. Domestic and foreign scholars have also proposed a variety of control strategies. Londhe [10] used fuzzy logic control such as PID as the feedback part in order to enhance the overall closed-loop stability of the UVMS. Qiao and Zhang [11] adopted an adaptive nonsingular integral terminal sliding mode control. It predicts dynamic uncertainty and time-varying external interference to achieve trajectory tracking of autonomous underwater vehicles. Li [12] used adaptive antijamming control. Through real-time estimation and compensation of disturbance forces, higher control accuracy is achieved. Dai [13–15] presents an indirect adaptive control scheme which consists of three parts: an extended Kalman filter (EKF) estimation compensative system, a model-based computed torque controller (CTC), and an H∞ robust compensative tracking controller. Yan et al. [16] proposed a new trajectory tracking scheme for the constrained nonlinear UVMS. Zhan and Jin [17] adopted redundant resolution algorithm to predict local state dynamics as a fuzzy average of global state dynamics, and this is a discrete-time optimal control method. The response speed of the system disturbance adjustment is improved, and the anti-interference optimization of the control is realized. The dynamic stability of the system is improved.
Chen et al. [18] described the visual servo control method of the manipulator on land. With reference to this idea, the control process of the underwater manipulator is as follows.
First, stereo calibration of the underwater camera is performed, then underwater image enhancement technology is used to improve the quality of the original image, then hand-eye calibration algorithm is used to calculate the three-dimensional coordinates of the underwater target in the robot coordinates, and the robot is solved by inverse kinematics rotation angle of each joint. Finally, the servo motor is controlled to drive the joints of the manipulator to move, and the visual servo control experiment of the underwater manipulator is completed.
Retinex fusion visual enhancement algorithm effectively improves the details and color information of the dark area of the artificial light source image. This is unmatched by other current image enhancement algorithms
The redundant resolution can avoid the self-motion of the redundant system. At the same time, local key parameters are used to predict the characteristics of global parameters, which reduce the amount of control calculations compared to other control algorithms and improve the response speed of system disturbance adjustment.
This paper studies optimization algorithm for the abovementioned two points of image enhancement and manipulator control grabbing. Firstly, this paper uses Retinex theory to establish an image fusion enhancement algorithm to improve underwater image capture and analysis capabilities. From the perspective of fluid mechanics, various complex disturbance mechanisms are analyzed, such as complex underwater flow disturbance, uncertain disturbance of a system, and unstructured random disturbance. Secondly, a mathematical model of the UVMS in the actual water flow environment is also established. The redundant resolution algorithm is used to optimize the control performance of the manipulator and establish a performance index closer to the actual. The UVMS is used by our team to carry out sea cucumber model grabbing experiments, and it obtains the real-time angle of servo feedback of each joint of the UVMS. The analysis data shows that the UVMS manipulator accurately tracks the target trajectory while maintaining the stability of the UVMS, realizing the task of UVMS underwater operation under certain interference. This indirectly verifies the superiority of image fusion algorithms in underwater imaging and detail capture. Figure 1 is an UVMS control research roadmap based on fusion visual image enhancement. It is mainly divided into two parts: fusion image enhancement and redundancy resolution algorithm. The fusion image enhancement algorithm can get a clear image to observe the underwater environment, and the redundant resolution algorithm can get a stable dynamic system. Only combining fusion image enhancement algorithm and redundant resolution algorithm can be good for underwater target grabbing.
2. Establishment of the UVMS Mathematical Model Based on Disturbance
The UVMS movement mainly involves three coordinate systems: the earth coordinate system, the body coordinate system, and the manipulator coordinate system [19], as shown in Figure 2.
Set as the geodetic coordinate system (configuration space), as the UVMS body coordinate system, and as the end effector coordinate system (task space). Vehicle dynamics are usually described by two coordinates: one is the fixed coordinate system and the other is the inertial frame of reference. The kinematic equation of the UVMS body can be expressed as follows: is the inertial force and system moment vector. is the Coriolis vector and centripetal effect, and is the hydrodynamic damping force vector. is a recovery (gravity and buoyancy) vector. is the moment input vector of thruster and control plane forces.
is the linear velocity and angular velocity vector of the UVMS body. is a vector of Euler angles. The suffix indicates that it corresponds to the vehicle.
The relationship between the attitude of the underwater manipulator and the system configuration is as shown in (2):where is a vector containing the position and orientation of the underwater manipulator relative to the earth’s fixed frame. represents the forward kinematics vector. Derivation of the abovementioned formula:where is the Jacobian matrix which constitutes the speed mapping relationship between the configuration space and task space. Equation (1) is transformed into the dynamic equation of the manipulator in the task space:where is the inertial matrix of the manipulator in the task space. is the Coriolis force vector and centripetal force matrix of the manipulator. is the vector of the hydrodynamic damping force of the manipulator. is the vector of recovery effect of the manipulator. is the vector of dynamic coupling of the manipulator. is the vector that controls the input.
The dynamic model formulation of the underwater manipulator is established using the recursive Newton–Eulerian. The equation of motion dynamics of an underwater manipulator can be expressed as follows: is the vector of joint variables. is an underwater disturbance vector.
Equation (5) is transformed into the dynamic equation of the underwater manipulator:where is the external disturbance vector in the task space. For the water flow in the underwater environment, it can be regarded as the circulation system of the fluid in the horizontal and vertical directions. It is a process of slowly changing low frequency caused by changes in tidal wind, wind friction, and fluid density. External disturbance vector can be described aswhere is the current velocity vector. is a positive definite diagonal matrix. is a Gaussian white noise vector. Both and are constants. Taking into account the uncertainty of the parameters in the actual system, we add these factors to get
Summarizing system uncertainty and external environmental interference as concentrated uncertainty [20], equation (9) is simplified to
Subtracting the abovementioned two formulas, we get
3. UVMS Fusion Image Enhancement Method
The basic characteristics of underwater imaging are mainly expressed as follows:(1)The gray effect: with the attenuation of light signals underwater, the imaging effect of underwater targets is affected by flocs and underwater low-light environment(2)Low contrast: because of the scattering of light in the underwater environment, the image contrast of the target taken by the underwater camera is poor(3)The distinction is greater(4)The image effect is worse
Due to the water flow disturbance and the near-surface disturbance caused by the underwater environment of the UVMS, the natural light source is prone to generate spots in the disturbed seawater, causing uneven illumination of the feedback image. In the deep sea, the UVMS relies on artificial light sources to obtain images and videos, which exacerbates the problem of uneven image illumination. This causes the decrease in the observation range of the UVMS, which brings greater difficulties to subsequent underwater operations.
Aiming at the abovementioned problems, UVMS fusion image enhancement algorithm based on bilateral filtering and Retinex algorithm is proposed. Figure 3 is a flowchart of UVMS fusion visual image enhancement algorithm. Firstly, the initial image is bilaterally filtered to obtain a coarse image. Secondly, the initial image is divided by the coarse image to obtain a detailed image. Thirdly, the rough image is improved by Retinex enhancement algorithm. Finally, the processed rough image and details are processed. The improved rough image and detailed image are fused to obtain the desired image.
The physical basis of the algorithm is that, in the process of observing the object to the final image, the visual system automatically removes the influence of illumination factors such as the intensity of the light source and the uneven illumination and only retains the properties of the object itself.
3.1. Bilateral Filtering
Figure 4 shows two typical underwater original images. As can be seen from the figure, the underwater image shows the characteristics of low contrast and poor clarity.
The bilateral filtering algorithm proves that using bilateral filtering to process images provides better edge retention than Gaussian filtering, effectively avoiding the occurrence of halo phenomenon. The bilateral filtering expression is as follows:where is the brightness of the output image at . is the brightness of the input image at .
In equation (12), represents the square of the distance from each pixel to the center of the image .
In equation (13), represents the square of the difference between the gray value of each pixel and the center point .
The abovementioned two enhancement algorithms get two enhanced images, which are fused to enhance the image. The fusion formula is shown in follows:
In (14), is the values of R, G, and B of the merged image. is the R, G, and B values of the MSR multiscale Retinex- (MSR-) enhanced image. is the R, G, and B values of the enhanced image using bilateral filtering algorithm. m is the control coefficient and satisfies 0 < m < 1.
3.2. Retinex-Improved Enhancement Algorithm
The rough image collected by the underwater camera is improved by Retinex enhancement algorithm.
Firstly, the input image is precorrected to reduce the dominant color. Secondly, the improved MSR algorithm is performed using the intensity channel to estimate the reflectance and illumination components. After that, the image is restored and the dynamics are compensated. Finally, the color of the original image is alternately retained according to the requirements. Figure 5 is the Retinex-improved enhanced algorithm flow.
3.2.1. Color Precorrection
Since the density of water in seawater is much greater than that of air, water can absorb light energy. As the depth increases, the underwater image becomes darker and darker. The color will be unbalanced according to the decrease in the wavelength, and the underwater image will always be mainly green or blue. Therefore, color precorrection of underwater images is an indispensable step. Here, and are introduced as the average and deviation values of the input image, and the maximum value and the minimum value of each channel can be obtained by
In equation (14), , is the dynamics nominal, and is the color equalization scale.
The image after color precorrection can be expressed as follows:
3.2.2. MSR Algorithm with Intensity Channel
The color of an object observed by a person is not determined by the light intensity, but by the surface reflection ability of the object [21]. The image observed by the human eye can be expressed as a formula:where represents the reflected light image, which determines the intrinsic properties of the image. represents the illuminance image. represents the image obtained by the observer [22]. The reflection coefficient unique to an object is
Taking the logarithm of both sides at the same time, we can get
The illumination component of the image is removed from the original image, thereby obtaining the reflection component of the image surface for the purpose of image enhancement.
Single-scale Retinex (SSR) is constructed by introducing a Gaussian surround function, but the SSR cannot provide dynamic range compression and color recovery at the same time [23]. The multiscale Retinex (MSR) compensates for the abovementioned shortcomings by weight superposition. The definition is as follows:where is the number of scales, is the weight of each scale, is the output image of the SSR, is the convolution operation, and is the Gaussian surround function for each scale. , is the intensity channel used to save the image color, and is the number of image channels. The expression is as follows:
3.2.3. Dynamic Compensation
As shown in (19), the MSR algorithm obtains an enhanced image from the logarithmic domain by linear transformation. Therefore, gains and deviations are often considered during the recovery process. These two feature quantities usually need to be adjusted for a given image, and it is difficult to apply to all underwater image enhancements. The effectiveness and practicability of the algorithm are greatly reduced.
In [24], the authors applied the adaptive stretching method in the CIE-Lab color model and established a stretching formula normalized to the brightness parameter and, finally, obtained a clear image with high contrast and saturation. We use the method of dynamic adaptive stretching to compensate image information. The compensation formula is as follows:where and are the maximum and minimum values of each channel after MSR processing, and are the average and deviation values of the MSR output image, and is the dynamic range, .
The final compensated image is expressed as follows:
is the dynamically compensated MSR image.
3.2.4. Color Retention
In MSR algorithm, color information is easily lost, which leads to a decrease in color saturation. With the emergence of Multiscale Retinex with Color Restoration (MSRCR), the abovementioned problem has been solved [25]. But, color gain and deviation face similar problems with image information. This limits the use of MSRCR. If we enlarge the original image chrominance information by a certain proportion, we can avoid the excessive color gain and deviation and generate an image closer to the original color.where , where is the color retention coefficient, is the average of the original image, is the intensity channel average of MSR algorithm after dynamic compensation, and is the dynamically compensated MSR image.
4. Underwater Manipulator Redundancy Resolution Algorithm
The precise model of the manipulator in the task space is established above. On this basis, this section will describe the servo antidisturbance control method of the underwater manipulator under visual enhancement.
The UVMS is a combined system of different speeds, and there is system self-motion. Redundancy can be used to determine the desired trajectory of the body and end effector without affecting its motion in the workspace.
is the Jacobian matrix in this research, and n is the number of joints. When the velocity profile of the underwater manipulator is given, the joint speed of the robot and the speed of the UVMS can be determined by redundant resolution.where is the Moore–Penrose pseudoinverse matrix of the Jacobian matrix J. This can be calculated according to the following equation:where in equation (27) is a specific solution that is determined by the given speed of an underwater manipulator. in equation (26) is the homogeneous solution obtained by projecting on the zero space of the Jacobian matrix J. It represents a self-motion that does not affect the motion of the task. Therefore, the vector J can be defined to obtain the optimal solution for the specified performance metric.
When a UVMS robot hovers over an underwater object, it will cause vibration or even instability of the UVMS body. The distance between the mass center position of the UVMS and the ZMP position is set as the performance index as in equation (27). In this paper, performance metrics are minimized along with redundancy analysis.where is the weight matrix. UVMS manipulators are redundant so that the extra redundancy used in determining the zero moment point (ZMP) position can be used for other tasks, such as trajectory optimization. For this type of work, another performance metric can be used to create additional performance metrics for the angular trajectory of the manipulator joint, as shown in the following equation:where and represent the maximum and minimum limit angles of the i-th chain and represents a weight matrix. In this paper, the limit angle of the node angle is set as a constraint considering the practical application of the manipulator. In order to satisfy these two constraints, a performance index that can establish the angular trajectory of the manipulator joint to improve stability is proposed. The performance of the entire UVMS over a limited range of manipulator joint angles is expressed as follows:
As shown in (30), the gradient analysis method (GAM) is used to perform redundancy analysis on the proposed performance index, and the solution in zero space can be obtained.where and represent the gradient vectors of the positive gradient constant and the performance index, respectively.
5. Experiment
The self-developed UVMS is used for underwater vision enhancement experiment and robot visual servo grasping experiment, respectively. The UVMS is equipped with a three-degree-of-freedom electric manipulator and a network camera, and it is equipped with 4 electric thrusters. The solid composite foam with a water absorption rate less than 1% (24 h) is selected on the UVMS, which can withstand the underwater environment of 2000 ft to 36000 ft. It has built-in environmental awareness systems such as inertial navigation, bathymetry, and thermal conductivity depth (CTD). The system is shown in Figure 6.
5.1. Image Enhancement Contrast Experiment
We carried out fusion image enhancement algorithm on underwater images. Then, it is compared with MSR algorithm and bilateral filtering algorithm for imaging effect and data analysis.
The imaging effect in Figure 7 shows that MSR algorithm compensates the detailed information of the dust at the bottom of the pool, and this enhances the color of the underwater image while maintaining the color constancy. Bilateral filtering algorithm considers the influence of the similarity between the pixel and the central pixel in the convolution kernel, and this eliminates noise while preserving the edges, enhancing the layering of the image. However, the enhanced image is obviously pale in color and distorted in color. In contrast, the fusion enhancement algorithm compensates for the imaging shortcomings of both, and it can better restore the shape of the sea cucumber model and the details of the experimental environment. The overall picture is natural and closer to the actual underwater environment. Because the amount of image detail increases, this provides a wealth of reference data for subsequent underwater target recognition and ranging.
(a)
(b)
(c)
(d)
The data comparison in Table 1 shows that fusion enhancement algorithm has greatly improved the brightness, contrast, and clarity. The effectiveness of fusion algorithm in detail processing is confirmed by the improvement of information entropy. In the future, we will study 3D feature extraction and positioning technology of underwater targets based on point cloud data [26, 27].
5.2. UVMS Anti-Interference Grab Experiment
The overview of the visual servo anti-interference crawling experiment is as follows: First, inertial navigation, sounding, and other sensors are fused to detect the attitude of the system under the disturbance, and the control method based on redundant resolution algorithm is used to establish the anti-interference controller of the manipulator; it compensates the displacement of the UVMS body that has not been corrected or overcompensated in time. The vector data will be combined with the visually resolved data to obtain the final joint motor feedback angle.
Because the research object is the UVMS with 3 degrees of freedom control ability, the model parameters are shown in Table 2. The parameter selection of the controller is as follows:
We simulated 1–3 sections of mixed water flow interference in the range of 5–15s:
We generate a three-dimensional trajectory diagram of the end effector of the UVMS underwater operation process (Figure 8).
The abovementioned simulation trajectory can be seen: The manipulator has a smooth transition during the interference period, there is no severe fluctuation and the servo control performance is excellent, and the underwater target is tracked accurately and quickly. As shown in Figure 8, the sudden disturbance brings displacement in the y-axis direction of the manipulator and antidisturbance controller makes the large-scale displacement caused by disturbance be compensated, and this allows the system to self-adjust in a shorter time. It is confirmed that the antidisturbance controller based on redundant resolution algorithm has an excellent balance control effect on the UVMS under the interference of water flow, and this improves the UVMS underwater control performance.
The turbulent boundary layer between the shallow water and the sea floor caused by the current has a great impact on underwater fishing operations. In order to achieve the effect of water flow turbulent layer, two horizontal and vertical 250 W aeration surge water cannons are set at the critical point of the water surface within 30 cubic meters centered on the UVMS operating point. We make the disturbance of the water flow under the fourth-level wind, and the reference velocity is 5 m/s.
We take the hovering UVMS centroid as the reference origin. The three-dimensional space point (unit: m) of the grab target in the task space obtained by visual ranging is 0.632, 0.643, −0.992. The reverse rotation angles of shoulder and elbow joints are 25.42° and 30.11, respectively. The relevant data during the movement of the manipulator is recorded in real time by feedback hardware such as underwater encoders and travel switches in each joint. Figure 9 is the experimental process of grabbing a sea cucumber model. Figure 10 shows the joint real-time feedback data. Data show that the final position of the shoulder joint of the underwater manipulator is 25.60°, and the final position of the elbow joint is 29.91°.
(a)
(b)
Experimental analysis shows that, under the interference of water flow, it is difficult for the manipulator to reach the ideal position, and the shoulder joint angle is too large. However, the angle error remains within ±2°, and this increases the success rate by 20% and the desired control effect is achieved.
Figure 11 shows the servo control capture process. The experimental results show that the grasping process is stable and the underwater manipulator reaches the target position to complete the designated task. At present, the optimal control methods of onshore industrial manipulator are relatively mature. In the next step, in order to obtain better underwater grasping effect and efficiency, we plan to use these methods in the acquisition of the underwater manipulator [28–30].
(a)
(b)
(c)
(d)
6. Conclusions
This paper proposes an optimized solution for the disadvantages of poor UVMS underwater image imaging quality and weak anti-interference ability. Fusion image enhancement algorithm is used to improve the underwater image capture and analysis capabilities. First, building an UVMS mathematical model with all the uncertainties, the model provides a basis for the design of antidisturbance controllers to obtain better UVMS antidisturbance parameters. In order to verify the performance of the anti-interference controller based on the redundant resolution algorithm and the enhancement effect of the image, we conducted an underwater sea cucumber model crawling experiment. The experiment measured the real-time angle of the servo feedback of each joint of the underwater mechanical arm and the three-dimensional trajectory of the manipulator. Analysis data show that the underwater manipulator accurately tracks the target trajectory while maintaining UVMS stability. This realizes the task of UVMS underwater operation under the disturbance of uncertainty. This verifies the effectiveness of the image fusion algorithm in underwater imaging and detail capture.
The research can be applied to the direction of aquaculture, and it can more clearly observe the growth of aquatic products and improve the grasping efficiency of underwater manipulator to improve the profit of aquaculture. Based on this article, the fusion enhancement algorithm under the artificial light source in the deep sea and the state prediction accuracy of the manipulator antidisturbance controller will be further developed in the future. The experimental study of the UVMS actual seafloor operation under the disturbance of ocean current is carried out to improve the comprehensive control performance of the closed-loop control system.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This research was supported by Industrial Foresight Project of Zhenjiang City (GY2018018), Jiangsu Graduate Scientific Research and Practice Innovation Program (SJCX20 1474), and the National Natural Science Foundation of China (51809128).