Elsevier

ISA Transactions

Volume 110, April 2021, Pages 247-257
ISA Transactions

Research article
Vision-based neural predictive tracking control for multi-manipulator systems with parametric uncertainty

https://doi.org/10.1016/j.isatra.2020.10.057Get rights and content

Highlights

  • The kinematics and dynamics for visual servoing systems of the multi-manipulator are modeled.

  • A two-layer control scheme is proposed to handle the coordination problem for trajectory tracking systems of the multi-manipulator.

  • A model predictive controller combined with a recurrent neural network and an extended state observer is used to solve the optimization problem.

  • The input-to-state practical stability of the system and the maximal admissible bound of the uncertainty are given.

Abstract

To deal with the coordination problem for multi-manipulator trajectory tracking systems with parametric uncertainties, this paper proposes a two-layer control scheme incorporating a model predictive strategy and an extended state observer. In the kinematic layer, the visual information is implemented and a visual servoing error model is derived by the image-based visual servoing strategy. A recurrent neural network model predictive control approach is proposed to obtain velocities which are regarded as the reference signals for the dynamic layer. For dynamics, a linear time-varying dynamic model of the multi-manipulator system coupled with the object is established, where the parametric uncertainty is recognized as an added disturbance. An extended state observer is sequentially designed to estimate the disturbance by using pole placement method. The input-to-state practical stability of the system is further analyzed with a bounded disturbance. Finally, simulations and comparison are given to verify the effectiveness and robustness of the proposed algorithm.

Introduction

Vision is a class of non-contact measurement and is an essential part for robots to be intelligent. Since visual information is abundant and flexible, it has been applied widely in many fields [1]. Visual servoing is one of the main effective methods to make fully use of the visual sensors to control systems with visual feedback signals. Since the signals can enhance the ability of fault tolerance, robustness and flexibility for multi-manipulator systems, engineers and scholars pay more attention to visual servoing systems for multi-manipulator [2], [3], [4]. Generally, three categories are classified for visual servoing systems [5], [6]: position-based visual servoing [7], [8], image-based visual servoing (IBVS) [9], [10] and hybrid visual servoing [11]. Position-based visual servoing systems utilize the three-dimensional information to reconstruct the geometric model between target and camera. The hybrid visual servoing system is a complex approach combining three-dimensional and two-dimensional information with a high computational cost. IBVS designs controllers with two-dimensional coordination which has inherent robustness for camera calibration and model error.

So far, many approaches have been proposed to control systems such as proportion–integral feedback control, adaptive control, fuzzy control [12], [13], [14], [15], [16], [17]. For example, Sun et al. [16] combine a radial basis function neural network with finite-time control to deal with the unknown smooth functions existing in the nonstrict feedback nonlinear system. In [17], a robust fuzzy adaptive control scheme combined with event-triggered mechanism is proposed for the strict-feedback nonlinear system to reduce communication burden and the impact of external disturbance. Although, these approaches perform well under specific scenarios, they cannot handle the constraints well in a general framework. For multi-manipulator systems, one key issue is to explore a control scheme which can handle constraints effectively since various constraints such as input constraints, state constraints and kinodynamic constraints always exist. Model predictive control (MPC) is known as a critical way of solving the constraints [18], [19]. In [20], explicit MPC is proposed to control the air-fuel ratio where the solution space is partitioned by using a data compression technique. In [21], linear matrix inequalities are derived for the vision-based manipulator system to solve the MPC optimization subject to physical limitations and visibility constraints. Nevertheless, explicit MPC relies on the rationality of space division and linear matrix inequalities suffer from high dimensional computational burdens. With the development of high computing performance, recurrent neural network (RNN) is gradually utilized to solve constrained optimal problems [22], [23], [24]. As RNN relies less on models and can solve multi-constrained optimizations with a smaller parallel network in a theoretical and simple way, it has been widely used in robot control [25], [26]. In [25], the tracking problem for autonomous surface vehicles with unknown kinetics is discussed and transferred into a bound-constrained quadratic programming (QP) problem which is handled by RNN with fuzzy approximation. In [26], the design of a feedback-considered joint-drift-free law for redundant robot manipulators is derived as a constrained QP problem with a real-time finite-time varying-parameter RNN solving the QP problem. Due to the merits of RNN, the combination of MPC and RNN allows multi-manipulator systems to perform more effectively.

To complete the task of the robot arm holding the object and tracking the trajectory, the coordination problem should be dealt with. Precisely, the velocity of trajectory tracking should be calculated at the kinematics level and the torque generated in dynamics layer should be consistent with the velocity required by kinematics. However, it is difficult to accurately measure all the dynamic parameters of each manipulator, especially when the degree of freedom increases. Hence, for multi-manipulator systems, the influence of the uncertainty may deteriorate system performance. Besides, holding an object with uncertain volume and weight in a random pose will result in the uncertainty. Gueaieb et al. [27] apply a fuzzy system to approximate the uncertainty of the systems. Neural network is employed to estimate the uncertainty in [28], [29]. As a replaceable method, the uncertainty of a system can also be considered as an added disturbance [30]. The extended state observer (ESO) can compensate disturbances directly by feedforward correction and can circumvent the complex design of fuzzy sets and neural network structures. In [31], ESO is used to estimate nonlinear items and a back-stepping nonlinear controller is designed to solve the trajectory tracking control problem for the flexible manipulator with two joints. In [32], a time-varying ESO is designed to estimate the unknown dynamics and disturbance of the two-joint manipulator. In [33], a discrete-time ESO is proposed with event-triggered control to reduce the communication burden and attenuate the impact of the unmatched disturbances.

Motived by the above discussions, we design an ESO-based MPC control scheme for the multi-manipulator visual servoing system to reduce the influence of the parametric uncertainty and to enhance the robustness of the system. The main contributions are shown as follows:

  • (1)

    Based on the image-based visual servoing scheme, the kinematic visual servoing trajectory tracking system is modeled. Moreover, a recurrent neural network model predictive control (RNN-MPC) is proposed to provide reference signals for the dynamic layer by solving the MPC optimization with the input constraint.

  • (2)

    The dynamics coupled with the object is modeled as a time-varying linear system with parametric uncertainty. By considering the uncertainty as an added disturbance, the dynamic system is further rewritten as an extended equation, where ESO-based RNN-MPC control scheme is designed to estimate and attenuate the disturbance.

  • (3)

    The input-to-state practical stability is analyzed and the maximal admissible bound of the uncertainty is also given. Numerical simulations and comparison with two 2-DOF manipulators are given to demonstrate the rationality and efficacy of the proposed method.

The remainder of this paper is organized as follows. Section 2 introduces the modeling process of image-based visual servoing systems for eye-to-hand multi-manipulator both in the kinematic layer and in the dynamic layer. A model predictive controller combined with a recurrent neural network is proposed in Section 3 to deal with the corresponding nominal optimization problem. Section 4 designs a compensator to compensate the impact of the uncertainty by using the principle of the extended state observer, while the stability analysis of the trajectory tracking problem for the multi-manipulator system with parametric uncertainty is given in Section 5. Simulations and comparison are carried out in Section 6.

Section snippets

Problem formulation

The considered eye-to-hand multi-manipulator IBVS system is shown in Fig. 1, where an object is hold by two manipulators. The visual information is obtained by a monocular camera sensor which is installed on the shelf.

In general, two different frameworks are used for such IBVS systems: kinematics-based framework and hybrid framework [34], [35]. In the kinematics-based framework, the pixel coordinates can be tracked to the desired ones through the velocity controller. However, in practice, the

RNN-MPC design and analysis

In this section, we will propose a RNN-MPC method to design the controllers. Notice that the performance index for dynamic layer is defined as Eq. (27) shown in Section 5. Since the MPC control scheme for dynamics can be obtained in the same way as kinematics, the detailed process of the MPC design at the dynamic level is ignored. In what follows, we will discuss the design process for kinematics.

Compensator design

This section aims to design an ESO to estimate the disturbance existing in Eq. (11) and a compensator to attenuate the impact of the disturbance. Suppose that the disturbance satisfies the following incremental form at time step k: Di(k+1)=Di(k)+ΔDi(k)Combining Eq. (11), the extended state equation can be derived as follows: Xi(k+1)=ĀiXi(k)+B̄iui(k)+EΔDi(k)where Xi(k)=xi(k)Di(k), Āi=iI0I, B̄i=Bi0, E=0I. To estimate the disturbance, we denote Xi(k) as the output signal of (18). Then, the

Stability analysis

Combining Eqs. (20), (24), the following augmented system can be obtained X̃i(k+1)=ÃiX̃i(k)+B̃iui+ωi(k)where Ãi=ĀiLiCi00Ai, X̃(k)=Ei(k)xi(k), B̃=0Bi, ωi(k)=EΔDi(k)BiKicEiD(k).

In this section, the stability analysis is mainly concentrated on the stability of the above-mentioned augmented system (25) by using the proposed RNN-MPC algorithm.

Since X̃i(k) is dependent on the reference position qri and velocity q̇ri with their own maximum and minimum values, there exists a convex hull [Ai,l,Bi,

Illustrative simulations

In this section, numerical simulations and comparison are given to verify the effectiveness and the superiority of the proposed RNN-based approach in MATLAB2019a/Simulink.

Conclusions

In this paper, a two-level control strategy was proposed to deal with the parametric uncertainty by using model predictive control scheme incorporating a recurrent neural network and an extended state observer. At the kinematic level, the visual kinematic error model was derived by using image-based visual servoing method and the reference velocities were derived by utilizing the recurrent-neural-network-based model predictive control strategy. In the dynamic loop, the cooperative

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The authors thank the anonymous reviewers for their valuable suggestions to improve the quality of this paper. This work was supported by the National Natural Science Foundation of China under Grants 61973275, the NSFC-Zhejiang Joint Foundation for the Integration of Industrialization and Informatization, China under Grant U1709213, the Key R&D Foundation of Zhejiang, China under Grant 2020C01109, and the Talent Project of Zhejiang Association for Science and Technology, China under Grant

References (39)

  • LippielloV. et al.

    Image-based visual-impedance control of a dual-arm aerial manipulator

    IEEE Robot Autom Lett

    (2018)
  • ChaumetteF. et al.

    Visual servo control I: Basic approaches

    IEEE Robot Autom Mag

    (2006)
  • ChaumetteF. et al.

    Visual servo control II: Advanced approaches

    IEEE Robot Autom Mag

    (2007)
  • ZhangK. et al.

    Visual tracking and depth estimation of mobile robots without desired velocity information

    IEEE Trans Cybern

    (2020)
  • KeshmiriM. et al.

    Image-based visual servoing using an optimized trajectory planning technique

    IEEE/ASME Trans Mechatronics

    (2017)
  • GaoJ. et al.

    Hierarchical model predictive image-based visual servoing of underwater vehicles with adaptive neural network dynamic control

    IEEE Trans Cybern

    (2015)
  • LiuY. et al.

    Stabilizing a flexible beam handled by two manipulators via PD feedback

    IEEE Trans Automat Control

    (2000)
  • JabbariA.H. et al.

    Adaptive vision-based control of an unmanned aerial vehicle without linear velocity measurements

    ISA Trans

    (2016)
  • SunK. et al.

    Fuzzy adaptive finite-time fault-tolerant control for strict-feedback nonlinear systems

    IEEE Trans Fuzzy Syst

    (2020)
  • Cited by (0)

    View full text