Abstract
To achieve the purpose of accurately grasping a random target with the upper limb prosthesis, the acquisition of target localization information is especially important. For this reason, a novel type of random target localization algorithm is proposed. Firstly, an initial localization algorithm (ILA) that uses two 3D attitude sensors and a laser range sensor to detect the target attitude and distance is presented. Secondly, an error correction algorithm where a multipopulation genetic algorithm (MPGA) optimizes backpropagation neural network (BPNN) is utilized to improve the accuracy of ILA. Thirdly, a general regression neural network (GRNN) algorithm is proposed to calculate the joint angles, which are used to control the upper limb prosthetic gripper to move to the target position. Finally, the proposed algorithm is applied to the 5-DOF upper limb prosthesis, and the simulations and experiments are proved to demonstrate the validity of the proposed localization algorithm and inverse kinematics (IK) algorithm.
1. Introduction
According to the 2006 China National Disability Sample Survey, there were 2.26 million amputees, accounting for approximately 9.37% of the disabled [1]. From the information provided by the National Limb Loss Information Center, in 2005, there were 1.6 million who lost a limb in the United States, and this number is expected to reach 3.6 million by 2050 [2]. A large proportion of these amputees are upper limb amputees who do not have the means to perform some daily activities, such as drinking water, eating, and dressing. Upper limb prostheses that can capture and move objects according to the wearer’s needs are wearable robots from the perspective of human–computer integration [3]. However, for upper limb prostheses, the pose (position and orientation) information of a random target is not fixed, and inverse kinematics (IK) solution cannot be determined. Unfortunately, research on random target localization algorithms is mostly seen in the fields of mobile robots and industrial robots and rare in the field of upper limb prostheses. Therefore, how to accurately obtain the pose information of a random target has become a difficult problem that researchers need to urgently address.
Cao et al. [4] studied a method of using machine vision and a neural network (NN) to decrease the absolute position error of the robot. However, the NN was easily trapped in a local solution, which led to instability in the localization accuracy. Jiang et al. [5] proposed an algorithm which combined particle swarm optimization algorithm with back propagation neural network (BPNN) to solve the IK solution of the UR3 robot. However, there was a lack of blindness in the selection of the step size, which affected the accuracy of the solution. Yi et al. [6] proposed a positioning system using optical flow sensor based on the sensor fusion technique, However, in low-light and slippery environments, the system generates large errors. Jung et al. [7] selected the pose of robot by extracting the initial state of the surrounding environment of the robot and determined the sample distribution according to the given area map. The positioning success was determined by calculating the similarity between the robot sensor state and the estimated posture on a given map. But the positioning accuracy is greatly affected by the environment.
The research object of this paper is 5 degrees of freedom (DOF) upper limb prosthesis, as shown in Figure 1. It is not a commercial product, but a product developed by the research group and is an internal product of the laboratory. The remainder of this paper is structured as follows. In Section 2, the kinematic of the upper limb prosthesis is analyzed. The localization algorithm is introduced in Section 3. In Section 2.2, the IK algorithm is addressed. The proposed algorithm is validated through simulations and experiments in Section 4. Finally, Section 5 provides conclusions and future work.
2. Kinematic Analysis
2.1. Forward Kinematics
The relationship among the joint coordinate systems is illustrated in Figure 2, and the D-H parameters are shown in Table 1.
where , , , and .
The pose of end-effector (EE) in the base coordinate system is described as :
When is known, solving the pose of the EE in the base coordinate system is called positive kinematics. Conversely, when the pose of the EE in the base coordinate system is known, solving the joint angle is called IK.
Drinking water is a common activity in real life. To avoid the overflow of water from the cup during the movement of the upper limb prosthesis, this paper uses an MPU6050 hand attitude sensor to control the gripper to always hold and move in the horizontal direction. Therefore, the EE coordinate system is described as in the coordinate system . Additionally, the EE coordinate system is described as in the base coordinate system.
Because the attitude information of the gripping object is simplified, the key problem of the IK solution becomes obtaining the exact position of a random target.
2.2. IK Algorithm
According to equation (3) and the output values of MPGA-BPNN, the following equations can be obtained:
Through equation (4), the angular values of the front three joints can be obtained. However, this is a complex set of nonlinear transcendental equations that has no analytical solution. Therefore, a method for solving transcendental equations by using the fast nonlinear approximation ability of a general regression neural network (GRNN) [9–11] is use d in this paper. Once the learning samples are determined, the connection weights between the corresponding network structure and neurons are also determined. When using a GRNN to solve the problem, we need to complete the following work. First, in view of the training time and accuracy requirements, are evenly dispersed into 100 copies in the range of the joint angle change, and then the prosthetic joint space is obtained and saved as the matrix, which is used as the target sample of the GRNN. Then, the corresponding coordinates T are obtained by substituting into the first three rows of the fourth column of equation (3) and saved as the matrix, which is used as the input samples of GRNN. In this way, the IK solution model is established.
3. Localization Algorithm
3.1. Initial Localization Algorithm (ILA)
The three-dimensional attitude sensor [12], which relies on geomagnetic localization, aggregates three-axis acceleration, a three-axis gyroscope, and a three-axis magnetometer. This attitude sensor can detect and describe three-dimensional pose changes with respect to Earth coordinates (roll direction, pitch direction, and yaw direction). To obtain a random target initial position, two attitude sensors (MPU9250) and a distance sensor (CY30) need to be installed on the wearer, as shown in Figure 3. Attitude sensor 1 is installed on the artificial limb shoulder to calibrate the attitude with the rotation of the body, and attitude sensor 2 is installed on the head of the wearer to detect head movement relative to attitude sensor 1. When the wearer’s head is upright and looking straight ahead, the attitude data of sensor 1 and sensor 2 are consistent, and this moment is described as the preliminary state. When the wearer’s head turns up and down or turns left and right, the upper limb prosthesis can obtain the rotation or pitch angle of the head relative to the wearer’s body by comparing the data of attitude sensor 1 and attitude sensor 2. To confirm the operation target distance, a laser range sensor is installed on the side of the head (left ear) and emits a laser projected onto the operation target by adjusting the head attitude.
The coordinate systems are shown in Figure 4. S is the prosthetic basic coordinate system, N is the coordinate system about the middle of the neck, H is the coordinate system about the middle of the head, and E is the laser range sensor coordinate system. When the user’s head turns up and down, it is equivalent to N coordinates rotating by an angle around the S coordinates of the axis, and the origin of the N coordinate system is fixed at relative to the origin of the S coordinate system. When the user shakes his head left and right, it is equivalent to H coordinates rotating by an angle around the N coordinates of the axis, and the origin of the H coordinate system is fixed at relative to the origin of the N coordinate system. When the head is in the preliminary state, the origin of the E coordinate system is set in the positive direction of the axis about the H coordinate system, and the axis of the E coordinate system coincides with the axis of the H coordinate system. Furthermore, the axis of the E coordinate system coincides with the direction of the laser beam, and the axis of the E coordinate system and axis of the H coordinate system form a fixed angle , which is equivalent to the E coordinate system rotating around the axis of the H coordinate system. The origin of the E coordinate system is fixed at relative to the origin of the H coordinate system.
By comparing the relative attitude angle discrepancy between the two three-dimensional attitude sensors, the homogeneous transformation matrix of the N coordinate system to the S coordinate system can be derived as follows:
In the same way,
The target distance measured by the laser ranging sensor is d, and its homogeneous matrix D is
An upper limb prosthesis controller for the wearer that has a random target coordinate calculation program is prepared. By adjusting the head attitude, the wearer projects the laser emitted onto the operation target. Through coordinate transformation, the position of the target M in the S coordinate system can be obtained.where are associated with the physical parameters of the wearer, and the rotation angles and are found by comparing the data from the two attitude sensors. depends on the installation position of the laser range sensor on the wearer’s head.
3.2. Error Analysis
Although the ILA can locate a target in an unstructured environment, the attitude sensor used in the localization system is based on the principle of inertial attitude measurement. Drift accumulation error and the magnetic bias can produce positioning errors. In actual random localization experiments, we collected 100 deviation values from the three-dimensional attitude sensors relative to the roll, pitch, and yaw directions of the Earth. The detection accuracy of the two three-dimensional attitude sensors relative to the deviations in the roll and pitch directions of the Earth is relatively high (no more than 1°). However, the detection accuracy of the two three-dimensional attitude sensors relative to the deviation in the Earth’s yaw direction is poor (approximately 10°). This is very unfavorable for the purpose of determining the localization of a random target. Through histogram analysis of 100 data points in the yaw direction, it is found that the distribution of this group of data basically conforms to a Gaussian distribution. Therefore, we set a fixed sampling period for the deviation values of the two attitude sensors in the yaw direction to collect and calculate 100 values of the deviation in the yaw direction each time and then to calculate their mathematical expectations. The distribution curves of the deviation values before and after processing by this method are shown in Figure 5. Although this method improves the stability of the random target location of the upper limb prosthesis, localization accuracy still needs to be improved.
In addition, structural clearance and installation angle errors will inevitably occur in the process of prosthetic limb assembly. There are many reasons for the errors that ultimately reduce the localization accuracy of the system. The red laser spot in Figure 6 represents the desired position. It is obvious that the actual position of the gripper center deviates from the desired position. Also, the errors between the initial positioning coordinates and the desired position coordinates are shown in Figure 7. It is found that the positioning error by ILA is approximately 28 mm, which cannot meet the wearer’s requirements. Therefore, based on ILA, this paper further uses an error correction algorithm to improve the localization accuracy.
3.3. Error Correction Algorithm
3.3.1. BPNN Algorithm
NNs are currently popular in machine learning. Among them, BPNN is a multilayer feedforward NN whose characteristics are forward signal transmission and error back propagation. In the forward transmission process, the signal is processed layer by layer from the input layer through the hidden layer to the output layer. If the output layer cannot obtain the desired output, the signal is transmitted to back propagation, and the weight and threshold are adjusted according to the prediction error, so that the output of BPNN continues to be close to the expected output [13–15]. It is known that BPNN containing a hidden layer has sufficient accuracy to approximate any continuous function [16]. Figure 8 shows BPNN topology diagram, where are the initial location coordinates of the target point and are obtained by the method described in Section 3.1. are actual measured values of the target points corresponding to the aforementioned .
Because the initial weight and threshold are random, with the training of the BPNN, the weight and threshold will be constantly updated, and eventually the weight and threshold will converge to a certain value. However, the gradient descent training algorithm of the BPNN may produce local minima. Evolutionary algorithms (EAs) can be effectively used to find the optimal weight and threshold globally without calculating gradient information [17]. EAs include simulated annealing algorithm (SAA) [18, 19], particle swarm optimization (PSO) [20, 21], and genetic algorithm (GA). Compared with SAA and PSO, GA has the advantages of efficient heuristic search and parallel computing and has been widely used in function optimization and combination optimization. Therefore, this paper uses a GA combined with the BPNN (called GA-BPNN) to solve the problem that the BPNN falls into local minima solutions.
3.3.2. GA-BPNN Algorithm
A GA is an optimization method based on natural selection mechanism in the process of biological evolution [22–24]. This paper uses GA’s global search ability to search the optimal weights and thresholds of BPNN. Then, the prediction model with the optimal weight and threshold of BPNN is established, referred to as GA-BPNN. Figure 9 is a flowchart of GA-BPNN algorithm.
The deviation of the output values of the BPNN and the actual measured values is taken as the objective function of the GA, defined as follows:
The goal of GA is to find the weight and threshold of BPNN with the least sum of squares in all evolutionary generations and to develop in the direction of increasing the value of the fitness function. It is constructed as follows:
GAs include single-population genetic algorithm (SPGA) and multiple-population genetic algorithm (MPGA). The early SPGA is popular, but there are some shortcomings, such as premature convergence, poor local search ability, and serious assimilation of the late population. For this reason, multipopulation genetic algorithm (MPGA) [25, 26] was proposed. Besides of solving the shortcomings of single-population genetic algorithm, the convergence speed and accuracy have been improved. This paper uses MPGA and SPGA to search the optimal weights and thresholds of BPNN (called MPGA-BPNN and SPGA-BPNN, resp.) and observes the changes of fitness values obtained by these two algorithms. The specific parameters of the SPGA and MPGA are as follows:(a)SPGA: binary coding, roulette selection, and single-point crossover. The crossover probability is 0.85, and the mutation probability is 0.07. The evolutionary algebra is set to 200, and the total number of individuals is 1000.(b)MPGA: binary coding, roulette selection, and multipoint crossover. The crossover probability and the mutation probability of various groups are randomly generated in [0.4, 0.9] and [0.005, 0.1], respectively, and the immigration operator and artificial selection operator are introduced. The evolutionary algebra is set to 200, the number of populations is 10, and the number of individual populations is 100.
3.3.3. GA Evaluation
With the operation of SPGA-BPNN and MPGA-BPNN, the changes curve of fitness value is obtained, as shown in Figure 10. It is not difficult to find that the best fitness value obtained by MPGA is 0.996, which is much higher than the 0.56 obtained by SPGA. Therefore, this paper chooses MPGA-BPNN as error correction algorithm of ILA.
After the training, the and the average absolute error (AAE) of the output values of the MPGA-BPNN relative to the actual measurement value are obtained by (15) and (19). To evaluate the error correction ability of MPGA-BPNN, of the ILA, BPNN, and SPGA-BPNN are introduced as follows:where, are the output values of BPNN, are the output values of SPGA-BPNN, are the output values of MPGA-BPNN, and N is the number of test points.
4. Simulations and Experiments
In this section, simulations and experiments are constructed to prove the validity and practicability of the proposed algorithms. Simulations are implemented with the aid of the MATLAB R2015a tool and experiments are run using an Intel Core™ i5-2450M CPU @ 2.50 GHz and 2 GB RAM as the control platform.
4.1. Simulation Analysis of Localization Algorithm and IK Algorithm
To assess the accuracy of the proposed localization algorithm, 200 points were randomly selected from BPNN training samples as test points. The errors in the x-, y-, and z-directions between the actual position obtained by BPNN, SPGA-BPNN, and MPGA-BPNN and the desired position are obtained as shown in Figure 11. It is found that the average error by BPNN and SPGA-BPNN is 11 mm and 6.6 mm, respectively; however, the average error by our algorithm is 1 mm. In other words, compared with the initial positioning coordinates, BPNN and SPGA-BPNN error correction algorithm reduces position error by 57.1% and 78.6%, respectively; however, our algorithm reduces the error by 96.4%.
(a)
(b)
(c)
of the above 200 test points are selected to evaluate the accuracy of the IK algorithm. The corresponding joint angle values are obtained through the IK algorithm and brought into (3) to obtain corresponding position coordinates. Comparing with of the above 200 test points, the position errors in the x-, y-, and z-direction are obtained as shown in Figure 12. It is found that the average error is 0.035 mm.
The simulations show that the proposed localization algorithm and IK algorithm is valid. Furthermore, these proposed algorithms are applied to the developed 5-DOF upper limb prosthesis to verify the effectiveness.
4.2. Experiment Analysis of Localization Algorithm
The laser point was randomly hit 10 positions in the workspace by adjusting the head attitude, and the position coordinates of each test point were obtained by the ILA and used as input to the BPNN, SPGA-BPNN, and MPGA-BPNN. Then, the network output value by the BPNN, SPGA-BPNN, and MPGA-BPNN of each test point was obtained and compared with the corresponding actual measured values, as shown in Figure 13. It is easy to see that the output values of the MPGA-BPNN are closer to the actual measurement values.
(a)
(b)
(c)
(d)
Furthermore, through equation (15), , , , and of the test points are calculated, respectively, as shown in Table 2. The AAEs obtained by equation (19) are 48.71 mm, 18.92 mm, 11.18 mm, and 2.50 mm, respectively. Compared with the AAE of the initial position value, the AAE of the output values of the BPNN and SPGA-BPNN is reduced by 61.16% and 75.77%, respectively. However, the AAE of the output values of our algorithm is reduced by 94.87%. These results further illustrate that error correction algorithm of the MPGA-BPNN is more effective in the actual positioning of the upper limb prosthesis.
It is well known that classical visual location algorithms use point description to match features between the current view and the map. After consulting the related literature, we find that the visual positioning accuracy is approximately 5 mm. However, our positioning accuracy is 2.5 mm, which is higher than visual location algorithms. In addition, our method is an interactive localization method. In the case of multiple targets, visual localization algorithms cannot understand which random target the user is going to grasp, but our algorithm can aim the laser beam at a target and command the upper limb prosthesis to grasp it. Therefore, the proposed algorithm is more concise and accurate than visual localization algorithms.
4.3. Experiment Analysis of IK Algorithm
By inputting the output coordinates of MPGA-BPNN into GRNN solving model, the joint angles of the front three joints of the upper limb prosthesis were obtained, as shown in column 4 of Table 3. The upper limb prosthesis moves according to the obtained joint angles, and finally the gripper center accurately moves to the position of the laser point, as shown in Figure 14.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
To verify the accuracy of the joint angle solution, the joint angles solved by GRNN are substituted into equation (3) to obtain the coordinates of the center of the gripper, as shown in column 5 of Table 3. At this time, is regarded as the actual position, and is regarded as the desired position. The responses of the actual position and the desired position in the 3D coordinate system are obtained, as shown in Figure 15(a), and the projection results in the xy-, xz-, and yz-planes are illustrated in Figures 15(b)–15(d). It is easy to see that the IK algorithm can obtain a unique solution with high accuracy.
(a)
(b)
(c)
(d)
4.4. Practical Applications of Artificial Limbs
An experiment at which a cup is picked up from the table and moved to the mouth of the wearer is performed. First, the laser point is targeted to the center of the cup through wearer’s head movement. Then, through the target localization algorithm and IK algorithm, the gripper moved to the center of the cup. Finally, the human–computer interaction system was used to complete the process of gripping the cup and moving it to the mouth, as shown in Figure 16. The effectiveness of the localization algorithm and IK algorithm is further verified.
(a)
(b)
(c)
5. Conclusions and Future Work
To achieve the purpose of accurately grasping a random target with the upper limb prosthesis, a novel type of random target localization algorithm is proposed. Firstly, an initial localization algorithm (ILA) that uses two 3D attitude sensors and a laser range sensor to detect the target attitude and distance is presented. Secondly, MPGA-BPNN error correction algorithm is proposed to improve the localization accuracy of the target based on the ILA is utilized. Thirdly, GRNN algorithm is presented to calculate the prosthetic joint angles. Finally, the proposed algorithm is applied to the 5-DOF upper limb prosthesis. The simulations show that the average positioning error by MPGA-BPNN error correction algorithm is 1 mm, and the average error of the IK algorithm is 0.035 mm. Compared with ILA, the accuracy by MPGA-BPNN error correction algorithm is greatly improved and better than BPNN error correction algorithm and SPGA-BPNN error correction algorithm.
The laser test point positioning experiment shows that the AAE between the output values after error correction by MPGA-BPNN and the actual values is 2.5 mm, and the proposed IK algorithm can accurately obtain a unique IK solution. The experiment of gripping a cup from a table and moving it to the mouth of the wearer’s further verifies that the positioning algorithm and the IK algorithm can meet the requirements of the wearer.
In our future work, we will try to eliminate the restriction of the upper limb prosthesis’s DOF to meet the wearer’s need for more use. We also aim to consider using voice or brain waves to control the movement of the upper limb prosthesis.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this article.
Authors’ Contributions
All the authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by all authors. The first draft of the manuscript was written by Xinglei Zhang, and all authors commented on the previous versions of the manuscript. All authors read and approved the final version of the manuscript.
Acknowledgments
This work was supported by the Major Science and Technology Innovation Projects of Shandong Province (2017CXGC0919), the National Natural Science Foundation of China (61803235), and Key Research Programs of Shandong Province (2016GSF201197).