Abstract
With the significant increase of social informatization, the emerging information technology represented by machine vision has been applied to more and more scenes. Among them, the detection and extraction of human skeleton in a dance video based on this technology has a huge market demand in education and training. However, the existing detection and extraction technology has the problems of slow recognition speed and low extraction accuracy. Therefore, this paper proposes a neural network based on particle swarm optimization to detect and extract human skeletons in a dance video. Through the research and test on different data sets, it is found that the neural network based on particle swarm optimization algorithm has good detection and extraction ability and has high accuracy for the detection and recognition of human skeleton points. Among them, on all MPII data sets, the average accuracy of PSO-LSTM proposed in this paper is 3.9% higher than that of other optimal algorithms; on the PoseTrack data set, the average accuracy of detection and extraction is improved by 2.3%. The above results show that the neural network based on particle swarm optimization has fast detection speed and good extraction accuracy and can be used for the detection and extraction of human skeleton in a dance video.
1. Introduction
With the continuous improvement of social informatization, more and more scenes are exposed to the lens, followed by the accumulation of a large number of images and video materials. These materials contain rich data information, such as images and videos of human movement captured in different scenes and perspectives, with a large amount of data information of human skeleton detection. This information has broad application space in automatic driving, video retrieval, medical assistance, education, and teaching. For example, through the detection and extraction of human skeleton in a dance video, we can obtain the intuitive data information of dancers, which can not only assist the usual training and teaching but also be used for the evaluation basis of competition scoring. Therefore, the research of human skeleton detection based on images and videos has a lot of application space and market demand. In real life, because of the complexity of image and video acquisition scene and the diversity of human motion forms, the accuracy of human skeleton detection and extraction is difficult to meet the requirements of market application. Therefore, there is a big supply gap between the current research results of human skeleton detection and extraction and the huge market demand. Therefore, it is of great practical significance to find an efficient and accurate method for human skeleton point detection and extraction.
With the continuous development of machine learning, these researchers began to focus on the application of improved neural network, aiming to improve the accuracy of human skeleton point detection and extraction through the improvement and optimization of the algorithm. However, the diversity of human body shape, different shooting angles, and occlusion of multiple people bring great challenges to the detection and extraction of human skeleton points in natural scene images or videos. This is because the fat and thin body, the wide and tight clothes, and the different shooting angles will lead to the local deformation of the human body appearance, which will make the trained network detection model lose the ability to express the human body structure and reduce the accuracy of human skeleton point detection. The occlusion of human body will increase the problem that some pixels are difficult to match the human skeleton structure in network model detection, which makes it difficult to detect and extract a clear human skeleton point model. With the introduction of the optimized neural network, the adaptability and robustness of the detection and extraction model can be greatly enhanced, which provides a new direction for the detection and extraction of human skeleton points in complex scenes.
In order to solve the application requirements of human skeleton detection and extraction in a dance video, this paper proposes a detection and extraction model based on the PSO-enabled LSTM neural network, which aims to realize the detection and extraction of human skeleton in dance video sequences with faster speed and higher accuracy. In Section 1, the background and significance of human skeleton detection and extraction are briefly described; in Section 2, the research status of human skeleton detection and extraction is briefly introduced, the existing problems in this field are discussed, and the research work and research methods of this paper are summarized; in Section 3, first, the process of using particle swarm optimization to optimize the LSTM neural network is introduced and then the human skeleton of a dance video is detected and extracted based on the PSO-LSTM neural network model; in Section 4, the PSO-LSTM neural network and other mainstream networks are analyzed and evaluated by using the MPII data set and PoseTrack data set; Section 5 is a brief summary of the main conclusions.
2. Related Work
The earliest human skeleton model is the graphic structures (PS) proposed by McKinley’s group. The feature of this model is that it can quickly obtain the human recognition image from the image for the characterization of human structure, which has strong applicability and is classic, so that at this stage, most human skeleton extraction schemes are based on this model for subsequent improvement and development [1]. The extraction of human skeleton nodes in RGB images is mainly divided into traditional period and deep learning period. The former usually uses artificially designed features to detect key points. During this period, the commonly used feature extraction methods include HOG, shape content descriptor [2], and multimethod synthesis [3]; the tree structure model is used to model recognition features [4–6]. In the period of deep learning, because of the improvement of computer computing ability, the convolutional neural network as the representative of deep learning develops rapidly, so it is often used in image feature extraction. The advantage of this method is that people do not need to manually design the scheme and extract features, but automatically learn and extract the required features from the given target.
In the research of human skeleton extraction based on deep learning, most of them have used the image as the information source carrier. Andreassen et al. proposed the DeepPose method for single person skeleton extraction for the first time, which has become a sign that human skeleton extraction has changed from traditional method to deep learning direction [7]. Then, based on the DeepPose method, Hewamalage proposed a new idea of learning according to the relationship between joint points [8], but this method has great difficulties in labeling the coordinates of human key points. In order to obtain high-precision coordinates of key points of human body, Zenke and Vogels improved this method [9]. They regarded attitude estimation as a detection problem and finally obtained thermal map, which had better recognition effect. Johnson et al. proposed the CPM method based on sequential convolution structure, which has strong robustness and excellent performance [10]. In the same year, Qi innovatively proposed the bottom-up multiperson skeleton extraction scheme DeepCut, which also marks a long-term exploration on the single person skeleton extraction scheme. Researchers began to shift the research focus to the multiperson skeleton extraction scheme [11]. On this basis, many scholars improved and optimized the scheme [12, 13]. In addition to bottom-up, fruitful research results have been achieved in the top-down field, such as local extraction method [14], mask CNN [15], and cascade pyramid model [16].
To sum up, although many researchers have achieved fruitful results in the research of human skeleton extraction based on deep learning, there are still many problems to be solved about the detection and extraction scheme of human skeleton; the most significant of which is that the implementation effect of human skeleton detection and extraction based on videos is still not ideal. Videos contain more time domain information than pictures, so the video-based detection and extraction model has higher requirements for detection speed; at the same time, the instability of the video shooting scene will lead to camera shaking, motion blur, and so on, which increase the difficulty of video recognition. Nowadays, MHV [17], hidden Markov model [18], and so on are commonly used in video-based detection and extraction, but the detection effect is generally poor. In order to improve the detection speed and extraction accuracy of the model, this paper introduces the PSO-enabled LSTM neural network and takes the dance video as the detection carrier to realize the detection and extraction of human skeleton based on videos [19].
3. Human Skeleton Detection and Extraction in Dance Video Based on PSO-Enabled LSTM Neural Network
3.1. Long Short-Term Memory Neural Network Model Based on Particle Swarm Optimization
Particle swarm optimization (PSO) is a population-based random search optimization algorithm based on the foraging behavior of birds. It tracks the individual local and global optimal solutions through the defined fitness function:
In formulae (1) and (2), is the number of iterations; is the inertia factor; is the acceleration factor of particles, which is generally positive; is a random number between [0, 1]; and are the position and velocity of the velocity vector of the i-th particle in the d-dimensional component in the k-th iteration, respectively; is the d-dimensional component of the individual optimal solution of the i-th particle; and is the d-dimensional component of the group optimal solution.
Figure 1 shows the structure diagram of a two-layer long short-term memory (LSTM) neural network. The LSTM network is a kind of gated recurrent neural network [20]. It can enhance the transmission and communication of information between cells through the gate structure, which can effectively avoid the problems of gradient disappearance and gradient explosion in traditional networks. In addition to the network structure, the model parameters also have an important impact on the performance of LSTM. The traditional parameter determination is affected by human experience, which will lead to the difference between the prediction accuracy and the ideal accuracy, and it is difficult to give full play to the due performance of the model. The particle swarm optimization algorithm can find the optimal parameters of LSTM model in the iterative process, greatly reduce the intervention of human experience on the output results of the model, and give better play to the excellent performance of the model. In view of this, this paper uses the particle swarm optimization algorithm to optimize the long-term and short-term memory neural network, finds the optimal parameters of LSTM model in the iterative process through the particle swarm optimization algorithm, then establishes the LSTM model according to the optimized parameters to learn the temporal information in the video, and completes the extraction of human bone frame points based on the video.
Figure 2 shows the flowchart of using PSO algorithm to update internal weights for LSTM. The key parameters of the neural network model based on particle swarm optimization include the number of hidden layer neurons m and the learning rate LR. The specific process of the algorithm design optimization model is as follows [21]:(1)For the normalization of input data, because the LSTM neural network is sensitive to the size of input data [22–29], too large data scale will affect the model training effect, so it is necessary to ensure that the scale of input data is consistent. In equation (3), is the original input data; and are the maximum and minimum values of the original data, respectively; and is the normalized data.(2)The parameters of particle swarm optimization are initialized to keep the number of live population pop, the maximum number of iterations Tmax, the learning factor, and the range of particle position and velocity.(3)For the number of hidden layer neurons m and learning rate lr, the initial value is set according to experience, and the LSTM model is established and trained: In equation (4), P and Q are the number of training samples and test samples, respectively; and are the real and predicted values of training samples, respectively; and are the real value and predicted value of the test sample, respectively. In the particle swarm optimization algorithm, the relative error between the real value and the predicted value of the training set sample or the relative error between the real value and the predicted value of the test set sample is usually taken as the fitness function, and the model takes the average of the two. Bing can verify whether the model is overfitted at the same time.(4)The global optimal position and local optimal position are determined by the initial fitness value of the particle, and they are set as the historical optimal position. According to the algorithm, the velocity and position of the particles are updated and the fitness of the corresponding particles is calculated and compared with the local and global optimal solutions.(5)Judge whether the fitness of particles tends to be stable or reaches the maximum, if so, assign the optimal parameters to the PSO-LSTM model and repeat Step 4.(6)The PSO-LSTM model constructed by the optimal parameters processes the input data and analyzes and summarizes the output results according to the evaluation indexes.
3.2. Human Skeleton Detection and Extraction Based on PSO-LSTM Optimization Algorithm
The detection and extraction of human skeleton in a dance video are usually carried out in time domain and spatial domain, in which the spatial scene contained in a single video frame is spatial information and the target motion information carried between frames is temporal information. At present, the output results of neural networks commonly used in video detection and extraction are from the output layer to the hidden layer and then back to the output layer, and the layers are fully connected or partially connected. However, the nodes between layers are not connected, which causes the neural network to ignore the contact information between multiple frames in the time domain, affecting the final detection and extraction accuracy. Therefore, this paper introduces particle swarm optimization long short-term memory (PSO-LSTM) to learn the information in time domain to enhance the detection and extraction of human skeleton based on dance videos.
Figure 3 is a schematic diagram of human skeleton extraction in a dance video based on the PSO-LSTM neural network model. First, we need to obtain the dance video with human skeleton features and establish the dance image samples; second, the data image samples and key point labeling labels for neural network model training are obtained; Then, a confidence map is generated for each human body K in the image:
In equation (5), p is the pixel in the dance picture, j is the body part of the human body in the picture, is the true value part of the j part of the human body k in the picture, and is the peak control height.
Equation (6) is the confidence map of the body part distribution at the pixel obtained from the maximum aggregation of the confidence map of all parts, where k is the human body in the annotation map, j is the body part, and p is the pixel position. Then, the true value part association vector field of each pixel p is defined as
In equation (7), c is the human body limb, k is the human body, and is the calculation formula of unit vector from body part j1 to body part j2 in the limb direction, which is expressed as follows:
In equation (8), and denote the positions of the upper part of the limb at position j1 and position j2. The point set in the position correlation vector field is composed of the points existing on the limb, that is, p points satisfy the following conditions:
In equation (9), is the limb width in pixels, is the limb length, and is the vector perpendicular to . Finally, by averaging the correlation fields of all human bodies, the true value part correlation field is obtained:
In equation (10), represents the number of vectors in all k human bodies whose point p is not zero, that is, the average pixels of overlapping limbs of different individuals. Then, the sample feature map F is obtained by processing the image samples of the dance video, and the feature map F is input into the PSO-LSTM neural network to obtain the body candidate Dj and the position correlation field Lc of the predicted image:
In equation (11), J is the total number of body parts, j is the body part, Nj is the candidate number of body part j, m is the candidate number of body part j, and is the position of body part j selected for the m-th body detection.
According to the detection results in the previous step, the predicted correlation fields of the two candidate parts are sampled along the line segment, and the linear integral is calculated to evaluate the possibility of the two parts being connected:
In equation (12), represents the probability of connecting the m-th detection candidate body part j1 with the n-th detection candidate body part j2, u is between 0 and 1, represents the position correlation field of the parts j1 and j2 at point , represents any point on the line between and , and the value of satisfies the following conditions:
The variable is defined to indicate whether the two detection parts and are connected; let , where is the candidate number of body part j1 and is the candidate number of body part j2. The least edge is selected to connect all candidate parts into a growth tree, and then the part connection problem is reduced to the maximum weight bipartite graph matching problem, where the graph node is the body part detection candidate parts and the edge is the possible connection between a pair of candidate parts. Finally, the optimal connection between the two body parts is achieved by finding a match with the maximum weight for the selected edge:
In the formula, indicates whether the candidate parts and are connected; indicates the integration result of and ; indicates the matching set of body parts j1 and j2 to form limb c; m and n represent a part in the set and , respectively, so as to limit two limbs of the same type from appearing in the same body; and represents the best set of all human limbs c calculated based on the position correlation field.
Finally, all the predicted body parts are matched and the skeletons with the same candidate parts are connected to get a complete human skeleton image, which completes the detection and extraction of human skeleton in a dance video.
4. Application Effect of PSO-LSTM Neural Network Based on Particle Swarm Optimization
4.1. Test Results Based on MPII Data Set
In order to verify the effectiveness of PSO-LSTM neural network in human skeleton detection and extraction, the public MPII image data set [30] is selected and the individual data part containing different behavior patterns is labeled as the test data set. In the experiment of human skeleton detection and extraction, the detection accuracy of head, shoulder, elbow, wrist, hip, knee, and ankle is usually selected to score, while the comprehensive score of the whole model is usually based on MAP (mean average precision).
As shown in Figure 4, the recognition rates of LSTM and PSO-LSTM based on particle swarm optimization are tested on MPII 288 testing images and MPII full set, respectively. It can be clearly seen that the current human skeleton detection algorithm has good recognition effect on the head, shoulder, and elbow. The recognition effect of the wrist, knee, and ankle is poor. After using the particle swarm optimization algorithm, the recognition effect of axis, wrist, and ankle skeleton points can be significantly improved. The comprehensive evaluation index map of the model is improved by 0.2% and 0.4%, respectively, on different test data sets. This proves the effectiveness of PSO neural network in human skeleton point recognition.
As shown in Figure 5, on the basis of MPII 288 test image data set, PSO-LSTM based on particle swarm optimization and Varadarajan, DeeperCut, Iqbal, and DeepCut, which are the mainstream used for human-machine skeleton detection and recognition, are compared and tested. The results show that PSO-LSTM is superior to other algorithms in the detection of each human joint point. The test result of comprehensive evaluation index MAP is 79.3%, which is 3.2% higher than 76.1% of other optimal algorithms.
As shown in Figure 6, on the basis of the whole test image data set of MPII, the PSO-LSTM based on particle swarm optimization is compared with Varadarajan, DeeperCut, and Iqbal algorithms. The results show that the recognition rate of PSO-LSTM in the head node is slightly lower than that of Varadarajan, and it is better than other algorithms in the detection and recognition of other human joint points. The test result of comprehensive evaluation index map is 76.1%, which is 3.9% higher than 72.2% of other optimal algorithms.
4.2. Test Results Based on PoseTrack Data Set
In order to verify the adaptability of PSO-LSTM neural network in human skeleton detection and extraction, we choose the PoseTrack data set for the evaluation test. The test data set includes 550 videos of 66374 frames, which are divided into 292 training videos, 50 verification videos, and 208 test videos. The video length is mostly between 41 and 151 frames, and it takes about five seconds.
In Figure 7, the annotation statistics of PoseTrack is shown. In order to test the stability of tracking body joints and the ability of long-term tracking body joints, dense annotation needed every four frames in the verification and test data set, with a total of 23000 annotation frames and 153615 annotation poses. In addition, in order to increase the generalization ability of the model, the images are rotated randomly between −40°C and 40°C, and the PoseTrack data set is expanded and enhanced with random size of 0.8–1.2.
As can be seen from Figure 8, the PSO-LSTM neural network based on particle swarm optimization on PoseTrack data set also has excellent detection and extraction performance, and the detection and recognition accuracy of human joint points is also higher than that of the network without particle swarm optimization. Compared with the LSTM network, its comprehensive evaluation index is increased from 40.2% to 43.4%. In order to explore the difference between it and the current mainstream model detection ability, two kinds of networks with better detection effect, YOLO and SSD, are used as control experiments. The detection results on PoseTrack data set are shown in Figure 9.
As can be seen from Figure 9, the overall recognition accuracy of SSD network for human joint points is better than that of YOLO; the accuracy of PSO-LSTM neural network in human joint point recognition is better than other networks, which shows that the particle swarm optimization algorithm can effectively improve the accuracy of neural network in human joint point detection and recognition.
In Figure 10, the MAP scores of different network detectors on PoseTrack data set are compared. In the PoseTrack data set, the diagonal line of 20% head detection frame is used as the measurement standard, and it is specified that the detection is correct for the point with occlusion but uncertainty. The results show that the overall accuracy of PSO-LSTM is higher than other network models. Compared with the SSD network model, the MAP of PSO-LSTM is 2.3% higher.
Through the evaluation and test of the model on MPII and PoseTrack data sets, the results show that the recognition effect of the model is good when the number of people in the video is small and the limbs are unobstructed; when there are too many people in the video and the degree of limb overlap is high, the recognition effect of the model will be affected to some extent. At the same time, when the video is too long and there are multiple shots switching each other, the effect of the model will also be affected. Therefore, the model needs to be further optimized to achieve the adaptability to multiperson long-time video.
5. Conclusion
For the problems of slow detection speed and low extraction accuracy in human skeleton detection and extraction of a dance video, this paper proposes a neural network based on particle swarm optimization, which finds the optimal parameters of LSTM neural network in the iterative process through the particle swarm optimization algorithm, then uses the PSO-LSTM model established according to the optimized parameters to learn the temporal information in the video, and completes the extraction of human skeleton points based on videos. The PSO-LSTM model based on particle swarm optimization has good detection and recognition ability. Through the comparative test between the PSO-LSTM model and current mainstream algorithms in different data sets, the results show that the PSO-LSTM model has high accuracy in the detection and recognition of human skeleton points. The average accuracy of PSO-LSTM is 3.9% higher than that of other optimal algorithms. On the PoseTrack data set, the average accuracy of detection and extraction is improved by 2.3%. The above results show that the neural network based on particle swarm optimization can significantly improve the detection speed and extraction accuracy and can be used for the detection and extraction of human skeleton in a dance video.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Disclosure
This research was performed as part of the authors’ employment under Zhengzhou University of Aeronautics.
Conflicts of Interest
The authors declare that they have no known conflicts of interest or personal relationships that could have appeared to influence the work reported in this paper.