A multi-scale descriptor for real time RGB-D hand gesture recognition

Highlights

• A new RGB-D shape-based hand gesture recognition method is proposed.

• A new hand shape descriptor is proposed with emphasized finger feature.

• Different recognition engines are explored and compared for different applications.

• Sota accuracies are achieved on several benchmarks, as well as excellent efficiency.

• A demo video is given for a real life application of human-computer interaction.

Abstract

The development of depth cameras, e.g., the Kinect sensor, provides new opportunities for human computer interaction (HCI). Although the Kinect sensor has been extensively applied for human tracking, human action recognition and hand gesture recognition, real time hand gesture recognition is still a challenging problem. In this paper, a new real time hand gesture recognition method is proposed. Since fingers are the most important clue for hand gesture classification, a finger-emphasized multi-scale descriptor is proposed. The proposed descriptor incorporates three types of parameters of multiple scales to make a discriminative representation of the hand shape. Furthermore, the features of fingers are emphasized for hand gesture analysis. Three solutions to hand gesture recognition are then investigated with DTW, SVM, and neural network. Extensive experiments are conducted and the results show that the proposed method is robust to noise, articulations and rigid transformations. The comparison with state-of-the-art methods verifies the accuracy and efficiency of our method.

Introduction

Hand gesture recognition has been an important topic in computer vision for its extensive applications in human-computer interaction (HCI), including virtual reality, sign language recognition and computer games [1]. Adopting hand gesture as an interface allows communications and manipulations in the non-contact environments. Traditional methods attach sensors or markers to the fingers, e.g., the data gloves [2], [3], to capture hand gestures via electro-mechanical or magnetic sensing. These methods are effective in providing complete and real-time measurements of hand gestures, however, they hinder the natural motion of hand and are unapplicable in non-contact environments. Moreover, the devices are expensive for casual use and require complex calibration.

The vision-based hand gesture recognition methods [4], [5], [6], [7], [8] give an alternative solution to the problems, which can be used naturally in non-contact environments. However, due to the limitations of the optical sensors, the captured images are sensitive to lighting conditions and cluttered backgrounds. Thus these methods usually cannot detect and track the hand robustly. Therefore, the traditional vision-based methods are far from satisfactory for real-life applications.

With the development of the depth cameras, e.g., the Kinect sensor [9], hand gesture recognition can be explored in a new form. The hand gesture can be detected and recognized by the fusion of both the color image and depth map. However, the hand occupies a small area of the image with significant articulations, noises and distortions, which affects the recognition result. The classic shape recognition methods, e.g., the shape-context-based methods [10], [11] and the skeleton-based methods [12], [13], cannot recognize hand gestures robustly under severe articulations and distortions. The part-based methods [14], [15] were proposed to solve these problems, but they cannot capture complete hand shape features for sufficient robustness and accuracy. Some recent contour based methods [16] represent shapes with both local and global features in order to capture full shape information, but these methods are not so efficient for real time applications. It is still a challenging problem to use depth sensor for real time hand gesture recognition.

In this work, a new real time hand gesture recognition method is proposed. Inspired by the invariant multi-scale shape descriptor IMD [16], we propose a finger-emphasized multi-scale descriptor (FMD) for hand shape representation. Since the finger features are the most important clue for hand gesture classification, the shape features of fingers are emphasized in FMD, which makes the hand shape representation discriminative. The extracted invariant shape features are robust to rigid transformation, articulation and noise. Moreover, the multi-scale representation with both local and semi-global shape features makes the FMD a complete descriptor. The proposed hand pose representation can be combined with traditional classification methods for sequential data, such as Dynamic Time Warping (DTW), Support Vector Machine (SVM), and Back Propagation Neural Network (BPNN), for various applications.

Fig. 1 shows the framework of the proposed real time hand gesture recognition system. The Kinect sensor is used to capture both the color image and depth map of hand gestures as input. Hand is detected and segmented from the cluttered background. The hand shape is then represented by the proposed FMD descriptor, and recognized by the recognition methods.

Extensive experimental results validate that our method is robust to noise, articulated variation and rigid transformation. The recognition accuracy is evaluated on the latest challenging hand gesture datasets [14], [17], [18], and our method outperforms state-of-the-art methods a lot. The mean running time of our method based on BP neural network is less than 1ms, which supports real time applications. Moreover, the real time hand gesture recognition is implemented and the demo is shown in the supplementary material.

After a brief review of the related work in Section 2, the hand gesture detection is introduced in Section 3. Section 4 describes the FMD descriptor and the recognition engines are presented in Section 5. The experimental results and empirical evaluations are given in Section 6. This work is concluded in Section 7. This paper is an extension of the conference paper [19].