A novel shape matching descriptor for real-time static hand gesture recognition

Highlights

• Novel shape matching method for real-time hand gesture recognition.

• Novel shape matching problem formulation.

• Comparison of several shape matching methods.

• Novel static hand gesture matching benchmark dataset.

Abstract

The current state-of-the-art hand gesture recognition methodologies heavily rely in the use of machine learning. However there are scenarios that machine learning cannot be applied successfully, for example in situations where data is scarce. This is the case when one-to-one matching is required between a query and a dataset of hand gestures where each gesture represents a unique class. In situations where learning algorithms cannot be trained, classic computer vision techniques such as feature extraction can be used to identify similarities between objects. Shape is one of the most important features that can be extracted from images, however the most accurate shape matching algorithms tend to be computationally inefficient for real-time applications. In this work we present a novel shape matching methodology for real-time hand gesture recognition. Extensive experiments were carried out comparing our method with other shape matching methods with respect to accuracy and computational complexity. Our method outperforms the other methods and provides a good combination of accuracy and computational efficiency for real-time applications.

Introduction

Hand gestures are a fundamental aspect in human to human interaction, especially in non-verbal communications (Rautaray and Agrawal, 2015). Thus it comes as no surprise that research involving hand gestures is growing in the field of human–computer interaction. Multiple different applications can benefit through the use of communication with hand gestures such as telerobotics (Zhong et al., 2013), virtual reality and augmented reality applications, gaming (Zhu and Yuan, 2014) and sign language translation (Parton, 2006).

Classic computer vision methods along with machine learning techniques have been proposed over the years in order to facilitate human–computer interaction through the use of extracting semantic information from hand gestures. Some of the main areas of research regarding hand gestures include hand gesture detection, segmentation and recognition (Ibraheem and Khan, 2012). In recent years the main line of research that predominated the research community for computer vision problems is the use of machine learning, specifically deep learning (Krizhevsky et al., 2012).

However, the bottleneck of deep learning is its reliance on vast amount of data, which is not always available in real-world scenarios, making machine learning approaches impossible to train. A representative example could be a scenario where a dataset consists of a large number of classes (i.e. 300) and each class contains only 1 example per class.

In this work we turn our attention to a real-world research problem that can be used in practical applications, real-time hand gesture recognition. Real-time hand gesture recognition has the potential to be used in real-time virtual reality applications, telerobotics and video games. The problem we are addressing can be defined as: given a predefined dataset of $N$ hand gestures and a query hand gesture image, identify the most perceptually similar hand gesture from the dataset. Matching perceptually similar hand gestures can be used as an efficient way to retrieve further information regarding the hand gesture. For example, each hand gesture in the dataset can have additional features associated with it that are calculated offline, such as its skeleton information and its 3D shape. Using the matched hand gesture from the dataset, these features can be associated with the query hand gesture and retrieved in real-time. This process eliminates the requirement to calculate these details from scratch and is of great interest in real-time applications that require fast processing, yet they only have limited amount of computational resources, such as mobile phones.

The process of hand gesture matching is part of the complete pipeline of real-time hand gesture recognition following the hand gesture segmentation procedure (Azad et al., 2014) and alignment procedure where every hand gesture is rotated by its orientation angle, $\theta$°. Hand gesture segmentation techniques cannot provide 100% accuracy because (a) of the difficulty in segmenting the highly articulated human hands and (b) complex backgrounds and varying lighting conditions make the segmentation process challenging (Stergiopoulou et al., 2014).

In order to address the problem of imperfect segmentation we turn our attention to a class of algorithms that have lost research interest over the years because of deep learning, namely shape matching algorithms. It is hypothesized that a good shape matching algorithm can provide a robust methodology to address shape distortions and other deformations and identify similarities between shapes. Nevertheless, the most accurate shape matching algorithms currently found in literature tend to require high computational resources with non-linear matching complexities. For example the widely used shape matching method shape context has an average time matching complexity of $O\left(n^3\right)$ (Belongie et al., 2002).

In this work we propose a novel shape matching algorithm that outperforms some of the most well known shape matching algorithms. Extensive experiments were carried out on three datasets, our collected segmented hand gesture dataset, a novel version of the MPEG-7 shape matching dataset (Ralph, 1999) that is widely used for comparing shape matching algorithms and a camel silhouette dataset consisting of all the binary images from the camel class available in the MPEG-7 dataset. The MPEG-7 and camel datasets were used to investigate the transferability of our method in different shape matching setting.

To summarize, our contributions are as follows:

• We present a novel shape matching problem formulation inspired from a real-world practical application, namely hand gesture matching for real-time applications.

• We propose a novel shape matching algorithm that provides a good trade-off with respect to accuracy and computational efficiency for real-time shape matching of highly articulated shapes such as hand gestures.

• We propose a novel hand gesture dataset that can be used to compare different shape matching algorithms with respect to accuracy and computational complexity.