American sign language recognition is still a research focus in computer vision community. Recently, most researches mainly extract low-level features for hand gesture recognition. These approaches perform poorly on recognizing gestures posed like a fist. In this paper, we propose a novel multimodal framework for sign language recognition system which exploits the Leap Motion Controller (LMC) and a webcam. We compute two sets of features. The first set is the angles at hand joints acquired by the LMC sensor. When, hand poses like a fist, the positions of the thumb joints captured by the LMC are not very precise. So, we should incorporate the second set of features extracted from the hand shape contour provided by a webcam. In this paper, we introduce a new mid-level feature, called Contour Segment Code (CSC), to represent hand shape contour. The proposed shape representation, first, extracts meaningful landmarks from the hand shape contour. CSC then encodes different segments of the hand contour into a code based on the shape landmarks. The extracted landmarks precisely determine the hand direction. The proposed method is tested by creating a very challenging dataset composed of 64,000 samples. Our experiments study the performance of the LMC and characteristics of CSC in different scenarios. The experimental results demonstrate the privileged performance of the proposed method against the systems which use depth images.

美国手语识别仍然是计算机视觉界的研究重点。近来，大多数研究主要提取低级特征用于手势识别。这些方法在识别像拳头一样的手势时效果不佳。在本文中，我们提出了一种新颖的手势语言识别系统的多模式框架，该框架利用了Leap Motion控制器（LMC）和网络摄像头。我们计算两组特征。第一组是由LMC传感器获取的手关节角度。当手像拳头一样摆姿势时，LMC捕获的拇指关节的位置不是很精确。因此，我们应该合并从网络摄像头提供的手形轮廓中提取的第二组特征。在本文中，我们介绍了一个新的中级功能，称为轮廓线段代码（CSC），用于表示手形轮廓。首先，提出的形状表示从手形轮廓提取有意义的界标。然后，CSC根据形状界标将手轮廓的不同部分编码为代码。提取的地标可精确确定手的方向。通过创建一个由64,000个样本组成的非常具有挑战性的数据集，对提出的方法进行了测试。我们的实验研究了在不同情况下LMC的性能和CSC的特性。实验结果证明了该方法相对于使用深度图像的系统具有优越的性能。通过创建一个由64,000个样本组成的非常具有挑战性的数据集，对提出的方法进行了测试。我们的实验研究了在不同情况下LMC的性能和CSC的特性。实验结果证明了该方法相对于使用深度图像的系统具有优越的性能。通过创建一个由64,000个样本组成的非常具有挑战性的数据集，对提出的方法进行了测试。我们的实验研究了在不同情况下LMC的性能和CSC的特性。实验结果证明了该方法相对于使用深度图像的系统具有优越的性能。