Head-mounted device-based human-computer interaction often requires egocentric recognition of hand gestures and fingertips detection. In this paper, a unified approach of egocentric hand gesture recognition and fingertip detection is introduced. The proposed algorithm uses a single convolutional neural network to predict the probabilities of finger class and positions of fingertips in one forward propagation. Instead of directly regressing the positions of fingertips from the fully connected layer, the ensemble of the position of fingertips is regressed from the fully convolutional network. Subsequently, the ensemble average is taken to regress the final position of fingertips. Since the whole pipeline uses a single network, it is significantly fast in computation. Experimental results show that the proposed method outperforms the existing fingertip detection approaches including the Direct Regression and the Heatmap-based framework. The effectiveness of the proposed method is also shown in-the-wild scenario as well as in a use-case of virtual reality.

基于头戴式设备的人机交互通常需要以自我为中心的手势识别和指尖检测。在本文中，介绍了一种以自我为中心的手势识别和指尖检测的统一方法。所提出的算法使用单个卷积神经网络来预测一次前向传播中手指类别和指尖位置的概率。指尖位置的集合不是从全连接层直接回归指尖位置，而是从全卷积网络回归。随后，采用整体平均值来回归指尖的最终位置。由于整个管道使用单个网络，因此计算速度非常快。实验结果表明，所提出的方法优于现有的指尖检测方法，包括直接回归和基于热图的框架。所提出方法的有效性也在野外场景以及虚拟现实用例中得到了证明。