The hand gesture recognition system is a noncontact and intuitive communication approach, which, in turn, allows for natural and efficient interaction. This work focuses on developing a novel and robust gesture recognition system, which is insensitive to environmental illumination and background variation. In the field of gesture recognition, standard vision sensors, such as CMOS cameras, are widely used as the sensing devices in state-of-the-art hand gesture recognition systems. However, such cameras depend on environmental constraints, such as lighting variability and the cluttered background, which significantly deteriorates their performances. In this work, we propose an event-based gesture recognition system to overcome the detriment constraints and enhance the robustness of the recognition performance. Our system relies on a biologically inspired neuromorphic vision sensor that has microsecond temporal resolution, high dynamic range, and low latency. The sensor output is a sequence of asynchronous events instead of discrete frames. To interpret the visual data, we utilize a wearable glove as an interaction device with five high-frequency (>100 Hz) active LED markers (ALMs), representing fingers and palm, which are tracked precisely in the temporal domain using a restricted spatiotemporal particle filter algorithm. The latency of the sensing pipeline is negligible compared with the dynamics of the environment as the sensor’s temporal resolution allows us to distinguish high frequencies precisely. We design an encoding process to extract features and adopt a lightweight network to classify the hand gestures. The recognition accuracy of our system is comparable to the state-of-the-art methods. To study the robustness of the system, experiments considering illumination and background variations are performed, and the results show that our system is more robust than the state-of-the-art deep learning-based gesture recognition systems. Note to Practitioners—This article addresses the robustness of the hand gesture recognition system that is important for gesture recognition-based applications. Existing methods rely on either the large-volume data to train a deep learning model or to restrict the applied environments (e.g., an ideal environment without dynamic background). However, a vision-based deep learning model requires large computational resources, while the ideal environment limits the practicality of the system. In this work, we introduce a biologically inspired neuromorphic vision sensor and an ALM glove and build a novel gesture recognition system to tackle the above issue. The neuromorphic vision sensor has a microsecond temporal resolution and a high dynamic range. With these properties, the sensing system of our prototype operates in a very low-latency space, which, in turn, ensures that our gesture recognition system is robust to illumination variance and dynamic background. Thus, this work is valuable to the research of illumination-robust gesture recognition systems. Preliminary experiments suggest that our system prototype is feasible, but it has not yet been incorporated into an online gesture recognition system nor tested with complex gestures. In future work, we will concentrate on the improvement of the signal processing methods that advance the current system to complex and practical applications.

中文翻译：

---

具有基于事件的神经形态视觉传感器的新型照明鲁棒手势识别系统


手势识别系统是一种非接触式、直观的通信方式，从而实现自然、高效的交互。这项工作的重点是开发一种新颖且强大的手势识别系统，该系统对环境照明和背景变化不敏感。在手势识别领域，标准视觉传感器（例如 CMOS 相机）被广泛用作最先进的手势识别系统中的传感设备。然而，此类相机依赖于环境限制，例如照明变化和杂乱的背景，这会显着降低其性能。在这项工作中，我们提出了一种基于事件的手势识别系统，以克服不利约束并增强识别性能的鲁棒性。我们的系统依赖于受生物启发的神经形态视觉传感器，该传感器具有微秒时间分辨率、高动态范围和低延迟。传感器输出是一系列异步事件而不是离散帧。为了解释视觉数据，我们利用可穿戴手套作为交互设备，具有五个高频 (>100 Hz) 有源 LED 标记 (ALM)，代表手指和手掌，使用受限时空粒子在时域中精确跟踪这些标记过滤算法。与环境动态相比，传感管道的延迟可以忽略不计，因为传感器的时间分辨率使我们能够精确地区分高频。我们设计了一个编码过程来提取特征并采用轻量级网络对手势进行分类。我们系统的识别精度可与最先进的方法相媲美。 为了研究系统的鲁棒性，进行了考虑照明和背景变化的实验，结果表明我们的系统比最先进的基于深度学习的手势识别系统更鲁棒。从业者注意事项——本文讨论了手势识别系统的鲁棒性，这对于基于手势识别的应用程序非常重要。现有方法要么依赖大量数据来训练深度学习模型，要么限制应用环境（例如，没有动态背景的理想环境）。然而，基于视觉的深度学习模型需要大量的计算资源，而理想的环境限制了系统的实用性。在这项工作中，我们引入了一种受生物学启发的神经形态视觉传感器和 ALM 手套，并构建了一种新颖的手势识别系统来解决上述问题。神经形态视觉传感器具有微秒时间分辨率和高动态范围。凭借这些特性，我们原型的传感系统可以在非常低的延迟空间中运行，这反过来又确保了我们的手势识别系统对照明变化和动态背景具有鲁棒性。因此，这项工作对于光照鲁棒手势识别系统的研究具有重要意义。初步实验表明我们的系统原型是可行的，但尚未合并到在线手势识别系统中，也没有进行复杂手势的测试。在未来的工作中，我们将集中精力改进信号处理方法，将当前系统推向复杂和实际的应用。