This paper presents a sign recognition system for a sign tutoring assistive humanoid robot. In this study, a specially designed 5-fingered robot platform with expressive face (Robovie R3) is used for interaction and communication with deaf or hard of hearing children using signs and visual cues. The robot is able to recognize and generate accurately a selected set of signs from Turkish sign language using various hand, arm and head gestures as relevant feedback. This paper focuses on the sign recognition system of the robot to recognize the human participant’s signing during the interaction. The system is based on two different approaches including a conventional method involving artificial neural network combined with hidden Markov model and a deep learning based method involving long short-term memory. The system is tested both on offline and real-time settings within an interaction game scenario with deaf or hard of hearing children. During the study, besides testing the sign recognition system, participants’ subjective evaluations and impressions were also collected and examined. The robot is perceived as likable and intelligent by the children, based on the questionnaires; and the proposed sign recognition system enables robust real-time interaction and communication of the assistive robot with children in sign language.

中文翻译：

---

儿童辅助机器人指示牌的指示牌识别系统

本文提出了一种用于手势指导辅助类人机器人的手势识别系统。在这项研究中，专门设计的具有表情表情的五指机器人平台（Robovie R3）用于通过手势和视觉提示与聋哑或重听儿童进行互动和交流。机器人能够使用各种手势，手臂和头部手势作为相关反馈，从土耳其手语中识别并准确生成一组选定的手语。本文着重研究机器人的手势识别系统，以在交互过程中识别参与者的签名。该系统基于两种不同的方法，包括使用人工神经网络结合隐马尔可夫模型的传统方法和使用长短期记忆的基于深度学习的方法。该系统已在具有聋哑或重听儿童的互动游戏场景中的离线和实时设置下进行了测试。在研究过程中，除了测试符号识别系统外，还收集并检查了参与者的主观评价和印象。根据问卷调查，机器人被孩子们视为可爱和聪明。提出的手势识别系统可以使辅助机器人与儿童以手势语言进行强大的实时交互和通信。根据问卷调查；提出的手势识别系统可以使辅助机器人与儿童以手势语言进行强大的实时交互和通信。根据问卷调查；提出的手势识别系统可以使辅助机器人与儿童以手势语言进行强大的实时交互和通信。