当前位置: X-MOL 学术arXiv.cs.IR › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Sign and Search: Sign Search Functionality for Sign Language Lexica
arXiv - CS - Information Retrieval Pub Date : 2021-07-28 , DOI: arxiv-2107.13637
Manolis Fragkiadakis, Peter van der Putten

Sign language lexica are a useful resource for researchers and people learning sign languages. Current implementations allow a user to search a sign either by its gloss or by selecting its primary features such as handshape and location. This study focuses on exploring a reverse search functionality where a user can sign a query sign in front of a webcam and retrieve a set of matching signs. By extracting different body joints combinations (upper body, dominant hand's arm and wrist) using the pose estimation framework OpenPose, we compare four techniques (PCA, UMAP, DTW and Euclidean distance) as distance metrics between 20 query signs, each performed by eight participants on a 1200 sign lexicon. The results show that UMAP and DTW can predict a matching sign with an 80\% and 71\% accuracy respectively at the top-20 retrieved signs using the movement of the dominant hand arm. Using DTW and adding more sign instances from other participants in the lexicon, the accuracy can be raised to 90\% at the top-10 ranking. Our results suggest that our methodology can be used with no training in any sign language lexicon regardless of its size.

中文翻译:

手语和搜索:手语词典的手语搜索功能

手语词典是研究人员和学习手语的人的有用资源。当前的实现允许用户通过其光泽或通过选择其主要特征(例如手形和位置)来搜索标志。本研究侧重于探索反向搜索功能,用户可以在该功能中在网络摄像头前签署查询标志并检索一组匹配标志。通过使用姿势估计框架 OpenPose 提取不同的身体关节组合(上半身、惯用手的手臂和手腕),我们比较了四种技术(PCA、UMAP、DTW 和欧几里德距离)作为 20 个查询标志之间的距离度量,每个由 8 个参与者执行在 1200 符号词典上。结果表明,UMAP 和 DTW 可以分别以 80\% 和 71\% 的准确率预测使用优势手手臂运动的前 20 个检索到的标志的匹配标志。使用 DTW 并在词典中添加更多其他参与者的符号实例,在前 10 名排名中准确率可以提高到 90\%。我们的结果表明,无论大小如何,我们的方法都可以在没有任何手语词典训练的情况下使用。
更新日期:2021-07-30
down
wechat
bug