Lei Zhang
Skip Abstract Section
Abstract
In this article, we propose WiSign that recognizes the continuous sentences of American Sign Language (ASL) with existing WiFi infrastructure. Instead of identifying the individual ASL words from the manually segmented ASL sentence in existing works, WiSign can automatically segment the original channel state information (CSI) based on the power spectral density (PSD) segmentation method. WiSign constructs a five-layer Deep Belief Network (DBN) to automatically extract the features of isolated fragments, and then uses the Hidden Markov Model (HMM) with Gaussian mixture and Forward-Backward algorithm to recognize sign words. In order to further improve the accuracy, WiSign also integrates the language model N-gram, which uses the grammar rules of ASL to calibrate the recognized results of sign words. We implement a prototype of WiSign with commercial WiFi devices and evaluate its performance in real indoor environments. The results show that WiSign achieves satisfactory accuracy when recognizing ASL sentences that involve the movements of the head, arms, hands, and fingers.
- Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras. 2015. WiGest: A ubiquitous WiFi-based gesture recognition system. In IEEE Conference on Computer Communications.Google Scholar
- Fadel Adib, Zachary Kabelac, Dina Katabi, and Robert C. Miller. 2014. 3D tracking via body radio reflections. In 11th USENIX Symposium on Networked Systems Design and Implementation.Google Scholar
- Fadel Adib and Dina Katabi. 2013. See through walls with WiFi! Computer Communication Review 43, 4 (2013), 75--86.Google Scholar
- Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury, Kevin Caves, and Frank Deruyter. 2011. Using mobile phones to write in air. In International Conference on Mobile Systems, Applications, and Services.Google ScholarDigital Library
- Kamran Ali, Alex X. Liu, Wei Wang, and Muhammad Shahzad. 2015. Keystroke recognition using WiFi signals. In International Conference on Mobile Computing and NETWORKING.Google ScholarDigital Library
- Marcell Assan and Kirsti Grobel. 1997. Video-based Sign Language Recognition Using Hidden Markov Models. 97--109.Google Scholar
- Bo Chen, Vivek Yenamandra, and Kannan Srinivasan. 2015. Tracking keystrokes using wireless signals. In International Conference on Mobile Systems, Applications, and Services.Google ScholarDigital Library
- Hanyi Dai and Gong Zhang. 2014. The research and tests of Wi-Fi interference based on IEEE 802.11n CSI-tool. In Informatization Research.Google Scholar
- Emre Ertin, Nathan Stohs, Santosh Kumar, Andrew Raij, Mustafa Al’Absi, and Siddharth Shah. 2011. AutoSense: Unobtrusively wearable sensor suite for inferring the onset, causality, and consequences of stress in the field. In ACM Conference on Embedded Networked Sensor Systems.Google ScholarDigital Library
- Gaolin Fang, Wen Gao, and Debin Zhao. 2007. Large-vocabulary continuous sign language recognition based on transition-movement models. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 37, 1 (2007), 1--9.Google ScholarDigital Library
- Wen Gao, Gaolin Fang, Debin Zhao, and Yiqiang Chen. 2004. A Chinese sign language recognition system based on SOFM/SRN/HMM. Pattern Recognition 37, 12 (2004), 2389--2402.Google ScholarDigital Library
- Kazuyuki Imagawa, Hideaki Matsuo, Rinichiro Taniguchi, Daisaku Arita, Shan Lu, and Seiji Igi. 2000. Recognition of local features for camera-based sign language recognition system. In International Conference on Pattern Recognition.Google ScholarCross Ref
- Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In Usenix Conference on Networked Systems Design and Implementation.Google Scholar
- Hamed Ketabdar and Mehran Roshandel. 2010. Towards using embedded magnetic field sensor for around mobile device 3D interaction. In International Conference on Human Computer Interaction with Mobile Devices and Services.Google ScholarDigital Library
- Simon Lang, Marco Block, and Raúl Rojas. 2012. Sign language recognition using Kinect. In International Conference on Artificial Intelligence and Soft Computing.Google ScholarDigital Library
- Hong Li, Wei Yang, Jianxin Wang, Yang Xu, and Liusheng Huang. 2016. WiFinger: Talk to your smart devices with finger-grained gesture. In ACM International Joint Conference on Pervasive and Ubiquitous Computing.Google ScholarDigital Library
- Mohandes M., Aliyu S., and Deriche M. 2014. Arabic sign language recognition using the leap motion controller. In IEEE International Symposium on Industrial Electronics.Google Scholar
- Yongsen Ma, Gang Zhou, shuangquan Wang, Hongyang Zhao, and Woosub Jung. 2018. SignFi: Sign language recognition using WiFi. In ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.Google Scholar
- Pedro Melgarejo, Xinyu Zhang, Parameswaran Ramanathan, and David Chu. 2014. Leveraging directional antenna capabilities for fine-grained gesture recognition. In 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing.Google ScholarDigital Library
- Rajalakshmi Nandakumar, Bryce Kellogg, and Shyamnath Gollakota. 2014. Wi-Fi gesture recognition on existing devices. Eprint Arxiv 3, 2 (2014), 17--17.Google Scholar
- Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. E-Gesture: A collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In International Conference on Mobile Systems, Applications, and Services.Google ScholarDigital Library
- Lionel Pigou, Sander Dieleman, Pieter Jan Kindermans, and Benjamin Schrauwen. 2014. Sign Language Recognition Using Convolutional Neural Networks.Google Scholar
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In 19th Annual International Conference on Mobile Computing 8 Networking.Google ScholarDigital Library
- Jiacheng Shang and Jie Wu. 2017. A robust sign language recognition system with multiple Wi-Fi devices. In The Workshop on Mobility in the Evolving Internet Architecture.Google ScholarDigital Library
- Yang Si, Song Ren, Qinkun Xiao, Yuan Ma, Kun Zhong, and Yang Xuemeng. 2018. Sign language recognition algorithm based on color and depth image. Science Technology and Engineering (2018).Google Scholar
- Thad Starner, Joshua Weaver, and Alex Pentland. 1998. Real-time American sign language recognition using desk and wearable computer based video. IEEE Transactions on Pattern Analysis 8 Machine Intelligence 20, 12 (1998), 1371--1375.Google ScholarDigital Library
- Gordon L. Stuber, John R. Barry, Steve W. Mclaughlin, Ye (Geoffrey) Li, Mary Ann Ingram, and Thomas G. Pratt. 2004. Broadband MIMO-OFDM wireless communications. In IEEE 92, 2 (2004), 271--294.Google ScholarCross Ref
- Chao Sun, Tianzhu Zhang, Bingkun Bao, Changsheng Xu, and Tao Mei. 2013. Discriminative exemplar coding for sign language recognition with Kinect. IEEE Transactions on Cybernetics 43, 5 (2013), 1418--1428.Google ScholarCross Ref
- Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu Han Kim. 2015. WiDraw: Enabling hands-free drawing in the air on commodity WiFi devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking.Google ScholarDigital Library
- Aditya Virmani and Muhammad Shahzad. 2017. Position and orientation agnostic gesture recognition using WiFi. In International Conference on Mobile Systems, Applications, and Services.Google ScholarDigital Library
- Christian Vogler and Dimitris Metaxas. 1998. ASL recognition based on a coupling between HMMs and 3D motion analysis. In International Conference on Computer Vision.Google ScholarCross Ref
- Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: Virtual touch screen in the air using RF signals. In ACM Conference on SIGCOMM.Google ScholarDigital Library
- Wei Wang, Alex X. Liu, Muhammad Shahzad, Kang Ling, and Sanglu Lu. 2015. Understanding and modeling of WiFi signal based human activity recognition. In International Conference on Mobile Computing and NETWORKING.Google ScholarDigital Library
- Xuyu Wang, Lingjun Gao, and Shiwen Mao. 2016. PhaseFi: Phase fingerprinting for indoor localization with a deep learning approach. In Global Communications Conference.Google Scholar
- Kaishun Wu, Xiao Jiang, Youwen Yi, Gao Min, and Lionel M. Ni. 2012. FILA: Fine-grained indoor localization. In IEEE INFOCOM.Google Scholar
- Jie Yang, Yong Ge, Hui Xiong, Yingying Chen, and Hongbo Liu. 2010. Performing joint learning for passive intrusion detection in pervasive wireless environments. In IEEE INFOCOM.Google Scholar
- Chengwei Yao and Gencai Chen. 2016. Hyperparameters adaptation for restricted Boltzmann machines based on free energy. In International Conference on Intelligent Human-Machine Systems and Cybernetics.Google ScholarCross Ref
- Koji Yatani and Khai N. Truong. 2012. BodyScope: A wearable acoustic sensor for activity recognition. In ACM Conference on Ubiquitous Computing.Google Scholar
- Nan Yu, Wei Wang, Alex X. Liu, and Lingtao Kong. 2018. QGesture: Quantifying gesture distance and direction with WiFi signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 23--73.Google ScholarDigital Library
- Jie Zhang, Xiaolong Zheng, yong Tang, Zhan, Tianzhang Xing, Xiaojiang Chen, Dingyi Fang, Rong Li, Xiaoqing Gong, and Feng Chen. 2016. Privacy leakage in mobile sensing: Your unlock passwords can be leaked through wireless hotspot functionality. Mobile Information Systems 2016 (2016), 1--14.Google ScholarDigital Library
- Ouyang Zhang and Kannan Srinivasan. 2016. Mudra: User-friendly fine-grained gesture recognition using WiFi signals. In 12th International on Conference on Emerging Networking EXperiments and Technologies.Google ScholarDigital Library
- Xiaolong Zheng, Jiliang Wang, Longfei Shangguan, Zimu Zhou, and Yunhao Liu. 2016. Smokey: Ubiquitous smoking detection with commercial WiFi infrastructures. In IEEE INFOCOM.Google Scholar
- Xiaolong Zheng, Jiliang Wang, Longfei Shangguan, Zimu Zhou, and Yunhao Liu. 2017. Design and implementation of a CSI-based ubiquitous smoking detection system. IEEE/ACM Transactions on Networking 25, 6, pp (2017), 3781--3793.Google ScholarDigital Library
- Zimu Zhou, Longfei Shangguan, Xiaolong Zheng, Lei Yang, and Yunhao Liu. 2017. Design and implementation of an RFID-based customer shopping behavior mining system. IEEE/ACM Transactions on Networking 25 (2017), 2405--2418.Google ScholarDigital Library
Index Terms
- WiSign: Ubiquitous American Sign Language Recognition Using Commercial Wi-Fi Devices
Recommendations
Exploiting phonological constraints for handshape inference in ASL video
CVPR '11: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern RecognitionHandshape is a key linguistic component of signs, and thus, handshape recognition is essential to algorithms for sign language recognition and retrieval. In this work, linguistic constraints on the relationship between start and end handshapes are ...
American and Russian Sign Language Dactyl Recognition and Text2Sign Translation
Analysis of Images, Social Networks and TextsAbstractSign language is the main way to communicate for people from deaf community. However, common people mostly do not know sign language. In this paper, we overview several real-time sign language dactyl recognition systems using deep convolutional ...
Accurate and Accessible Motion-Capture Glove Calibration for Sign Language Data Collection
Motion-capture recordings of sign language are used in research on automatic recognition of sign language or generation of sign language animations, which have accessibility applications for deaf users with low levels of written-language literacy. ...
Comments