DF-WiSLR: Device-Free Wi-Fi-based Sign Language Recognition

Highlights

• Pioneers the study of Indian Sign Language gestures comprising single word and sentences utilizing the CSI of Wi-Fi.

• Expands the training data using augmentation scheme.

• Higher order feature extraction and selection for improved recognition accuracy.

• Adopted several machine learning classifiers and a deep learning classifier for evaluation.

• Study the impact of environmental factors and gesture orientation on recognition performance.

Abstract

Recent advancements in wireless technologies enable pervasive and device free gesture recognition that enable assisted living utilizing off the shelf commercial Wi-Fi devices. This paper proposes a Device-Free Wi-Fi-based Sign Language Recognition (DF-WiSLR) for recognizing 30 static and 19 dynamic sign gestures. The raw Channel State Information (CSI) acquired from the Wi-Fi device for 49 sign gestures, with a volunteer performing the sign gestures in home and office environments. The proposed system adopts machine learning classifiers such as SVM, KNN, RF, NB, and a deep learning classifier CNN, for measuring the gesture recognition accuracy. To address the practical limitation of building a voluminous dataset, DF-WiSLR augments the originally acquired CSI values with Additive White Gaussian Noise (AWGN). Higher-order cumulant features of orders 2, 3, and 4 are extracted from the original and augmented data, as the machine learning classifiers demand manual feature extraction. To reduce the computational complexity of machine learning classifiers, an informative and reduced optimal feature subset is selected using MIFS. Whilst the pre-processed original and augmented CSI values directly fed as input to an 8-layer deep CNN, it performs auto feature extraction and selection. DF-WiSLR reported better recognition accuracies with SVM for static and dynamic gestures in both home and office environments. SVM achieved 93.4% 98.8% and 98.9% accuracies in home and office environments respectively, for static gestures. For dynamic gestures, 92.3% recognition accuracy achieved in home environment. On augmented data, the corresponding gesture recognition accuracy values reported are 97.1%, 99.9%, 99.9%, and 98.5%.

Introduction

Sign languages are a form of communication among the hearing impaired and children with Autism Spectrum Disorder (ASD). Sign languages are not universal, and hence every part of the globe follows a unique sign language with its grammar and lexicon and attracts research interests in assisted living environment. Device-based recognition automates the recognition scheme with the deployment of sophisticated commercial devices in the sensing environment. Such methods use commercial devices like depth cameras [1], and wearable inertial or motion sensors [2], [3], [4]. However, device-based sensing methods considered to be invasive and obtrusive, hence not a preferable choice for the majority of recognition applications [5], [6].

Whereas, device-free recognition paradigm, achieve non-intrusive and privacy-preserving gesture recognition gathering the reflection pattern of the signal due to human movement [7], [8]. Regardless of numerous signal information, Received Signal Strength Information (RSSI) [9], and Channel State Information (CSI) [10], [11], [12], [13], [14], [15] are of importance for establishing a seamless recognition. Universal Software Radio Peripheral (USRP) [16], [17], Wi-Fi routers [18], and RFID [19] are the widely adopted devices that capture essential signal information for device-free recognition. The hassle-free and pervasive characteristics of Wi-Fi signals motivate the present work to adopt commercial Wi-Fi routers for performing device-free gesture recognition of sign language.

This paper proposes a Device-Free Wi-Fi-based Sign Language Recognition (DF-WiSLR) system, pioneers Wi-Fi CSI dataset acquisition for 49 Indian Sign Language (ISL) gestures (static $+$ dynamic). Static gestures are simple signs comprising alphabets, numbers and single words whereas dynamic gestures include compounding signs involving sentences. DF-WiSLR performs the recognition task in a confined environment by gathering CSI that unveils both amplitude and phase information enabling fine-grained gesture recognition [13]. The Wi-Fi CSI values gathered from wireless routers are prone to noise, thus necessitate pre-processing to make it usable for any sensing applications. State of the art applies regression or filtering techniques for removing the noise and applies feature extraction or dimensionality reduction technique [20], [21], [22]. However, the present work eliminates the application of filtering techniques on raw CSI values as there is a chance of losing essential signal information.

DF-WiSLR adopts classical machine learning classifiers such as Support Vector Machine (SVM), K-Nearest Neighbor (KNN), Random Forest (RF), Naï ve–Bayes (NB) and a deep learning classifier, Convolutional Neural Network (CNN) for measuring the sign gesture recognition accuracy. The performance of any machine learning classifiers relies upon the handcrafted feature, whereas, deep learning classifiers perform auto feature extraction and selection. For machine learning classifiers, the present work applies a ‘standard score’ normalization on raw CSI and extract higher-order cumulant features of order two, three, and four. Subsequently, applies a Mutual Information Feature Selection (MIFS) algorithm on the larger set of extracted features to reduce the computational complexity [23]. Alternatively, DF-WiSLR pre-processes the raw CSI values by applying the Multiple Linear Regression (MLR) technique before feeding it as input to the CNN.

The proposed study, computes the training and testing time of the measured recognition accuracy to identify best performing classifier with reduced computational effort. Nevertheless, recognition accuracy of learning-based approach improves with increasing number of training instances. However, collecting voluminous training instances from volunteers will be a burdensome task under some practical situations. Data augmentation address this limitation by expanding the data diversity eliminating the need of physical data collection [24]. DF-WiSLR implements data augmentation by applying an Additive White Gaussian Noise (AWGN) with different Signal to Noise Ratio (SNR) values on the initially acquired signal data that expands the size of the training data.

The remainder of the paper is organized as follows: Section 2 briefly discusses the related work in gesture recognition. Section 3 provides information on materials and methods adopted in this study. Details on experimental data acquisition and implementation of the learning algorithms are presented in Section 4. Section 5 discusses the performance of the learning algorithms with recognition accuracy and computational time as evaluation measures. Section 6 summarizes and concludes the present work.