Device-free single-user activity recognition using diversified deep ensemble learning

https://doi.org/10.1016/j.asoc.2020.107066Get rights and content

Abstract

WiFi-based human activity recognition (HAR) aims to recognize human activities in an off-the-shelf manner that only relies on the commercial Wi-Fi devices already installed in environments. The recent trend in HAR research is to train classifiers on top of statistical or deep neural features extracted from channel state information (CSI) data. Unfortunately, existing methods only take into account the temporal-correlation within each CSI subcarrier, while ignoring the spatial-correlation between different subcarriers. This issue has not been fully exploited yet, resulting a limited performance. To address this issue, we propose WiAReS, a WiFi-based device-free activity recognition system that takes both temporal-correlation and spatial-correlation into account. WiAReS embarks on diversified deep ensemble methods 2̌for single-user activity recognition where one user performs a single activity at a given time. More specifically, it adopts convolutional neural network (CNN) to automatically extract features from CSI measurements with the preservation of the locality of both spatial patterns and temporal patterns. To further improve recognition accuracy upon CNN-extracted features, we propose a novel ensemble architecture that fuses a multiple layer perception (MLP), a random forest (RF) and a support vector machine (SVM). Our system obtains the CSI data in PHY layer of off-the-shelf WiFi devices by installing Atheros-CSI-Tool on AR9590 based WiFi network interface cards (NICs). Comprehensive experiments have been conducted in three real environments with environmental variation to evaluate the performance of the proposed WiAReS. The experimental results demonstrate that the proposed WiARes system significantly outperforms existing methods.

Introduction

Human activity recognition (HAR) is a key component for a number of applications such as smart homes, security monitoring, fall detection for elderly people, search-and-rescue systems, and many other Internet of Things (IoT)-based applications [1].

The importance of HAR has led to many research efforts in recent years. Traditional HAR approaches rely on dedicated devices such as cameras [2], wearable sensors [3] and radars [4]. However, they all suffer from high device cost and low efficacy, since their abilities are limited by the prerequisites of deployment. Specifically, the camera-based methods requires line-of-sight (LOS) condition, wearable sensors need to be carried all the time, and radar-based methods have limited sensing range due to the short wavelength. The requirements for both devices and their deployments limit their sensing ability and hinder their further applications in daily-life scenes.

Observing the disadvantages of above approaches, recently, WiFi, as an ubiquitously available devices, has been introduced to perform device-free human activity recognition [5]. The assumption on which WiFi-based solution relies is that a certain human activity could incur unique variations in WiFi signals, as shown in Fig. 1. WiFi-based solutions use commercial off-the-shelf (COTS) Wi-Fi devices that have already installed in environments to recognize activities [6], [7], [8], [9], [10]. These methods regard human activity recognition as a classification problem that trains classifiers (e.g., SVM and MLP) on top of statistical features from WiFi signals. There are many advantages of WiFi-based solutions. They does not require LOS path, has better coverage, without carrying any sensors or devices, and could better protect users’ privacy. Due to their ubiquitous availability and easy deployment, WiFi-based solutions become a popular choice for human activity recognition applications.

Commonly, there are two types of WiFi signal information that are employed in WiFi-based HAR systems: received signal strength (RSS) and channel state information (CSI). The RSS is a scalar value indicating the received signal strength, whereas the CSI is the multiple-channels information directly derived from the PHY layer. Compared with the RSS, the CSI contains more environment information, which could implicitly benefits the recognition of activities. Thus, the challenging issue is how to effectively exploit and refine the useful information from the plentiful CSI data to facilitate performance improvement in HAR system. Recently, pioneer research efforts have sought for automatically generating meaningful features from raw data with the advance of deep learning (DL) techniques. For example, Yousefi et al. [11] and Chen et al. [12] adopted long short-term memory (LSTM), an RNN-family neural network, to recognize activities, and achieved state-of-the-art results on many datasets.

Despite their usefulness, current DL-based methods share a common drawback. They only take into account the temporal-correlation within each CSI subcarrier, while ignoring the spatial-correlation between different subcarriers. In facts, due to the physical structure of CSI channels, correlation exists between adjacent subcarriers, indicating that a human activity would incur coupling effects among adjacent subcarriers. Ignoring the spatial-correlation will make HAR system not be able to fully exploit the implicit spatial patterns implied in multi-subcarriers CSI data, and as a consequence, weaken their recognition ability.

In this paper, we propose a novel method that takes both temporal-correlation and spatial-correlation into account. Our model adopt convolutional neural network (CNN) to automatically extract CSI features. The proposed methods take advantages of both highly discriminative features and efficient yet diversified classifiers. CNN has been proven successful in extracting features from data that presents locality of spatial patterns. Thus it is a perfect choice for representing spatial and temporal patterns, both computationally and semantically. On top of CNN-extracted features, we use a novel ensemble architecture that fuses a MLP, a random forest (RF) and a support vector machine (SVM) to achieve higher recognition accuracy. Moreover, we have built a prototype WiFi-based human activity recognition system (WiAReS) that applies our proposed HAR method on multiple subcarriers with Athero NIC. With this prototype system, we demonstrate that our method is practically better in recognizing human activities.

The core contributions of this paper are summarized as follows.

  • 1.

    We propose a novel ensemble model for human activity recognition that takes both spatial and temporal patterns into consideration. This model uses CNNs to extract comprehensive features from raw CSI data in terms of both temporal-correlation and spatial-correlation, and combines triple classifiers to improve the accuracy in activity recognition.

  • 2.

    We develop a prototype HAR system WiARes. Our WiAReS was built upon Atheros devices that have 56 subcarriers of CSI measurements for each TX–RX antenna pair. This push the accuracy even further to achieve new state-of-the-art results on many benchmark datasets.

  • 3.

    We conduct experiments with hardware testbed in two clutter indoor laboratories, and evaluate the proposedschemes extensively. The results demonstrate that our system significantly outperform existing systems.

The rest of the paper is structured as follows. Section 2 reviews the related work on device-free wireless activity recognition. Section 3 introduces the CSI, followed by the motivation behind the proposed schemes. Section 4 presents the proposed WiAReS using CSI features. The experimental evaluations are demonstrated in Section 5. Section 5 gives the conclusion remarks.

Section snippets

Related work

Due to ubiquitous availability in indoor areas, WiFi based human activity recognition has gained tremendous attention in recent years. Sigg et al. [13], [14] explored methods to realize activity recognition using the RSS of wireless signals, which relies on the fluctuations of the received signal strength affected by human behavior. WiGest [6] leveraged the effect of hand motion on received wireless signal strength to recognize hand gesture with off-the-shelf WiFi devices. The proposed system

Preliminary information

In this section, we first reviews the CSI. Next, the motivation of the proposed ensemble scheme for activity recognition is discussed.

The proposed WiAReS with WiFi CSI

In this section, we first present the design of our WiAReS using WiFi CSI measurements. Then, the proposed hybrid structure of CNN, RF and SVM for activity recognition is introduced in details.

Experimental evaluation

In this section, we discuss the implementation of WiARes and present the activity evaluation of users under various environments to show its accuracy and robustness.

Conclusion

This paper presents a WiFi CSI-based activity recognition system WiAReS that considers the influence of human behaviors on surrounding CSI measurements. First, this paper explores the challenges associated with subcarrier correlation information loss. We use CNNs to learn optimized features involved with subcarrier correlation and time-domain correlation from raw CSI measurements on multiple subcarriers. Second, we propose a diversified deep ensemble system using features learned from

CRediT authorship contribution statement

Wei Cui: Methodology, Software. Bing Li: Writing - original draft. Le Zhang: Data collection. Zhenghua Chen: Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (61903231), Shandong Province Natural Science Foundation (ZR2018PF011), the A*STAR Industrial Internet of Things Research Program under the RIE2020 IAF-PP Grant A1788a0023.

References (42)

  • SunL. et al.

    Widraw: Enabling hands-free drawing in the air on commodity wifi devices

  • WangW. et al.

    Understanding and modeling of wifi signal based human activity recognition

  • WangH. et al.

    Human respiration detection with commodity wifi devices: do user location and body orientation matter?

  • WangY. et al.

    Wifall: Device-free fall detection by wireless networks

    IEEE Trans. Mob. Comput.

    (2017)
  • YousefiS. et al.

    A survey on behavior recognition using wifi channel state information

    IEEE Commun. Mag.

    (2017)
  • ChenZ. et al.

    Wifi csi based passive human activity recognition using attention based blstm

    IEEE Trans. Mob. Comput.

    (2019)
  • SiggS. et al.

    Rf-sensing of activities from non-cooperative subjects in device-free recognition systems using ambient and local signals

    IEEE Trans. Mob. Comput.

    (2014)
  • SiggS. et al.

    Leveraging rf-channel fluctuation for activity recognition: Active and passive systems, continuous and rssi-based signal features

    Proceedings of International Conference on Advances in Mobile Computing & Multimedia

    (2013)
  • HalperinD. et al.

    Tool release: Gathering 802.11 n traces with channel state information

    ACM SIGCOMM Comput. Commun. Rev.

    (2011)
  • XieY. et al.

    Precise power delay profiling with commodity wi-fi

    IEEE Trans. Mob. Comput.

    (2018)
  • QianK. et al.

    Widar2. 0: Passive human tracking with a single wi-fi link

    Proc. ACM MobiSys

    (2018)
  • Cited by (31)

    View all citing articles on Scopus
    View full text