Learning personalized ADL recognition models from few raw data

https://doi.org/10.1016/j.artmed.2020.101916Get rights and content

Highlights

Abstract

Recognition of activities of daily living (ADL) is an essential component of assisted living systems based on actigraphy. This task can nowadays be performed by machine learning models which are able to automatically extract and learn relevant features but, most of time, need to be trained with large amounts of data collected on several users. In this paper, we propose an approach to learn personalized ADL recognition models from few raw data based on a specific type of neural network called matching network. The interest of this few-shot learning approach is three-fold. Firstly, people perform activities their own way and general models may average out important individual characteristics unlike personalized models that could thus achieve better performance. Secondly, gathering large quantities of annotated data from one user is time-consuming and threatens privacy in a medical context. Thirdly, matching networks are by nature weakly dependent on the classes they are trained on and can generalize easily to new activities without needing extra training, thus making them very versatile for real applications. Our results show the effectiveness of the proposed approach compared to general neural network models, even in situations with few training data.

Introduction

As life expectancy increases, more and more elderly people show difficulties in their everyday life, and allowing them to stay at home is a social and public health issue.1 Many people even struggle performing basic need activities. They are also particularly exposed to chronic diseases: diabetes, cancer, psychological and cognitive disorders, heart diseases, Parkinson and Alzheimer, etc. They are finally vulnerable in simple daily life activities where they could fall, make a wrong move or loose attention. Furthermore, 15% of the world population lives with a form of handicap, between 2 and 4% suffering severe disabilities.2 All these persons could benefit from eHealth services and particularly activity monitoring [1]. The process of recording the everyday life activities of a subject using sensors (inertial sensors, for instance) is called actigraphy. Instead of organizing regular visits at the hospital, the patient can be monitored in his/her house with several upsides: it improves the quality of life of the patient and shortens hospital stays while facilitating the diagnosis as important data are collected in the usual environment of the patient. Of course, clinical visits are indispensable but can only take a snapshot of the patient's condition and may occur too late during the disease development [2].

Inertial data are particularly interesting and nowadays easy to record with smartphone sensors (or smart watches, clothes, etc.) which can be carried without stigmatizing the person. They are judged less intrusive at home and more respectful of privacy. They also allow for a continuous monitoring of the person whereas home sensors only work where they are installed and do not target a specific inhabitant. These data can then be used to provide eHealth services regarding the recorded level of activity and the distribution of those activities during the day. Traditionally, the level of autonomy has been evaluated thanks to several criteria related to the Activities of Daily Living (ADL, e.g. having lunch, watching television, etc.). This evaluation appears as a pertinent factor for the clinical evaluation of elderly people [3]. It is possible to observe, with the help of actigraphy systems, changes in a person's behavior and so the possible loss of autonomy. Data obtained this way allow to perform ADL or posture classification and prediction and to automatically detect falls. Early approaches consisted in (manually) defining expert rules and thresholds for the sensor values leading to posture recognition and then activities, nowadays machine learning models can automatically learn to classify these data. Moreover, neural network models are able to automatically extract the relevant features to process and are able to adapt easily to new activities and new users [4]. To be able to eventually equip people with such systems, high classification accuracy is required, particularly for critical events such as falls. Moreover, compliance with legal regulations, such as the General Data Protection Regulation3 (GDPR), and privacy preservation are a necessity.

We tackle both of these issues in this extended version of our paper [5] by proposing a model able to perform personalized ADL classification from few raw data. Contrary to a common practice [6], we advocate for personalized models instead of general models: better performances can be achieved with personalized models since each user has his/her own way of doing his/her activities. It is for example, possible to recognize a person by analyzing his/her gait [7]. In broader perspectives, these personalized models may be more compact as well as easier and faster to train, adapted to smartphones and embedded wearable devices with less energy consumption [8] in the critical context of climate change. However, most of the time, due to privacy concerns and the time needed to annotate each sample, we have very few data coming from a single user in order to effectively train supervised activity recognition models. To overcome this issue, we propose an ADL recognition model based on the matching network architecture [9], and performing few-shot learning, that is, a model able to recognize classes from just one or few samples. Matching networks are by design weakly dependent on the classes they are trained on and therefore can adapt with only one new annotated sample to any new activity class performed by the user, making them very versatile and suited for real environments. Exploiting this property, we demonstrate that the performance can be improved by using another inertial dataset that contains different classes to pretrain the encoding part of the network and that further acts as a validation set to prevent overfitting. The final results show that our approach called SSMN (sequence-to-sequence matching networks) achieves comparable performances with classical neural network approaches trained on a whole dataset and further obtains over 90% accuracy on one-shot fall classification.

The paper is organized as follows. We summarize in Section 2 previous work on personalized activity recognition and few-shot learning. We then describe our approach for few-shot personalized activity recognition based on matching networks in Section 3. In Section 4, we report the results of several experiments on the MobiAct V2 Dataset [10] and the UCI HAR dataset [11] and assess the utility of the different components of the model and its capacity to predict classes that have not been used for training. Finally, conclusions and perspectives are drawn in Section 5.

Section snippets

Related work

Human activity recognition is a very broad computer science field which aims to recognize what a person is doing by analyzing data related to this person recorded from various sensors or instruments. It has numerous applications: from crime detection on video surveillance images to gesture recognition when performing a physical activity. It can be performed in several contexts (Lara et al. [12] listed seven types: ambulation, transportation, phone usage, exercise/fitness, military, upper body

Sequence processing with recurrent neural networks

Postures, ADL or falls exhibit a dynamic characteristic signature (e.g. walking, running, going upstairs) or are, on the contrary, more static (i.e. lying, sitting, standing, etc.). Considering the whole sequence of raw data to classify a posture is thus pertinent [25]. A classical approach when working with sequences is to extract several signal feature vectors from subsequences of the signal in order to build a classifier. This approach is efficient in numerous cases but, as the window size

Preliminary experiment: personalized postures classification

We first propose a preliminary experiment on a dataset called Postures where personalized models are learned on inertial data with a standard GRU only. This dataset will be used afterwards as support dataset as explained in Section 3.3. The Postures dataset was created by Quach [39]. The data has been acquired using a 9-axes Inertial Measurement Unit (accelerometer, gyroscopes and magnetometer, IMU) on 9 subjects executing the same sequence several times. Each user produced 5 sequences apart

Conclusions and perspectives

We presented in this paper an approach for personalized ADL classification based on matching networks combined with sequence-to-sequence pretraining (SSMN). This approach presents two major advantages which make it very relevant to be implemented for real applications. First, it addresses the problem of limited training data that is encountered when learning personalized models by being able to learn from just a few examples. Second, it is very versatile regarding each new activity a user could

Conflict of interest

None declared.

References (48)

  • R. San-Segundo et al.

    HMM adaptation for improving a human activity recognition system

    Algorithms

    (2016)
  • H. Pigot et al.

    The role of intelligent habitats in upholding elders in residence

    5th international conference on simulations in biomedicine

    (2003)
  • A. Avci et al.

    Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: a survey

    ARCS, VDE

    (2010)
  • S. Katz et al.

    Studies of illness in the aged: the index of ADL: a standardized measure of biological and psychosocial function

    JAMA

    (1963)
  • C. Tan et al.

    A survey on deep transfer learning

    International conference on artificial neural networks

    (2018)
  • P. Compagnon et al.

    Personalized posture and fall classification with shallow gated recurrent units

    2019 IEEE 32nd international symposium on computer-based medical systems (CBMS)

    (2019)
  • C. Debes et al.

    Monitoring activities of daily living in smart homes: Understanding human behavior

    IEEE signal processing magazine

    (2016)
  • L. Wang et al.

    Silhouette analysis-based gait recognition for human identification

    IEEE PAMI

    (2003)
  • E. Strubell et al.

    Energy and policy considerations for deep learning in NLP

    (2019)
  • O. Vinyals et al.

    Matching networks for one shot learning

    Advances in neural information processing systems

    (2016)
  • C. Chatzaki et al.

    Human daily activity and fall recognition using a smartphone's acceleration sensor

    ICT4AWE

    (2016)
  • D. Anguita et al.

    A public domain dataset for human activity recognition using smartphones

    ESANN

    (2013)
  • O.D. Lara et al.

    A survey on human activity recognition using wearable sensors

    IEEE Commun Surv Tutorials

    (2013)
  • L. Bao et al.

    Activity recognition from user-annotated acceleration data

    International conference on pervasive computing

    (2004)
  • X. Sun et al.

    Large-scale personalized human activity recognition using online multitask learning

    IEEE Trans Knowl Data Eng

    (2012)
  • B. Longstaff et al.

    Improving activity classification for health applications on mobile devices using active and semi-supervised learning

    2010 4th international conference on pervasive computing technologies for healthcare

    (2010)
  • L.T. Nguyen et al.

    Recognizing new activities with limited training data

    Proceedings of the 2015 ACM international symposium on wearable computers

    (2015)
  • D. Wu et al.

    One shot learning gesture recognition from RGBD images

    Computer vision and pattern recognition workshops (CVPRW)

    (2012)
  • A. Bellet et al.

    Metric learning

    Synth Lect Artif Intell Mach Learn

    (2015)
  • S. Sani et al.

    Personalised human activity recognition using matching networks

    International conference on case-based reasoning

    (2018)
  • R. DiPietro et al.

    Analyzing and exploiting NARX recurrent neural networks for long-term dependencies

    Workshop track – ICLR

    (2017)
  • P. Tsinganos et al.

    A smartphone-based fall detection system for the elderly

    Proceedings of the 10th international symposium on image and signal processing and analysis

    (2017)
  • Y. Zhao et al.

    Deep residual BIDIR-LSTM for human activity recognition using wearable sensors

    Mathematical problems in engineering 2018

    (2018)
  • W. Jiang et al.

    Human activity recognition using wearable sensors by deep convolutional neural networks

    Proceedings of the 23rd ACM international conference on Multimedia

    (2015)
  • View full text