Learning personalized ADL recognition models from few raw data
Introduction
As life expectancy increases, more and more elderly people show difficulties in their everyday life, and allowing them to stay at home is a social and public health issue.1 Many people even struggle performing basic need activities. They are also particularly exposed to chronic diseases: diabetes, cancer, psychological and cognitive disorders, heart diseases, Parkinson and Alzheimer, etc. They are finally vulnerable in simple daily life activities where they could fall, make a wrong move or loose attention. Furthermore, 15% of the world population lives with a form of handicap, between 2 and 4% suffering severe disabilities.2 All these persons could benefit from eHealth services and particularly activity monitoring [1]. The process of recording the everyday life activities of a subject using sensors (inertial sensors, for instance) is called actigraphy. Instead of organizing regular visits at the hospital, the patient can be monitored in his/her house with several upsides: it improves the quality of life of the patient and shortens hospital stays while facilitating the diagnosis as important data are collected in the usual environment of the patient. Of course, clinical visits are indispensable but can only take a snapshot of the patient's condition and may occur too late during the disease development [2].
Inertial data are particularly interesting and nowadays easy to record with smartphone sensors (or smart watches, clothes, etc.) which can be carried without stigmatizing the person. They are judged less intrusive at home and more respectful of privacy. They also allow for a continuous monitoring of the person whereas home sensors only work where they are installed and do not target a specific inhabitant. These data can then be used to provide eHealth services regarding the recorded level of activity and the distribution of those activities during the day. Traditionally, the level of autonomy has been evaluated thanks to several criteria related to the Activities of Daily Living (ADL, e.g. having lunch, watching television, etc.). This evaluation appears as a pertinent factor for the clinical evaluation of elderly people [3]. It is possible to observe, with the help of actigraphy systems, changes in a person's behavior and so the possible loss of autonomy. Data obtained this way allow to perform ADL or posture classification and prediction and to automatically detect falls. Early approaches consisted in (manually) defining expert rules and thresholds for the sensor values leading to posture recognition and then activities, nowadays machine learning models can automatically learn to classify these data. Moreover, neural network models are able to automatically extract the relevant features to process and are able to adapt easily to new activities and new users [4]. To be able to eventually equip people with such systems, high classification accuracy is required, particularly for critical events such as falls. Moreover, compliance with legal regulations, such as the General Data Protection Regulation3 (GDPR), and privacy preservation are a necessity.
We tackle both of these issues in this extended version of our paper [5] by proposing a model able to perform personalized ADL classification from few raw data. Contrary to a common practice [6], we advocate for personalized models instead of general models: better performances can be achieved with personalized models since each user has his/her own way of doing his/her activities. It is for example, possible to recognize a person by analyzing his/her gait [7]. In broader perspectives, these personalized models may be more compact as well as easier and faster to train, adapted to smartphones and embedded wearable devices with less energy consumption [8] in the critical context of climate change. However, most of the time, due to privacy concerns and the time needed to annotate each sample, we have very few data coming from a single user in order to effectively train supervised activity recognition models. To overcome this issue, we propose an ADL recognition model based on the matching network architecture [9], and performing few-shot learning, that is, a model able to recognize classes from just one or few samples. Matching networks are by design weakly dependent on the classes they are trained on and therefore can adapt with only one new annotated sample to any new activity class performed by the user, making them very versatile and suited for real environments. Exploiting this property, we demonstrate that the performance can be improved by using another inertial dataset that contains different classes to pretrain the encoding part of the network and that further acts as a validation set to prevent overfitting. The final results show that our approach called SSMN (sequence-to-sequence matching networks) achieves comparable performances with classical neural network approaches trained on a whole dataset and further obtains over 90% accuracy on one-shot fall classification.
The paper is organized as follows. We summarize in Section 2 previous work on personalized activity recognition and few-shot learning. We then describe our approach for few-shot personalized activity recognition based on matching networks in Section 3. In Section 4, we report the results of several experiments on the MobiAct V2 Dataset [10] and the UCI HAR dataset [11] and assess the utility of the different components of the model and its capacity to predict classes that have not been used for training. Finally, conclusions and perspectives are drawn in Section 5.
Section snippets
Related work
Human activity recognition is a very broad computer science field which aims to recognize what a person is doing by analyzing data related to this person recorded from various sensors or instruments. It has numerous applications: from crime detection on video surveillance images to gesture recognition when performing a physical activity. It can be performed in several contexts (Lara et al. [12] listed seven types: ambulation, transportation, phone usage, exercise/fitness, military, upper body
Sequence processing with recurrent neural networks
Postures, ADL or falls exhibit a dynamic characteristic signature (e.g. walking, running, going upstairs) or are, on the contrary, more static (i.e. lying, sitting, standing, etc.). Considering the whole sequence of raw data to classify a posture is thus pertinent [25]. A classical approach when working with sequences is to extract several signal feature vectors from subsequences of the signal in order to build a classifier. This approach is efficient in numerous cases but, as the window size
Preliminary experiment: personalized postures classification
We first propose a preliminary experiment on a dataset called Postures where personalized models are learned on inertial data with a standard GRU only. This dataset will be used afterwards as support dataset as explained in Section 3.3. The Postures dataset was created by Quach [39]. The data has been acquired using a 9-axes Inertial Measurement Unit (accelerometer, gyroscopes and magnetometer, IMU) on 9 subjects executing the same sequence several times. Each user produced 5 sequences apart
Conclusions and perspectives
We presented in this paper an approach for personalized ADL classification based on matching networks combined with sequence-to-sequence pretraining (SSMN). This approach presents two major advantages which make it very relevant to be implemented for real applications. First, it addresses the problem of limited training data that is encountered when learning personalized models by being able to learn from just a few examples. Second, it is very versatile regarding each new activity a user could
Conflict of interest
None declared.
References (48)
- et al.
HMM adaptation for improving a human activity recognition system
Algorithms
(2016) - et al.
The role of intelligent habitats in upholding elders in residence
5th international conference on simulations in biomedicine
(2003) - et al.
Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: a survey
ARCS, VDE
(2010) - et al.
Studies of illness in the aged: the index of ADL: a standardized measure of biological and psychosocial function
JAMA
(1963) - et al.
A survey on deep transfer learning
International conference on artificial neural networks
(2018) - et al.
Personalized posture and fall classification with shallow gated recurrent units
2019 IEEE 32nd international symposium on computer-based medical systems (CBMS)
(2019) - et al.
Monitoring activities of daily living in smart homes: Understanding human behavior
IEEE signal processing magazine
(2016) - et al.
Silhouette analysis-based gait recognition for human identification
IEEE PAMI
(2003) - et al.
Energy and policy considerations for deep learning in NLP
(2019) - et al.
Matching networks for one shot learning
Advances in neural information processing systems
(2016)
Human daily activity and fall recognition using a smartphone's acceleration sensor
ICT4AWE
A public domain dataset for human activity recognition using smartphones
ESANN
A survey on human activity recognition using wearable sensors
IEEE Commun Surv Tutorials
Activity recognition from user-annotated acceleration data
International conference on pervasive computing
Large-scale personalized human activity recognition using online multitask learning
IEEE Trans Knowl Data Eng
Improving activity classification for health applications on mobile devices using active and semi-supervised learning
2010 4th international conference on pervasive computing technologies for healthcare
Recognizing new activities with limited training data
Proceedings of the 2015 ACM international symposium on wearable computers
One shot learning gesture recognition from RGBD images
Computer vision and pattern recognition workshops (CVPRW)
Metric learning
Synth Lect Artif Intell Mach Learn
Personalised human activity recognition using matching networks
International conference on case-based reasoning
Analyzing and exploiting NARX recurrent neural networks for long-term dependencies
Workshop track – ICLR
A smartphone-based fall detection system for the elderly
Proceedings of the 10th international symposium on image and signal processing and analysis
Deep residual BIDIR-LSTM for human activity recognition using wearable sensors
Mathematical problems in engineering 2018
Human activity recognition using wearable sensors by deep convolutional neural networks
Proceedings of the 23rd ACM international conference on Multimedia
Cited by (5)
Recognizing Human Activities of Daily Living Using Mobile Sensors for Health monitoring
2022, Internet of Things and Data Mining for Modern Engineering and Healthcare ApplicationsMachine Learning for Healthcare Wearable Devices: The Big Picture
2022, Journal of Healthcare EngineeringMachine Learning Based Fall Detector with a Sensorized Tip
2021, IEEE Access