A systematic approach to multi-task learning from time-series data
Introduction
Recently, DARPA [1] has promoted advances towards human–machine symbiosis where learning from little data is a critical challenge forward. This paper aims at addressing the limitation of annotated time-series data for personalized prediction models by way of transfer learning with a new systematic approach to building multi-task learning (MTL) models. Time-series sequences are of particular importance due to the present and foreseen abundance and advancements of the integration of wearable and IoT devices into everyday settings and applications.
Personalization in modeling is motivated by the fact that it is exhibited in many natural human interactions. Personalization is the model’s ability to extract features, also called descriptors, that are unique for a particular task (e.g. a particular user). Personalization becomes particularly important when the training data is a mixture of data from multiple tasks. As an illustration, consider human activity in which the model would need to be able to extract features that are unique to every user. Different people exhibit unique identifiers in posture, stride, and arm movements as people vary with age, body builds, and health conditions [2]. These differences resemble what we will refer to as unique identifiers, that is, features in the data that are tailored to a unique entity. Through capturing unique descriptors of an individual or entity being learned in a model, it is expected to witness gains in the accuracy of predictive models from sensing data [3], [4]. By accuracy, we are referring to the model’s ability to correctly label a classification task such as the activity for a particular user, which is also assessed by the log-loss likelihood measure.
In practice, nevertheless, a shortcoming of personalized modeling is that annotated datasets collected for a unique entity are typically not large enough. This limitation is often due to the time and effort involved in data labeling and that rich historical data is naturally limited for an entity, especially in cases where data is related to medical symptoms of different diseases such as glucose levels [5], seizure prediction [6], [7], [8], mental health [9], postural stability [10], and Parkinson’s disease [11], [12]. Hence, the size of personalized annotated datasets constitutes a bottleneck in achieving the desired performance in machine learning models.
This paper aims at addressing the limitation in annotated time-series data for personalized prediction models by way of transfer learning with new MTL models. The goal is to have personalized prediction models that are unique to individual entities while achieving knowledge transfer by exploiting the available commonality across entities. As an example, consider personalized activity recognition for individual users, where activity can be approached by considering consecutive action units to constitute high-level activities, such as walking or climbing stairs. These high-level activities include information patterns that can be understood as shared group behavior and other patterns that can be recognized as personalized to each user.
Traditionally, most researchers have approached the problem of learning models by one of two extreme approaches, population or personalized models, as shown in Fig. 1. Population models employ data collected from multiple entities to learn one unified prediction model for all entities [13], [14], while personalized models learn separate prediction models from each unique entity’s dataset [3], [15]. In both extreme cases, one task is learned at a time from the data. As a result, these models belong to a class of models called single task learning (STL). On the other hand, in the middle region of Fig. 1, multitask learning (MTL) models, introduced by R. Caruana [16], offer a tradeoff between both extremes where a compromise between specificity and generality of a model is achieved by learning personalized entity models while allowing for information sharing across entities. MTL learns multiple tasks simultaneously, leveraging their common information to improve the performance of all tasks equally. Several works have highlighted the superiority of MTL modeling in comparison to traditional STL methods, including the work of X. Sun et al. [17] and P. Lui et al. [18]. While their work sets strong grounds for MTL models, we present a generic and systematic approach for designing, evaluating, and improving MTL models.
This work proposes a framework for systematically investigating and building accurate personalized time-series MTL models while considering three primary design components: features capturing the time dynamics in data, similarity metrics reflecting degrees of commonality and uniqueness across entities, and generalization metrics to prevent overfitting. The framework enables the introduction of efficient new MTL models and advancing the prior state-of-the-art. The approach is successfully applied and tested resulting in two models for personalized continuous activity recognition. The first is an MTL deep learning approach that makes use of Convolutional Neural Networks (CNN) and Gated Recurrent Units (GRU), and the second advances on a previously developed MTL feature-based learning approach from the work of Sun et al. [17].
The main contributions of this work are:
- 1.
We present a systematic approach for designing and evaluating MTL models for personalized modeling from time-series data that is generic and applicable to any machine learning method.
- 2.
We propose a hierarchical MTL deep learning model for time-series classification that is composed of aconvolutional-recurrent neural network capable of (1) extracting features in an automated manner that is optimized and shared across entities and (2) extracting unique and shared relationships across entities through hard-parameter sharing of network layers across entities learned in parallel.
- 3.
We show an advancement of a prior MTL feature-based model for time-series data [17] by incorporating the 3 MTL components in the proposed systematic approach.
- 4.
We achieve state-of-the-art performance of the proposed MTL framework on three widely known human activity recognition benchmark datasets (OPPORTUNITY, UCI HAR, and DSA) and ALKAN of human activity recognition in-the-wild.
The proposed models are tested on the classification task of Human Activity Recognition (HAR) from sensing devices and evaluated on benchmark HAR datasets of OPPORTUNITY [19], UCI HAR [20], and DSA [21], as well as ALKAN [22] of HAR in-the-wild. The proposed methods show superiority in comparison to previous work on MTL models for time-series data and baseline approaches as well as competitive results when benchmarked against related work in literature, highlighting the strength of the proposed MTL systematic design approach for personalized modeling of time-series data.
The paper outline is as follows: Section 2 presents a summary and qualitative comparison of relevant and recent work. Section 3 defines the problem statement, general formulation, and design components. Section 4 presents the proposed MTL feature-based learning model, and Section 5 covers the MTL deep learning model. In Section 6, we describe the datasets, experimental setup, corresponding results and discuss the findings. We conclude the work in Section 7.
Section snippets
Related work
This section presents three outlooks on prior work: single-task learning (STL) personalized, multitask learning (MTL) feature-based, and MTL deep learning modeling approaches. We end the section with a qualitative comparison of prior literature in addressing the target research challenges and contributions of this work.
Problem statement
This work addresses the problem of designing accurate personalized multitask learning (MTL) models for prediction from time-series data. In this section, we describe the problem motivation and introduce the MTL design components that are required to formulate an accurate MTL time-series solution. We focus on the class of time-series problems of the following characteristics:
For input, the developed model should be capable of receiving multiple time-series channels, which may include multiple
Multitask feature-based learning for time-series (MTFL-TS)
To ensure a systematic approach in the design of the multitask feature-based learning for time-series (MTFL-TS) model, we investigate the different elements of the MTL framework in Eq. (3). In particular, we explore the following:
Dynamic, along with static, feature set to capture the rich temporal patterns in the consecutive data samples of the time-series.
Similarity measures capable of capturing common information across all user entities learned simultaneously.
Regularization measures to
Multitask deep learning for time-series (MTDL-TS)
In this section, we show how the framework can be used to derive a new MTL deep learning for time-series (MTDL-TS) model composed of a hierarchical convolutional-recurrent network. We build the network components by again addressing the same three design elements of MTL models for time-series.
The input to the network can be composed of multiple sensor time-series sequences, where each sensor is labeled as , . Each sensor generates readings over time and across multiple channels, thus
Experiments & results
We evaluate our framework and the presented MTFL-TS and MTDL-TS methods under the classification task of human activity recognition. Activity recognition is chosen to evaluate the multitask personalized time-series models since it has been widely studied, provides a large array of widely used benchmarkdatasets, and serves to provide a good illustration of the potentials of MTL and sequential information extraction in time-series.
Conclusion
In summary, this work targets addressing the limitation of annotated data for personalized models by making use of multi-task learning. We propose an MTL framework for accurate personalized predictions from sensing data that leverages similarity across entities being learned together for enriching limited annotated datasets and improved model generalization. The contributions of this work are three folds. First, we propose a systematic method for formulating solutions of MTL time-series models
CRediT authorship contribution statement
Reem A. Mahmoud: Conceptualization, Methodology, Software, Formal analysis, Investigation, Writing - original draft, Visualization. Hazem Hajj: Conceptualization, Methodology, Verification, Writing - review & editing, Supervision, Project adminstration. Fadi N. Karameh: Conceptualization, Methodology, Verification, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
The authors would like to acknowledge that this work was supported by the American University of Beirut’s University Research Board (AUB-URB) .
Reem A. Mahmoud is currently a Ph.D. candidate in the Electrical and Computer Engineering (ECE) department at the American University of Beirut (AUB). She received her B.S. in Electrical Engineering with high distinction from Alfaisal University in Riyadh, Saudi Arabia, and her M.E. from AUB in 2017. Her research is in theoretical Machine Learning (ML) with an interest in personalized ML, branching into problems of learning from limited data sources and advancing traditional transfer learning
References (64)
- et al.
Walking speed influences on gait cycle variability
Gait Posture
(2007) - et al.
Epileptic seizures may begin hours in advance of clinical onset: a report of five patients
Neuron
(2001) - et al.
Seizure prediction in patients with focal hippocampal epilepsy
Clin. Neurophysiol.
(2017) - et al.
The opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Pattern Recognit. Lett.
(2013) - et al.
Task-dependent multi-task multiple kernel learning for facial action unit detection
Pattern Recognit.
(2016) - et al.
Joint modeling of heterogeneous sensing data for depression assessment via multi-task learning
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.
(2018) - et al.
A robust human activity recognition system using smartphone sensors and deep learning
Future Gener. Comput. Syst.
(2018) Real-time human activity recognition from accelerometer data using convolutional neural networks
Appl. Soft Comput.
(2018)- et al.
Optimizing multi-sensor deployment via ensemble pruning for wearable activity recognition
Inf. Fusion
(2018) Darpa announces $2 Billion campaign to develop next wave of ai technologies
(2018)
A personalized pacing system for real-time physical activity advising
Psyche: personalised monitoring systems for care in mental health
Estimation of future glucose concentrations with subject-specific recursive linear models
Diabetes Technol. Therapeutics
An adaptive pattern learning framework to personalize online seizure prediction
IEEE Trans. Big Data
Complexity index from a personalized wearable monitoring system for assessing remission in mental health
IEEE J. Biomed. Health Inf.
Using accelerometer and gyroscopic measures to quantify postural stability
J. Athl. Train.
A 10-year study of the incidence of and factors predicting dementia in parkinson’s disease
Neurology
Combining multitask learning and short time series analysis in parkinson’s disease patients stratification
Emotion recognition in human-computer interaction
IEEE Signal Process. Mag.
Imagenet classification with deep convolutional neural networks
Cell phone-based biometric identification
Large-scale personalized human activity recognition using online multitask learning
IEEE Trans. Knowl. Data Eng.
Recurrent neural network for text classification with multi-task learning
A public domain dataset for human activity recognition using smartphones
Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units
Comput. J.
A large scale gathering system for activity data with mobile sensors
Identifying user traits by mining smart phone accelerometer data
Regularized multi–task learning
Toward personalized activity recognition systems with a semipopulation approach
IEEE Trans. Hum.-Mach. Syst.
Bayesian online multitask learning of gaussian processes
IEEE Trans. Pattern Anal. Mach. Intell.
Personalization without user interruption: Boosting activity recognition in new subjects using unlabeled data
Cited by (8)
Multi-Time Scale Aware Host Task Preferred Learning for WEEE return prediction
2024, Expert Systems with ApplicationsAerodynamic data predictions based on multi-task learning[Formula presented]
2022, Applied Soft ComputingMomentum portfolio selection based on learning-to-rank algorithms with heterogeneous knowledge graphs
2024, Applied IntelligenceGAN-based patient information hiding for an ECG authentication system
2023, Biomedical Engineering LettersMULTIMIX TFT: A MULTI-TASK MIXED-FREQUENCY FRAMEWORK WITH TEMPORAL FUSION TRANSFORMERS
2023, Proceedings of Machine Learning ResearchMulti-objective Learning to Overcome Catastrophic Forgetting in Time-series Applications
2022, ACM Transactions on Knowledge Discovery from Data
Reem A. Mahmoud is currently a Ph.D. candidate in the Electrical and Computer Engineering (ECE) department at the American University of Beirut (AUB). She received her B.S. in Electrical Engineering with high distinction from Alfaisal University in Riyadh, Saudi Arabia, and her M.E. from AUB in 2017. Her research is in theoretical Machine Learning (ML) with an interest in personalized ML, branching into problems of learning from limited data sources and advancing traditional transfer learning methods.
Hazem Hajj is an Associate Professor with the American University of Beirut (AUB) and a senior member of IEEE. Hazem joined AUB in 2008, and was a visiting Associate Professor at the University of Texas-Austin for one year in 2015–16. Before joining AUB, Hazem was a principal engineer at Intel Corporation. He received his Ph.D. from the University of Wisconsin-Madison in 1996, and his Bachelor from AUB with distinction. Hazem’s research interests include Machine Learning and Energy-Aware Computing, with special interests in Context Aware Sensing and Emotion Recognition. He has over 100 research publications in reputable journals and international conferences.
Fadi N. Karameh is an Associate Professor in the Electrical and Computer Engineering Department at the American University of Beirut (AUB) in Beirut, Lebanon. Prof. Karameh joined AUB in 2003 shortly after graduating from the Laboratory of Information and Decision Systems at the Massachusetts Institute of Technology (MIT) in Cambridge, USA. His research includes system-theoretic approaches in identification, estimation and signal processing, with an emphasis on neurophysiological signals and systems. His interdisciplinary interests include developing identification and estimation tools for understanding nonlinear dynamic large-scale interactions in brain cortical networks.