Elsevier

Applied Soft Computing

Volume 96, November 2020, 106586
Applied Soft Computing

A systematic approach to multi-task learning from time-series data

https://doi.org/10.1016/j.asoc.2020.106586Get rights and content

Highlights

  • We address data deficiency limitations for personalized time-series prediction.

  • We propose a framework with 3 design components for multitask time-series models.

  • We introduce a new multitask learning convolutional-recurrent neural network.

  • We advance a state-of-the-art feature-based model through the proposed framework.

  • We present a comparison between the proposed deep learning and feature-based models.

Abstract

There have been significant advances in machine learning due to the profusion in data collection and computing resources. However, the need for large annotated datasets to train machine learning models remains a problematic constraint. To address the limitation of annotated data for personalized prediction, we propose a framework to enrich annotated time-series (TS) sensing data by way of transfer learning with new multi-task learning (MTL) models. Compared to previous MTL approaches for TS, this work introduces three contributions. First, we propose a systematic method to examine the efficiency of MTL approaches by exploring options for three key characteristics of MTL with TS: the choice of features that efficiently capture temporally dynamic information, the similarity measure that effectively models the commonality and uniqueness across tasks being learned, and the choice of regularization for achieving the best tradeoff between a model’s generalizability and accuracy. Second, we present an MTL deep learning model that is shown to achieve state-of-the-art performance for personalized human activity recognition from time-series. Experimental results on three benchmark activity recognition datasets and one activity recognition in-the-wild dataset show that the proposed framework provides performance gains over prior work while presenting a unified approach for designing MTL solutions for personalized time-series classification problems.

Introduction

Recently, DARPA [1] has promoted advances towards human–machine symbiosis where learning from little data is a critical challenge forward. This paper aims at addressing the limitation of annotated time-series data for personalized prediction models by way of transfer learning with a new systematic approach to building multi-task learning (MTL) models. Time-series sequences are of particular importance due to the present and foreseen abundance and advancements of the integration of wearable and IoT devices into everyday settings and applications.

Personalization in modeling is motivated by the fact that it is exhibited in many natural human interactions. Personalization is the model’s ability to extract features, also called descriptors, that are unique for a particular task (e.g. a particular user). Personalization becomes particularly important when the training data is a mixture of data from multiple tasks. As an illustration, consider human activity in which the model would need to be able to extract features that are unique to every user. Different people exhibit unique identifiers in posture, stride, and arm movements as people vary with age, body builds, and health conditions [2]. These differences resemble what we will refer to as unique identifiers, that is, features in the data that are tailored to a unique entity. Through capturing unique descriptors of an individual or entity being learned in a model, it is expected to witness gains in the accuracy of predictive models from sensing data [3], [4]. By accuracy, we are referring to the model’s ability to correctly label a classification task such as the activity for a particular user, which is also assessed by the log-loss likelihood measure.

In practice, nevertheless, a shortcoming of personalized modeling is that annotated datasets collected for a unique entity are typically not large enough. This limitation is often due to the time and effort involved in data labeling and that rich historical data is naturally limited for an entity, especially in cases where data is related to medical symptoms of different diseases such as glucose levels [5], seizure prediction [6], [7], [8], mental health [9], postural stability [10], and Parkinson’s disease [11], [12]. Hence, the size of personalized annotated datasets constitutes a bottleneck in achieving the desired performance in machine learning models.

This paper aims at addressing the limitation in annotated time-series data for personalized prediction models by way of transfer learning with new MTL models. The goal is to have personalized prediction models that are unique to individual entities while achieving knowledge transfer by exploiting the available commonality across entities. As an example, consider personalized activity recognition for individual users, where activity can be approached by considering consecutive action units to constitute high-level activities, such as walking or climbing stairs. These high-level activities include information patterns that can be understood as shared group behavior and other patterns that can be recognized as personalized to each user.

Traditionally, most researchers have approached the problem of learning models by one of two extreme approaches, population or personalized models, as shown in Fig. 1. Population models employ data collected from multiple entities to learn one unified prediction model for all entities [13], [14], while personalized models learn separate prediction models from each unique entity’s dataset [3], [15]. In both extreme cases, one task is learned at a time from the data. As a result, these models belong to a class of models called single task learning (STL). On the other hand, in the middle region of Fig. 1, multitask learning (MTL) models, introduced by R. Caruana [16], offer a tradeoff between both extremes where a compromise between specificity and generality of a model is achieved by learning personalized entity models while allowing for information sharing across entities. MTL learns multiple tasks simultaneously, leveraging their common information to improve the performance of all tasks equally. Several works have highlighted the superiority of MTL modeling in comparison to traditional STL methods, including the work of X. Sun et al. [17] and P. Lui et al. [18]. While their work sets strong grounds for MTL models, we present a generic and systematic approach for designing, evaluating, and improving MTL models.

This work proposes a framework for systematically investigating and building accurate personalized time-series MTL models while considering three primary design components: features capturing the time dynamics in data, similarity metrics reflecting degrees of commonality and uniqueness across entities, and generalization metrics to prevent overfitting. The framework enables the introduction of efficient new MTL models and advancing the prior state-of-the-art. The approach is successfully applied and tested resulting in two models for personalized continuous activity recognition. The first is an MTL deep learning approach that makes use of Convolutional Neural Networks (CNN) and Gated Recurrent Units (GRU), and the second advances on a previously developed MTL feature-based learning approach from the work of Sun et al. [17].

The main contributions of this work are:

  • 1.

    We present a systematic approach for designing and evaluating MTL models for personalized modeling from time-series data that is generic and applicable to any machine learning method.

  • 2.

    We propose a hierarchical MTL deep learning model for time-series classification that is composed of aconvolutional-recurrent neural network capable of (1) extracting features in an automated manner that is optimized and shared across entities and (2) extracting unique and shared relationships across entities through hard-parameter sharing of network layers across entities learned in parallel.

  • 3.

    We show an advancement of a prior MTL feature-based model for time-series data [17] by incorporating the 3 MTL components in the proposed systematic approach.

  • 4.

    We achieve state-of-the-art performance of the proposed MTL framework on three widely known human activity recognition benchmark datasets (OPPORTUNITY, UCI HAR, and DSA) and ALKAN of human activity recognition in-the-wild.

The proposed models are tested on the classification task of Human Activity Recognition (HAR) from sensing devices and evaluated on benchmark HAR datasets of OPPORTUNITY [19], UCI HAR [20], and DSA [21], as well as ALKAN [22] of HAR in-the-wild. The proposed methods show superiority in comparison to previous work on MTL models for time-series data and baseline approaches as well as competitive results when benchmarked against related work in literature, highlighting the strength of the proposed MTL systematic design approach for personalized modeling of time-series data.

The paper outline is as follows: Section 2 presents a summary and qualitative comparison of relevant and recent work. Section 3 defines the problem statement, general formulation, and design components. Section 4 presents the proposed MTL feature-based learning model, and Section 5 covers the MTL deep learning model. In Section 6, we describe the datasets, experimental setup, corresponding results and discuss the findings. We conclude the work in Section 7.

Section snippets

Related work

This section presents three outlooks on prior work: single-task learning (STL) personalized, multitask learning (MTL) feature-based, and MTL deep learning modeling approaches. We end the section with a qualitative comparison of prior literature in addressing the target research challenges and contributions of this work.

Problem statement

This work addresses the problem of designing accurate personalized multitask learning (MTL) models for prediction from time-series data. In this section, we describe the problem motivation and introduce the MTL design components that are required to formulate an accurate MTL time-series solution. We focus on the class of time-series problems of the following characteristics:

  • For input, the developed model should be capable of receiving multiple time-series channels, which may include multiple

Multitask feature-based learning for time-series (MTFL-TS)

To ensure a systematic approach in the design of the multitask feature-based learning for time-series (MTFL-TS) model, we investigate the different elements of the MTL framework in Eq. (3). In particular, we explore the following:

  • Dynamic, along with static, feature set to capture the rich temporal patterns in the consecutive data samples of the time-series.

  • Similarity measures capable of capturing common information across all user entities learned simultaneously.

  • Regularization measures to

Multitask deep learning for time-series (MTDL-TS)

In this section, we show how the framework can be used to derive a new MTL deep learning for time-series (MTDL-TS) model composed of a hierarchical convolutional-recurrent network. We build the network components by again addressing the same three design elements of MTL models for time-series.

The input to the network can be composed of multiple sensor time-series sequences, where each sensor is labeled as Sk, k{1...K}. Each sensor generates readings over time and across multiple channels, thus

Experiments & results

We evaluate our framework and the presented MTFL-TS and MTDL-TS methods under the classification task of human activity recognition. Activity recognition is chosen to evaluate the multitask personalized time-series models since it has been widely studied, provides a large array of widely used benchmarkdatasets, and serves to provide a good illustration of the potentials of MTL and sequential information extraction in time-series.

Conclusion

In summary, this work targets addressing the limitation of annotated data for personalized models by making use of multi-task learning. We propose an MTL framework for accurate personalized predictions from sensing data that leverages similarity across entities being learned together for enriching limited annotated datasets and improved model generalization. The contributions of this work are three folds. First, we propose a systematic method for formulating solutions of MTL time-series models

CRediT authorship contribution statement

Reem A. Mahmoud: Conceptualization, Methodology, Software, Formal analysis, Investigation, Writing - original draft, Visualization. Hazem Hajj: Conceptualization, Methodology, Verification, Writing - review & editing, Supervision, Project adminstration. Fadi N. Karameh: Conceptualization, Methodology, Verification, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The authors would like to acknowledge that this work was supported by the American University of Beirut’s University Research Board (AUB-URB) .

Reem A. Mahmoud is currently a Ph.D. candidate in the Electrical and Computer Engineering (ECE) department at the American University of Beirut (AUB). She received her B.S. in Electrical Engineering with high distinction from Alfaisal University in Riyadh, Saudi Arabia, and her M.E. from AUB in 2017. Her research is in theoretical Machine Learning (ML) with an interest in personalized ML, branching into problems of learning from limited data sources and advancing traditional transfer learning

References (64)

  • ChangH.-Y. et al.

    A personalized pacing system for real-time physical activity advising

  • ParadisoR. et al.

    Psyche: personalised monitoring systems for care in mental health

  • Eren-OrukluM. et al.

    Estimation of future glucose concentrations with subject-specific recursive linear models

    Diabetes Technol. Therapeutics

    (2009)
  • XiaoC. et al.

    An adaptive pattern learning framework to personalize online seizure prediction

    IEEE Trans. Big Data

    (2017)
  • LanataA. et al.

    Complexity index from a personalized wearable monitoring system for assessing remission in mental health

    IEEE J. Biomed. Health Inf.

    (2015)
  • AlbertsJ.L. et al.

    Using accelerometer and gyroscopic measures to quantify postural stability

    J. Athl. Train.

    (2015)
  • HughesT. et al.

    A 10-year study of the incidence of and factors predicting dementia in parkinson’s disease

    Neurology

    (2000)
  • ValmarskaA. et al.

    Combining multitask learning and short time series analysis in parkinson’s disease patients stratification

  • CowieR. et al.

    Emotion recognition in human-computer interaction

    IEEE Signal Process. Mag.

    (2001)
  • KrizhevskyA. et al.

    Imagenet classification with deep convolutional neural networks

  • KwapiszJ.R. et al.

    Cell phone-based biometric identification

  • . Rich Caruana, Multitask learning: A knowledge-based source of inductive bias, in: Proc. of the 10th Int’l Conference...
  • SunX. et al.

    Large-scale personalized human activity recognition using online multitask learning

    IEEE Trans. Knowl. Data Eng.

    (2012)
  • LiuP. et al.

    Recurrent neural network for text classification with multi-task learning

    (2016)
  • AnguitaD. et al.

    A public domain dataset for human activity recognition using smartphones

  • BarshanB. et al.

    Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units

    Comput. J.

    (2014)
  • HattoriY. et al.

    A large scale gathering system for activity data with mobile sensors

  • WeissG.M. et al.

    Identifying user traits by mining smart phone accelerometer data

  • EvgeniouT. et al.

    Regularized multi–task learning

  • HongJ.-H. et al.

    Toward personalized activity recognition systems with a semipopulation approach

    IEEE Trans. Hum.-Mach. Syst.

    (2015)
  • PillonettoG. et al.

    Bayesian online multitask learning of gaussian processes

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2008)
  • FallahzadehR. et al.

    Personalization without user interruption: Boosting activity recognition in new subjects using unlabeled data

  • Cited by (8)

    View all citing articles on Scopus

    Reem A. Mahmoud is currently a Ph.D. candidate in the Electrical and Computer Engineering (ECE) department at the American University of Beirut (AUB). She received her B.S. in Electrical Engineering with high distinction from Alfaisal University in Riyadh, Saudi Arabia, and her M.E. from AUB in 2017. Her research is in theoretical Machine Learning (ML) with an interest in personalized ML, branching into problems of learning from limited data sources and advancing traditional transfer learning methods.

    Hazem Hajj is an Associate Professor with the American University of Beirut (AUB) and a senior member of IEEE. Hazem joined AUB in 2008, and was a visiting Associate Professor at the University of Texas-Austin for one year in 2015–16. Before joining AUB, Hazem was a principal engineer at Intel Corporation. He received his Ph.D. from the University of Wisconsin-Madison in 1996, and his Bachelor from AUB with distinction. Hazem’s research interests include Machine Learning and Energy-Aware Computing, with special interests in Context Aware Sensing and Emotion Recognition. He has over 100 research publications in reputable journals and international conferences.

    Fadi N. Karameh is an Associate Professor in the Electrical and Computer Engineering Department at the American University of Beirut (AUB) in Beirut, Lebanon. Prof. Karameh joined AUB in 2003 shortly after graduating from the Laboratory of Information and Decision Systems at the Massachusetts Institute of Technology (MIT) in Cambridge, USA. His research includes system-theoretic approaches in identification, estimation and signal processing, with an emphasis on neurophysiological signals and systems. His interdisciplinary interests include developing identification and estimation tools for understanding nonlinear dynamic large-scale interactions in brain cortical networks.

    View full text