Human activity recognition using robust adaptive privileged probabilistic learning

Abstract

In this work, a supervised probabilistic approach is proposed that integrates the learning using privileged information (LUPI) paradigm into a hidden conditional random field (HCRF) model, called HCRF+, for human action recognition. The proposed model employs a self-training technique for automatic estimation of the regularization parameters of the objective function. Moreover, the method provides robustness to outliers by modeling the conditional distribution of the privileged information by a Student’s t-density function, which is naturally integrated into the HCRF+ framework. The proposed method was evaluated using different forms of privileged information on four publicly available datasets. The experimental results demonstrate its effectiveness concerning the state of the art in the LUPI framework using both hand-crafted and deep learning-based features extracted from a convolutional neural network.

Appendix: Conditional distribution of the privileged information

Recall that $\mathbf {x} \in \mathbb {R}^{M_{\mathbf {x}} \times T}$ is an observation sequence of length T and $\mathbf {x}^{*} \in \mathbb {R}^{M_{\mathbf {x}^{*}} \times T}$ corresponds to the privileged information of the same length. We partition the original set $\left( \mathbf {x}^{*}, \mathbf {x}\right) ^{T} \in \mathbb {R}^{M \times T}$ into two disjoint subsets, where $\mathbf {x}^{*}$ forms the first $M_{{\mathbf {x}}^{*}}$ components of $\left( \mathbf {x}^{*}, \mathbf {x}\right) ^{T} \in \mathbb {R}^{M \times T}$ and $\mathbf {x}$ comprises the remaining $M - M_{\mathbf {x}}$ components. If the joint distribution $p(\mathbf {x},\mathbf {x}^{*};\mathbf {w})$ follows a Student’s t-law, with mean vector $\mu =\left( \mu _{\mathbf {x}^{*}}, \mu _{\mathbf {x}}\right) ^{T}$, a real, positive definite, and symmetric $M \times M$ covariance matrix $\Sigma = \begin{pmatrix} \Sigma _{\mathbf {x}^{*}\mathbf {x}^{*}} &{} \Sigma _{\mathbf {x}^{*}\mathbf {x}} \\ \!\!\Sigma _{\mathbf {x}\mathbf {x}^{*}} &{} \!\!\Sigma _{\mathbf {x}\mathbf {x}} \end{pmatrix}$ and $\nu \in [0, \infty )$ corresponds to the degrees of freedom of the distribution [28], then the conditional distribution $p(\mathbf {x}|\mathbf {x}^{*};\mathbf {w})$ is also a Student’s t-distribution:

$$\begin{aligned} \begin{aligned} p(\mathbf {x}^{*}|\mathbf {x};\mathbf {w})&= \text {St}(\mathbf {x}^{*};\mu ^{*},\Sigma ^{*},\nu ^{*}) \\&= \frac{ \Gamma \left( \left( \nu ^{*}+M\right) /2\right) |\Sigma _{\mathbf {x}\mathbf {x}}|^{1/2} }{ \left( \pi \nu ^{*}\right) ^{M_{\mathbf {x}}/2} \Gamma \left( \left( \nu ^{*}+M_{\mathbf {x}}\right) /2\right) |\Sigma ^{*}|^{1/2} } \\&\quad \quad \times \frac{ \left[ 1+\frac{1}{\nu ^{*}} \mathbf {x}^{T}\Sigma _{\mathbf {x}\mathbf {x}}^{-1}\mathbf {x} \right] ^\frac{\left( \nu ^{*}+M_{\mathbf {x}}\right) }{2} }{\left[ 1+\frac{1}{\nu ^{*}} Z^{T}{\Sigma ^{*}}^{-1}Z \right] ^\frac{\left( \nu ^{*}+M\right) }{2}} \, . \end{aligned} \end{aligned}$$

(24)

The mean $\mu ^{*}$, the covariance matrix $\Sigma ^{*}$, and the degrees of freedom $\nu ^{*}$ of the conditional distribution $p(\mathbf {x}^{*}|\mathbf {x};\mathbf {w})$ are computed by the respective parts of $\mu$ and $\Sigma$:

$$\begin{aligned} \mu ^{*}&= \mu _{\mathbf {x}^{*}} - \Sigma _{\mathbf {x}^{*}\mathbf {x}} \Sigma _{\mathbf {x}\mathbf {x}}^{-1} \left( \mathbf {x} - \mu _{\mathbf {x}}\right) \, , \end{aligned}$$

(25)

$$\begin{aligned} \Sigma ^{*}&= \frac{\nu _{\mathbf {x}^{*}} + \left( \mathbf {x} - \mu _{\mathbf {x}}\right) ^{T} \Sigma _{\mathbf {x}\mathbf {x}}^{-1}\left( \mathbf {x} - \mu _{\mathbf {x}}\right) }{\nu _{\mathbf {x}^{*}} + M_{{\mathbf {x}}^{*}}} \nonumber \\&\quad \quad \times \left( \Sigma _{\mathbf {x}^{*}\mathbf {x}^{*}} - \Sigma _{\mathbf {x}^{*}\mathbf {x}}\Sigma _{\mathbf {x}\mathbf {x}}^{-1} \Sigma _{\mathbf {x}\mathbf {x}^{*}}\right) \, , \end{aligned}$$

(26)

$$\begin{aligned} \nu ^{*}&= \nu _{\mathbf {x}^{*}} + M_{{\mathbf {x}}^{*}} \, . \end{aligned}$$

(27)

The parameters $(\mu ,\Sigma ,\nu )$ of the joint Student’s t-distribution $p(\mathbf {x}^{*},\mathbf {x};\mathbf {w})$, which are defined by the corresponding partition of the vector $\left( \mathbf {x}^{*}, \mathbf {x}\right) ^{T}$, are estimated using the expectation-maximization (EM) algorithm [28]. Then, the parameters of the conditional distribution $p(\mathbf {x}^{*}|\mathbf {x};\mathbf {w})$ are computed using Eqs. (25)–(27).

It is worth noting that by letting the degrees of freedom $\nu ^{*}$ to go to infinity, we can recover the Gaussian distribution with the same parameters. If the data contain outliers, the degrees of freedom parameter $\nu ^{*}$ are weak and the mean and covariance of the data are appropriately weighted in order not to take into account the outliers.