Extracting a low-dimensional predictable time series

Abstract

Large scale multi-dimensional time series can be found in many disciplines, including finance, econometrics, biomedical engineering, and industrial engineering systems. It has long been recognized that the time dependent components of the vector time series often reside in a subspace, leaving its complement independent over time. In this paper we develop a method for projecting the time series onto a low-dimensional time-series that is predictable, in the sense that an auto-regressive model achieves low prediction error. Our formulation and method follow ideas from principal component analysis, so we refer to the extracted low-dimensional time series as principal time series. In one special case we can compute the optimal projection exactly; in others, we give a heuristic method that seems to work well in practice. The effectiveness of the method is demonstrated on synthesized and real time series.

Appendix A derivation of (6)

We show how to derive the expression (6) in this appendix. For simplicity, we ignore the superscript $k+1$ in $A^{k+1}$, $A_i^{k+1}$, $i=1,\ldots ,M$, and $S_\tau ^{k+1}$, $\tau \in {\mathbf{Z}}$, and the superscript k in $W^k$.

When A is fixed, we have

$$\begin{aligned} \begin{array}{ll} f(w) {}= \mathop \mathbf{Tr}\left( -2A\begin{bmatrix}S_1 \\ S_2 \\ \vdots \\ S_M\end{bmatrix} +A\begin{bmatrix}S_0 {} S_1^T {} \cdots {} S_{M-1}^T\\ S_{1} {} S_0 {} \cdots {} S_{M-2}^T\\ \vdots {} \vdots {} \ddots {}\vdots \\ S_{M-1} {} S_{M-2} {} \cdots {} S_0 \end{bmatrix}A^T\right) \\ {}=-2\sum _{i=1}^M\mathop \mathbf{Tr}(A_iS_i) \\ {}\quad + \mathop \mathbf{Tr}\begin{bmatrix}S_0 {} S_1^T {} \cdots {} S_{M-1}^T\\ S_{1} {} S_0 {} \cdots {}S_{M-2}^T\\ \vdots {} \vdots {} \ddots {}\vdots \\ S_{M-1} {} S_{M-2} {} \cdots {} S_0 \end{bmatrix}\begin{bmatrix} A_1^TA_1 {} A_1^TA_2 {} \cdots {} A_1^TA_M\\ A_2^TA_1 {} A_2^TA_2 {} \cdots {} A_2^TA_M\\ \vdots {} \vdots {} \ddots {}\vdots \\ A_M^TA_1 {} A_M^TA_2 {} \cdots {} A_M^TA_M\end{bmatrix}. \end{array} \end{aligned}$$

We divide $A_i$, $i=1,2,\ldots ,M$ into the following submatrices,

$$\begin{aligned} A_i = \begin{bmatrix} A_{i,11} &{} A_{i,12} \\ A_{i,21} &{} A_{i,22} \end{bmatrix} \quad \text {for} \; i = 1,2,\ldots ,M, \end{aligned}$$

where $A_{i,11} \in {\mathbf{R}}^{k\times k}$, $A_{i,12} \in {\mathbf{R}}^{k\times 1}$, $A_{i,21} \in {\mathbf{R}}^{1\times k}$, $A_{i,22} \in {\mathbf{R}}$. With this notation, we can expand $\mathop \mathbf{Tr}(A_iS_i)$ as

$$\begin{aligned} \begin{array}{ll} \mathop \mathbf{Tr}(A_iS_i) {}= \mathop \mathbf{Tr}\begin{bmatrix}A_{i,11} {} A_{i,12} \\ A_{i,21} {} A_{i,22} \end{bmatrix} \begin{bmatrix}W^T\Sigma _iW {} W^T\Sigma _iw \\ w^T \Sigma _i W {} w^T\Sigma _iw \end{bmatrix}\\ {}= w^T(A_{i,22}\Sigma _i)w + (\Sigma _iWA_{i,12}+\Sigma _i^TW A_{i,21}^T)^Tw + d, \end{array} \end{aligned}$$

where d is a constant. For the second term in f(w), we have

$$\begin{aligned} \begin{array}{ll} {}\mathop \mathbf{Tr}\begin{bmatrix}S_0 {} S_1^T {} \cdots {} S_{M-1}^T\\ S_{1} {} S_0 {} \cdots {} S_{M-2}^T\\ \vdots {} \vdots {} \ddots {}\vdots \\ S_{M-1} {} S_{M-2} {} \cdots {} S_0 \end{bmatrix}\begin{bmatrix}A_1^TA_1 & {} A_1^TA_2 {} \cdots {} A_1^TA_M\\ A_2^TA_1 {} A_2^TA_2 {} \cdots {} A_2^TA_M\\ \vdots {} \vdots {} \ddots {}\vdots \\ A_M^TA_1 {} A_M^TA_2 {} \cdots {} A_M^TA_M\end{bmatrix} \\ {}=\mathop \mathbf{Tr}(S_0A_1^TA_1+S_1^TA_2^TA_1+\cdots +S_{M-1}^TA_M^TA_1) + \mathop \mathbf{Tr}(S_1A_1^TA_2+S_0A_2^TA_2+\cdots \\ \quad{}+S_{M-2}^TA_M^TA_2) + \cdots + \mathop \mathbf{Tr}(S_{M-1}A_1^TA_M+S_{M-2}^TA_2^TA_M+\cdots +S_0 A_M^TA_M) \\ {}= \sum _{i, j}S_{j-i}A_i^TA_j, \end{array} \end{aligned}$$

where $\mathop \mathbf{Tr}(S_{j-i}A_i^TA_j)$ can be expanded as

$$\begin{aligned} \begin{array}{ll}&\mathop \mathbf{Tr}(S_{j-i}A_i^TA_j) \\ {}= \begin{bmatrix}W^T\Sigma _{j-i}W {} W^T \Sigma _{j-i}w\\ w^T\Sigma _{j-i}W {} w^T\Sigma _{j-i}w \end{bmatrix} \begin{bmatrix} A_{i,11}^TA_{j,11} +A_{i,21}^TA_{j,21} {} A_{i,11}^T A_{j,12}+A_{i,21}^TA_{j,22}\\ A_{i,12}^TA_{j,11}+A_{i,22}A_{j,21} {} A_{i,12}^TA_{j,12} + A_{i,22}A_{j,22}\end{bmatrix}\\ {}= (A_{i,12}^TA_{j,11}+A_{i,22}A_{j,21})W^T\Sigma _{j-i}w + (A_{i,11}^TA_{j,12}+A_{i,21}^TA_{j,22})^TW^T\Sigma _{j-i}^Tw\\ \quad{}+ w^T(A_{i,12}^TA_{j,12}+A_{i,22}A_{j,22})\Sigma _{j-i}w. \end{array} \end{aligned}$$

Summing all terms, we can obtain the following expression for f(w),

$$\begin{aligned} f(w) = w^TBw - 2c^Tw+d, \end{aligned}$$

where d is a constant and

$$\begin{aligned} \begin{array}{ll} B &{}= \sum \limits _{1 \le i,j \le M} (A_{i,12}^TA_{j,12}+A_{i,22}A_{j,22})\Sigma _{j-i} - \sum \limits _{i=1}^M A_{i,22}(\Sigma _i+\Sigma _i^T), \\ c &{}= \sum \limits _{i=1}^M(\Sigma _iWA_{i,12}+\Sigma _i^TWA_{i,21}^T) - \sum \limits _{1\le i< j \le M}\Sigma _{j-i}^TW(A_{j,11}^TA_{i,12}+A_{i,22} A_{j,21}^T) \\ &{}\quad - \sum \limits _{1\le i < j \le M}\Sigma _{j-i}W(A_{i,11}^TA_{j,12}+A_{j,22} A_{i,21}^T). \end{array} \end{aligned}$$

The constant term can be ignored when we want to minimize f(w). It is easy to show that $B \succ 0$.