Change points detection and parameter estimation for multivariate time series

Abstract

In this paper, we propose a method to estimate the number and locations of change points and further estimate parameters of different regions for piecewise stationary vector autoregressive models. The procedure decomposes the problem of change points detection and parameter estimation along the component series. By reformulating the change point detection problem as a variable selection one, we apply group Lasso method to estimate the change points initially. Then, from the preliminary estimate of change points, a subset is selected based on the loss functions of Lasso method and a backward elimination algorithm. Finally, we propose a Lasso + OLS method to estimate the parameters in each segmentation for high-dimensional VAR models. The consistent properties of the estimation for the number and the locations of the change points and the VAR parameters are proved. Simulation experiments and real data examples illustrate the performance of the method.

Appendix Proofs of Theorems

Since the estimation problem decomposes across different time series $Y_{it}$ with regressive coefficient vector $B_{i\cdot }^j$, the number of change points $m_i$ and change points $t_k^i$, it is enough to show that the estimation ${\hat{B}}_{i\cdot }^j$, ${\hat{m}}_i$ and ${\hat{t}}_k^i$ are consistent and then apply the union bound to show that the whole parameter matrix B and change points are estimated consistent.

Suppose

$$\begin{aligned}&I_{min}^i=\min \limits _{1\le j\le m+1}\vert t_{j}^i-t_{j-1}^i\vert ,~~~~J_{min}^i=\min \limits _{\begin{array}{c} 1\le j\le m+1\\ 1\le k\le d \end{array}}\vert B^{j}_{ik}-B^{j-1}_{ik}\vert ,\\&J_{max}^i=\max \limits _{\begin{array}{c} 1\le j\le m+1\\ 1\le k\le d \end{array}}\vert B^{j}_{ik}-B^{j-1}_{ik}\vert . \end{aligned}$$

Lemma 1

Suppose Assumptions 1 and 2 hold. For any $c_0>0$, there exists some constant $C>0$ such that for $n\ge Clog(nd^2p)$

$$\begin{aligned}&P\left( \max \limits _{1\le h\le n}\max \limits _{1\le k\le d}\max \limits _{1\le l\le p}\vert \frac{1}{n}\sum _{t=h-l-1}^{n-p-l} Y_{k,p+t}\epsilon _{i,p+t+l}\vert \right. \nonumber \\&\quad \left. \ge c_0\sqrt{\frac{1}{n}log (nd^2p)}\right) \le Cexp^{-c_2log(nd^2p)} \end{aligned}$$

(32)

Proof

For any t, l, h, $cov(Y_{k,p+t}\epsilon _{i,p+t+l})=0$, it follows from Proposition 2.4(b) of Basu and Michailidis (2015) that for $d\times 1$ vectors u, v with zeros except the k, i-th elements, respectively,

$$\begin{aligned}&P\left( \left| u^\prime \left( \frac{1}{n}\sum _{t=h-l-1}^{n-p-l} Y_{k,p+t}\epsilon _{i,p+t+l}\right) v\right| \right. \nonumber \\&\left. \quad \ge k_1\eta \right) \le 6exp^{-k_2n\min (\eta ,\eta ^2)}, \end{aligned}$$

(33)

Suppose $\eta =k_3\sqrt{\frac{1}{n}log (nd^2p)}$, and $k_3>0$ be large enough yieal Eq. (32). $\square$

Lemma 2

Suppose Assumptions 1 and 2 hold. For any $c_i>0,i=1,2,3$,

$$\begin{aligned} \begin{aligned}&P\left\{ \max \limits _{\begin{array}{c} 1\le j\le m_i,\vert t_j-s\vert >n\gamma _n\\ s\le t_j \end{array}}\max \limits _{1\le k\le d}\max \limits _{1\le l\le p}\left( (t_j^i-s)^{-1}\left| \sum _{t=s}^{t_j^i-1} Y_{i,t}Y_{k,t+l}\right. \right. \right. \\&\left. \left. \left. \qquad -{\mathbb {E}}Y_{i,t}Y_{k,t+l}\right| \right) \ge c_3\sqrt{\frac{log(pd^2)}{n\gamma _n}}\right\} \\&\quad \le c_1 exp^{-c_2log(pd^2)}, \end{aligned} \end{aligned}$$

(34)

and

$$\begin{aligned}&P\left\{ \max \limits _{\begin{array}{c} 1\le j\le m_i,\vert t_j-s\vert >n\gamma _n \\ s\le t_j \end{array}}\max \limits _{1\le k\le d}\max \limits _{1\le l\le p}\left( (t_j^i-s)^{-1}\vert \sum _{t=s}^{t_j^i-1} Y_{k,t+l}\epsilon _{i,t}\right) \right. \nonumber \\&\left. \quad \ge c_3\sqrt{\frac{log(pd^2)}{n\gamma _n}}\right\} \nonumber \\&\quad \le c_1 exp^{-c_2log(pd^2)}. \end{aligned}$$

(35)

Proof

From Proposition 2.4 of Basu and Michailidis (2015),

$$\begin{aligned}&P\left( \left| \frac{\sum _{t=s}^{t_j^i-1} Y_{i,t}Y_{k,t+l}-{\mathbb {E}}Y_{i,t}Y_{k,t+l} }{t_j^i-s}\right| \ge k_1\eta \right) \nonumber \\&\quad \le 6exp^{-k_2n\gamma _n\min (\eta ,\eta ^2)}, \end{aligned}$$

(36)

and

$$\begin{aligned} P\left( \left| \frac{\sum _{t=s}^{t_j^i-1} Y_{k,t+l}\epsilon _{i,t}}{t_j^i-s }\right| \ge k_1\eta \right) \le 6exp^{-k_2n\gamma _n\min (\eta ,\eta ^2)}, \end{aligned}$$

(37)

Let $\eta =k_3\sqrt{\frac{log(pd^2)}{n\gamma _n}}$, we get the results. $\square$

Lemma 3

Let ${\hat{\phi }}_{i}$ and $\phi _i$ be defined as in Eq. (7), $\mathbf X _{l-1}$ denote the row in matrix $\mathbf X$ where $X_{l-1}$ is located in. Under the condition of Theorem 1, we have

$$\begin{aligned}&\sum _{l={\hat{t}}_j^i}^n\mathbf X _{l-1}\left( Y_{i,l} -\sum _{q=1}^l{\hat{\phi }}_{i,q}{} \mathbf X _{l-1}^T\right) +\frac{1}{2}n\lambda _n{\hat{\phi }}_{i{\hat{t}}_j}/\Vert {\hat{\phi }}_{i,{\hat{t}}_j} \Vert =0, \nonumber \\&\quad where ~~~~{\hat{\phi }}_{i,{\hat{t}}_j}\ne 0 \end{aligned}$$

(38)

and

$$\begin{aligned}&\left\| \sum _{l=q}^n\mathbf X _{l-1}\left( Y_{i,l}-\sum _{q=1}^l{\hat{\phi }}_{i,q}{} \mathbf X _{l-1}^T\right) \right\| \le \frac{n\lambda _n}{2}, \nonumber \\&\quad q=p+1,\ldots ,n. \end{aligned}$$

(39)

Lemma 3 concerns the KKT conditions of the group Lasso algorithm. Here, we omit the proof which can be deduced directly.

Proof of Theorem 1

From the similar proof strategy to Theorem 2.1 in Chan et al. (2014) (using Lemma 1 to replace the Lemma A.1), for some $m_n=o(\lambda _n^{-1})$, then with some $c_0>0$ and high probability as $n\rightarrow \infty$,

$$\begin{aligned} \frac{1}{n}\Vert \varvec{X}(\hat{\varvec{\phi }_i}-\varvec{\phi }_i)\Vert ^2\le 4pdc_0m_nM_B^i\sqrt{\frac{log(npd^2)}{n}}, \end{aligned}$$

(40)

where $\max \limits _{\begin{array}{c} 1\le k\le d\\ 1\le j\le m+1 \end{array}}\vert B_{ik}^j\vert \le M_B^i$. Then,

$$\begin{aligned} \frac{1}{n}\Vert \varvec{X}(\hat{\varvec{\phi }}-\varvec{\phi })\Vert ^2\le 4pd^2c_0m_nM_B\sqrt{\frac{log(npd^2)}{n}}. \end{aligned}$$

(41)

$\square$

Proof of Theorem 2

Firstly, for $i=1,\ldots ,d$, we prove

$$\begin{aligned} P\left( \max \limits _{1\le j\le m_i}\vert {\hat{t}}_j^i-t_j^i\vert \le n\gamma _n\right) \rightarrow 1,~~as~~n\rightarrow \infty . \end{aligned}$$

(42)

Suppose $T_{ij}=\{\vert {\hat{t}}_j^i-t_j^i\vert > n\gamma _n\}, j=1,2,\ldots ,m_i$, then

$$\begin{aligned}&P\left\{ \max \limits _{1\le j\le m_i}\vert {\hat{t}}_j^i-t_j^i\vert> n\gamma _n\right\} \le \sum _{j=1}^{m_i}P\{\vert {\hat{t}}_j^i-t_j^i\vert > n\gamma _n\}\nonumber \\&\quad =\sum _{j=1}^{m_i}P(T_{ij}). \end{aligned}$$

(43)

Let $C_n=\{\max \limits _{1\le j\le m_i}\vert {\hat{t}}_j^i-t_j\vert \le \min _i\vert t_j-t_{j-1}\vert /2\}$. To prove Eq. (42), it suffices to show that

$$\begin{aligned} \sum _{j=1}^{m_i}P(T_{ij}C_n)~~~~and~~~~\sum _{j=1}^{m_i} P(T_{ij}C_n^c)\rightarrow 0, \end{aligned}$$

(44)

where $C_n^c$ is the complement of the set $C_n$.

We only outline the proof for $\sum _{j=1}^{m_i} P(T_{ij} C_n)\rightarrow 0.$ The proof for $\sum _{j=1}^{m_i}P(T_{ij}C_n^c)\rightarrow 0$ is similar.

The number of change points $m_i$ has two cases, $m_i$ is fixed or $m_i\rightarrow \infty$. For the case $m_i$ is fixed, we consider two cases ${\hat{t}}_j^i<t_j^i$ and ${\hat{t}}_j^i>t_j^i$.

From the KKT conditions, we have

$$\begin{aligned} \left\| \sum _{t={\hat{t}}_j^i}^{t_j^i-1}X_{t-1}\left( Y_{it}-{\hat{B}}_{i\cdot }^{j+1}X_{t-1}^T\right) \right\| \le n\lambda _n, \end{aligned}$$

(45)

which implies that

$$\begin{aligned}&\left\| \sum _{t={\hat{t}}_j^i}^{t_j^i-1}X_{t-1}\epsilon _{it}+\sum _{t={\hat{t}}_j^i}^{t_j^i-1}X_{t-1}\left( (B_{i\cdot }^{j}-B_{i\cdot }^{j+1})X_{t-1}^T\right) \right. \nonumber \\&\left. \quad +\sum _{t={\hat{t}}_j^i}^{t_j^i-1}X_{t-1}\left( (B_{i\cdot }^{j+1}-{\hat{B}}_{i\cdot }^{j+1})X_{t-1}^T\right) \right\| \le n\lambda _n. \end{aligned}$$

(46)

Then, for ${\hat{t}}_j^i<t_j^i$,

let ${\mathscr {B}}_j=\Big \Vert \sum _{t={\hat{t}}_j^i}^{t_j^i-1}X_{t-1}\left( (B_{i\cdot }^{j})^T-(B_{i\cdot }^{j+1})^T\right) X_{t-1}\Big \Vert$

$$\begin{aligned} \begin{aligned}&P(T_{ij}C_n)\\&\quad \le P\left( \left\{ \frac{1}{3}{\mathscr {B}}_j\le n\gamma _n\right\} \bigcap \left\{ \vert {\hat{t}}_j^i-t_j^i\vert>n\gamma _n\right\} \right) \\&\qquad +P\left( \left\{ \left\| \sum _{t={\hat{t}}_j^i}^{t_j^i-1}X_{t-1}\epsilon _{it}\right\|>\frac{1}{3}{\mathscr {B}}_j\right\} \bigcap \left\{ \vert {\hat{t}}_j^i-t_j^i\vert>n\gamma _n\right\} \right) \\&\qquad +P\left( \left\{ \left\| \sum _{t={\hat{t}}_j^i}^{t_j^i-1}X_{t-1}\left( (B_{i\cdot }^{j+1}-{\hat{B}}_{i\cdot }^{j+1})X_{t-1}^T\right) \right\| \right. \right. \\&\left. \left. \quad >\frac{1}{3}{\mathscr {B}}_j\right\} \bigcap (T_{ij}C_n)\right) \\&\quad =P(T_{ij1})+P(T_{ij2})+P(T_{ij3}). \end{aligned} \end{aligned}$$

(47)

In the similar ways as Theorem 2.2 Chan et al. (2014), using Lemma 2 and Lemma 3 to replace their Lemma A.2 and Lemma A.3, respectively, we can get that $P(T_{ij1})\rightarrow 0$, $P(T_{ij2})\rightarrow 0$ and $P(T_{ij3})\rightarrow 0$. Combining these results, we have $P(T_{ij}C_n\cap \{{\hat{t}}_j^i<t_j^i\})\rightarrow 0$. The proof of $P(T_{ij}C_n\cap \{{\hat{t}}_j^i>t_j^i\})\rightarrow 0$ is similar. Then, $P(T_{ij}C_n)\rightarrow 0$.

From Lemma 2, one can prove the case $m_i\rightarrow \infty$. The rate of convergence of the $P(T_{ij}),j=1,\ldots ,m_i$ can be fast enough to get $\sum _{j=1}^{m_i} P(T_{ij})\rightarrow 0$. Then, Eq. (42) is proved.

Theorem 2 is proved from the definition of ${\hat{t}}_j,j=1,\ldots ,{\hat{m}}$ and $\max \limits _{1\le j\le m}\vert {\hat{t}}_j-t_j\vert \le \max \limits _{1\le i\le d}\max \limits _{1\le j\le m}\vert {\hat{t}}_j^i-t_j^i\vert$. $\square$

Proof of Theorem 3

Firstly, we prove that for $i=1,\ldots ,d$, if Assumptions 1 to 3 are satisfied, then

$$\begin{aligned} P\left( \vert {\mathscr {A}}_{ni}\vert \ge m_i\right) \rightarrow 1, \end{aligned}$$

(48)

and

$$\begin{aligned} P\left\{ d_H({\mathscr {A}}_{ni}, {\mathscr {A}}_i)\le n\lambda _n\right\} \rightarrow 1. \end{aligned}$$

(49)

Applying Lemma 3 and Lemma 2 to similar augment of Theorem 2, according to the procedure of proof of Chan et.al.(2014), Eqs. (48) and (49) can be proved.

Then, from ${\mathscr {A}}_n=\cup _{i=1}^d{\mathscr {A}}_{ni}$, we get $\left( \vert {\mathscr {A}}_{n}\vert \ge m\right) \subseteq \left( \vert {\mathscr {A}}_{ni}\vert \ge m_i\right)$ and $d_H({\mathscr {A}}_{n}, {\mathscr {A}})=d_H({\mathscr {A}}_{ni}, {\mathscr {A}}_i)$, which completes the proof of Theorem 3. $\square$

Proof of Theorem 4

In the proof of the theorem, we will apply the conclusion of Lemma 4 in Safikhani and Shojaie (2017), for $m_r<m$

$$\begin{aligned}&P\left( \min \limits _{t_1,\ldots ,t_{m_r}}L_n(t_1,\ldots ,t_{m_r};\eta _n)\right. \nonumber \\&\left. \quad >\sum _{t=1}^{n}\Vert \epsilon _t\Vert ^2+c_1I_{min}-c_2m_rn\gamma _nS^2\right) \rightarrow 1. \end{aligned}$$

(50)

We prove the first conclusion by showing (a)$P({\hat{m}}^*<m)\rightarrow 0$ and (b) $P({\hat{m}}^*>m)\rightarrow 0$.

For (a) $P({\hat{m}}^*<m)\rightarrow 0$, Theorem 3 implies that there are points ${\hat{t}}_{nj}\in {\mathscr {A}}_n$ such that $max_{1\le j\le m}\vert {\hat{t}}_{nj}-t_j\vert \le n\gamma _n$.

By similar arguments as in Theorem 4 (Safikhani and Shojaie (2017)), we get that

$$\begin{aligned} L_n({\hat{t}}_{n1},\ldots ,{\hat{t}}_{nm})\le \sum _{t=1}^n\Vert \epsilon _t\Vert +Kmn\gamma _nS^2. \end{aligned}$$

(51)

By Eq. (50), we get

$$\begin{aligned} \begin{aligned}&IC({\hat{t}}_1^*,\ldots ,{\hat{t}}_{{\hat{m}}^*}^*)\\&\quad =L_n({\hat{t}}_1^*,\ldots ,{\hat{t}}_{{\hat{m}}^*}^*)+{\hat{m}}^*\omega _n\\&\quad >\sum _{t=1}^{n}\Vert \epsilon _t\Vert ^2+c_1I_{min}-c_2{\hat{m}}^*n\gamma _nS^2+{\hat{m}}^*\omega _n\\&\quad \ge L_n({\hat{t}}_{n1},\ldots ,{\hat{t}}_{nm})+m\omega _n+c_1I_{min}\\&\qquad -c_2{\hat{m}}^*n\gamma _nS^2-(m-{\hat{m}}^*)\omega _n\\&\quad \ge L_n({\hat{t}}_{n1},\ldots ,{\hat{t}}_{nm})+m\omega _n. \end{aligned} \end{aligned}$$

(52)

The last inequality comes from the conditions $m\omega _n/I_{min}\rightarrow 0$ and $lim_{n\rightarrow \infty }n\gamma _nS^2/\omega _n\le 1$. Then, $P({\hat{m}}^*<m)\rightarrow 0$.

To prove (b), given ${\hat{m}}^*>m$,

$$\begin{aligned} L_n(t_1^*,\ldots ,t_{{\hat{m}}^*}^*,\eta _n)=\min \limits _{t_1,\ldots ,t_{{\hat{m}}^*}\subseteq {\mathscr {A}}_n}L_n(t_1,\ldots ,t_{{\hat{m}}^*}), \end{aligned}$$

(53)

Combining with Eq. (51), we get

$$\begin{aligned} \begin{aligned}&L_n({\hat{t}}_1^*,\ldots ,{\hat{t}}_{{\hat{m}}^*}^*,\eta _n) \le L_n({\hat{t}}_1,\ldots ,{\hat{t}}_{m},\eta _n)\\&\quad \le \sum _{t=1}^{n}\Vert \epsilon _t\Vert ^2+Kmn\gamma _nS^2 \end{aligned} \end{aligned}$$

(54)

When ${\hat{m}}^*>m$, we have that

$$\begin{aligned}&L_n({\hat{t}}_1^*,\ldots ,{\hat{t}}_{{\hat{m}}^*}^*,\eta _n) \ge L_n({\hat{t}}_1,\ldots ,{\hat{t}}_{{\hat{m}}^*}^*,{t}_1,\ldots ,{t}_{m},\eta _n)\nonumber \\&\quad \ge \sum _{t=1}^{n}\Vert \epsilon _t\Vert ^2-c_2{\hat{m}}^*n\gamma _nS^2, \end{aligned}$$

(55)

Combining Eqs. (54) and (55), with the similar discussion as in Eq. (52), $P({\hat{m}}^*>m)\rightarrow 0)$. The second conclusion follows as Theorem 4 in Safikhani and Shojaie (2017). $\square$

Proof of Theorem 5

From Theorem 4 in Zhao and YU (2006), under the irrepresentable condition (Assumption 7) and Assumption 8, for $\lambda _n/n\rightarrow 0$ and $\lambda _n/n^{\frac{1+c}{2}}\rightarrow \infty$ with $0\le c<1$, the probability of the Lasso selecting wrong models satisfies $P({\hat{S}}_i\ne S_i)=0(e^{-n^{c_2}})$.

The second part is proved from the results Theorem 3 in Liu and Yu (2013). $\square$