Batch Size Selection for Variance Estimators in MCMC

Abstract

We consider batch size selection for a general class of multivariate batch means variance estimators, which are computationally viable for high-dimensional Markov chain Monte Carlo simulations. We derive the asymptotic mean squared error for this class of estimators. Further, we propose a parametric technique for estimating optimal batch sizes and discuss practical issues regarding the estimating process. Vector auto-regressive, Bayesian logistic regression, and Bayesian dynamic space-time examples illustrate the quality of the estimation procedure where the proposed optimal batch sizes outperform current batch size selection methods.

Appendices

Appendix A: Proof of Theorem 1

This section presents a proof of Theorem 1 and the optimal batch size results for the generalized OBM estimators. We will use the fact that

$$ \text{MSE}\left( \hat{\Sigma}_{w, ij} \right) = \left( \text{Bias}(\hat{\Sigma}_{w,ij}) \right)^{2} + \text{Var}\left( \hat{\Sigma}_{w, ij} \right). $$

The bias and variance results of the generalized OBM estimators are important in their own right, and are presented here separately. Denote $\lim _{n\rightarrow \infty } f(n)/g(n)=0$ by f(n) = o(g(n)) and if f(n)/g(n) stays uniformly bounded for all n, then we write f(n) = O(g(n)). Recall notation $\bar {Y}_{l}(k)=k^{-1}{\sum }_{t=1}^{k}Y_{l+t}$ for l = 0,...,n − k and $\bar {Y} = n^{-1} {\sum }_{t=1}^{n} Y_{t}$. Let $\bar {Y}_{l}^{(i)}(k)$ and $\bar {Y}^{(i)}$ denote the i th component of $\bar {Y}_{l}(k)$ and $\bar {Y}$, respectively.

Theorem 2

Suppose $\text {E}_{F}\|g\|^{2+\delta } < \infty$ for δ > 0 and ${\sum }_{k=1}^{b}k{\Delta }_{2}w_{n}(k)=1$. If {X_t} is a polynomially ergodic Markov chain of order ξ > (2 + 𝜖)(1 + 2/δ) for some 𝜖 > 0, then

$$ \text{Bias}\left( \hat{\Sigma}_{w,ij} \right) = \sum\limits_{k=1}^{b}{\Delta}_{2} w_{n}(k) {\Gamma}_{ij} + o\left( \frac{b}{n} \right) + o\left( \frac{1}{b} \right). $$

Proof

By Vats and Flegal (2018, Theorem 2), $|{\Gamma }_{ij}| < \infty$. Then under Assumption 1, and following (Song and Schmeiser 1995), for all i and j,

$$ \text{Cov}[\bar{Y}_{l}^{(i)}(k),\bar{Y}_{l}^{(j)}(k)]-\text{Cov}[\bar{Y}^{(i)},\bar{Y}^{(j)}]=\frac{n-k}{kn}\left( {\Sigma}_{ij}+\frac{n+k}{kn}{\Gamma}_{ij}+o\left( \frac{1}{k^{2}}\right)\right). $$

(4)

Since ${\sum }_{k=1}^{b}k{\Delta }_{2}w_{n}(k)=1$, by Eq. 4,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}\left( \hat{\Sigma}_{w,ij}\right)&=& \sum\limits_{k=1}^{b}\frac{(n-k+1)(n-k)k{\Delta}_{2}w_{n}(k)}{n^{2}} {\Sigma}_{ij} \\ && + \sum\limits_{k=1}^{b}\frac{(n-k+1)(n^{2}-k^{2}){\Delta}_{2}w_{n}(k)}{n^{3}} {\Gamma}_{ij}+o\left( \frac{b}{n}\right)+o\left( \frac{1}{b}\right)\\ &=&{\Sigma}_{ij}+ \sum\limits_{k=1}^{b}\frac{(n-k+1)(n^{2}-k^{2}){\Delta}_{2}w_{n}(k)}{n^{3}} {\Gamma}_{ij}+o\left( \frac{b}{n}\right)+o\left( \frac{1}{b}\right)\\ &=&{\Sigma}_{ij}+ \sum\limits_{k=1}^{b}{\Delta}_{2}w_{n}(k) {\Gamma}_{ij}+o\left( \frac{b}{n}\right)+o\left( \frac{1}{b}\right). \end{array} $$

□

Next, we obtain $\text {Var}\left (\hat {\Sigma }_{w, ij} \right )$. The proof here is under a more general strong invariance principle. For a function $f: \mathsf {X} \to \mathbb {R}^{p}$, assume there exists a p × p lower triangular matrix L, a non-negative increasing function ψ on the positive integers, a finite random variable D, and a sufficiently rich probability space Ω such that for almost all ω ∈Ω and for all n > n₀,

$$ \left\lVert \sum\limits_{t=1}^{n} f(X_{t}) - n \mathrm{E}_{F} f - LB(n)\right\rVert<D(\omega)\psi(n) \quad \text{with probability 1.} $$

(5)

Under the conditions of Lemma 1, Vats et al. (2018) establish (5) with ψ(n) = n^1/2−λ for some λ > 0. Appendix B contains a number of preliminary results, followed by the proof of Theorem 3 in Appendix C.

Theorem 3

Suppose (5) holds for f = g and f = g² (where the square is element-wise) such that $\mathrm {E}_{F}D^{4} < \infty$ and Assumption 1 holds. If

• 1. ${\sum }_{k=1}^{b}({\Delta }_{2}w_{k})^{2} = O\left (1/b^{2}\right ),$

• 2. $b\psi ^{2}(n)\log n \left ({\sum }_{k=1}^{b}|{\Delta }_{2}w_{n}(k)| \right )^{2}\rightarrow 0$, and

• 3. $\psi ^{2}(n){\sum }_{k=1}^{b}|{\Delta }_{2}w_{n}(k)|\rightarrow 0$, then,

$$ \begin{array}{@{}rcl@{}} \text{Var} &&\left( \hat{\Sigma}_{w,ij}\right)\\ &=&[{\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2}] \bigg[\frac{2}{3}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}k^{3} \frac{1}{n}+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{\Delta}_{2}w_{u} {\Delta}_{2}w_{t+u} \left( \frac{2}{3}u^{3}+u^{2}t\right) \frac{1}{n}\bigg]+o\left( \frac{b}{n}\right)\\ &:=&\left( [{\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2}] S \frac{b}{n}+o(1) \right) \frac{b}{n}. \end{array} $$

Appendix B: Preliminaries

Proposition 1

If variable X and Y are jointly normally distributed with

$$ \left[ \begin{array}{l} X \\ Y \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}l_{11}& l_{12}\\ l_{12}& l_{22} \end{array}\right]\right), $$

then $\mathrm {E}[X^{2}Y^{2}]=2l_{12}^{2}+l_{11}l_{22}$.

Proposition 2

Janssen and Stoica (1987) If X₁, X₂, X₃, and X₄ are jointly normally distributed with mean 0, then

$$ \mathrm{E}[X_{1}X_{2}X_{3}X_{4}]= \mathrm{E}[X_{1}X_{2}] \mathrm{E}[X_{3}X_{4}] + \mathrm{E}[X_{1}X_{3}] \mathrm{E}[X_{2}X_{4}] + \mathrm{E}[X_{1}X_{4}] \mathrm{E}[X_{2}X_{3}]. $$

Recall B = {B(t),t ≥ 0} is a p-dimensional standard Brownian motion. Let B⁽ⁱ⁾(t) be the i th component of vector B(t). Denote $\bar {B}=n^{-1}B(n)$, $\bar {B}_{l}(k)=k^{-1}[B(l+k)-B(l)]$. Let Σ = LL^T, where L is a lower triangular matrix. Let C(t) = LB(t) and C⁽ⁱ⁾(t) be the i th component of C(t). Suppose $\bar {C}^{(i)}_{l}(k)=k^{-1}(C^{(i)}(l+k)-C^{(i)}(l))$, and $\bar {C}^{(i)}=n^{-1}C^{(i)}(n)$.

We now present some preliminary results and notation for the proof of Theorem 3.

Lemma 2

For any 0 < c₂ < c₁ < 1, define

$$ A_{2}=\frac{(c_{1}b)^{2}}{n^{2}}\mathrm{E}\left[\left[\sum\limits_{l=0}^{n-c_{1}b} \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right)\right]^{2}\right], \text{ and} $$

$$ A_{3} = -\frac{c_{1}c_{2}b^{2}}{n^{2}} \mathrm{E} \left[\sum\limits_{l=0}^{n-c_{1}b}(\bar{C}_{l}^{(i)}(c_{1}b) - \bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{1}b) - \bar{C}^{(j)})\right]\left[\sum\limits_{l=0}^{n-c_{2}b}(\bar{C}_{l}^{(i)}(c_{2}b) - \bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{2}b) - \bar{C}^{(j)})\right]. $$

Then, A₂ and A₃ simplify to

$$ A_{2} = \left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{c_{1}b}{n}+{\Sigma}_{ij}^{2}-4{\Sigma}_{ij}^{2}\cdot\frac{c_{1}b}{n}\right]+o\left( \frac{b}{n}\right) \text{ and }, $$

$$ A_{3} = \frac{(c_{2} - 3c_{1})c_{2}}{3c_{1}}({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})\cdot \frac{b}{n} + 2\left( c_{1}+c_{2}\right)\cdot{\Sigma}_{ij}^{2}\cdot\frac{b}{n} - {\Sigma}_{ij}^{2} +o\left( \frac{b}{n}\right). $$

Proof

Denote

$$ a_{1}=\sum\limits_{l=0}^{n-c_{1}b} \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right)^{2} \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right)^{2}, $$

(6)

$$ a_{2}=\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}(\bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)})(\bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)})(\bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)}), $$

(7)

$$ a_{3}=\sum\limits_{s=b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s} \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)} \right). $$

Then

$$ \begin{array}{@{}rcl@{}} A_{2}&= \frac{c_{1}b^{2}}{n^{2}} \mathrm{E}[a_{1}+2a_{2}+2a_{3}]. \end{array} $$

(8)

First we calculate E[a₁] in Eq. 6. Let $U_{t}^{(i)}=B^{(i)}(t)-B^{(i)}(t-1)$, then $U_{t}^{(i)} {\overset {iid}{\sim }} N(0,1)$ for t = 1, 2,...,n and

$$ \bar{B}_{l}^{(i)}(c_{1}b)-\bar{B}^{(i)}=\frac{(n-c_{1}b)}{nc_{1}b}\sum\limits_{t=l+1}^{l + c_{1}b}U^{(i)}_{t}-\frac{1}{n}\sum\limits_{t=1}^{l}U_{t}^{(i)}-\frac{1}{n}\sum\limits_{t=l+c_{1}b+1}^{n}U_{t}^{(i)}. $$

Notice that $\mathrm {E}[\bar {B}_{l}^{(i)}(c_{1}b)-\bar {B}^{(i)}]=0$ for l = 0,..., (n − c₁b) and

$$ \text{Var} \left[\bar{B}_{l}^{(i)}(c_{1}b)-\bar{B}^{(i)} \right]=\Big(\frac{n-c_{1}b}{nc_{1}b}\Big)^{2}c_{1}b+\frac{n-c_{1}b}{n^{2}}=\frac{n-c_{1}b}{c_{1}bn}, $$

therefore

$$ \bar{B}_{l}(c_{1}b)-\bar{B}_{n}\sim N\left( 0,\ \frac{n-c_{1}b}{c_{1}bn}I_{p}\right) \Rightarrow \bar{C}_{l}(c_{1}b)-\bar{C}_{n} \sim N\left( 0,\ \frac{n-c_{1}b}{c_{1}bn}LL^{T}\right). $$

(9)

Now consider $\mathrm {E}[(\bar {C}_{l}^{(i)}(c_{1}b)-\bar {C}^{(i)})^{2}(\bar {C}_{l}^{(j)}(c_{1}b)-\bar {C}^{(j)})^{2}]= {\mathrm {E}[Z_{i}^{2}}{Z_{j}^{2}}]$ where $Z_{i}=\bar {C}_{l}^{(i)}(c_{1}b)-\bar {C}^{(i)}$ and $Z_{j}=\bar {C}_{l}^{(j)}(c_{1}b)-\bar {C}^{(j)}$. Recall Σ = LL^T, then

$$ \left[ \begin{array}{l} Z_{i} \\ Z_{j} \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \frac{n-c_{1}b}{c_{1}bn}\left[\begin{array}{ll}{\Sigma}_{ii}& {\Sigma}_{ij}\\ {\Sigma}_{ij}& {\Sigma}_{jj} \end{array}\right]\right). $$

Apply Proposition 1,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}\left[ \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right)^{2} \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right)^{2} \right] & = \left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}[{\Sigma}_{ij}^{2} + {\Sigma}_{ij}^{2}+{\Sigma}_{ii}{\Sigma}_{jj}]. \end{array} $$

(10)

Replacing Eqs. 10 in 6

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{1}]&= \sum\limits_{l=0}^{n-c_{1}b}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}{\Sigma}_{ij}^{2}+o\left( \frac{n}{b}\right). \end{array} $$

(11)

To calculate E[a₂] for s = 1, 2,..., (c₁b − 1), we require

$$ \mathrm{E} \left[ \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)} \right) \right]. $$

Notice that

$$ \begin{array}{@{}rcl@{}} \text{Cov}(\bar{C}_{l}(c_{1}b)-\bar{C}_{n},\ \bar{C}_{l+s}(c_{1}b)-\bar{C}_{n}) &=L\cdot \mathrm{E} \left[ \left( \bar{B}_{l}(c_{1}b)-\bar{B}_{n} \right) \left( \bar{B}_{l+s}(c_{1}b)-\bar{B}_{n} \right)^{T} \right]\cdot L^{T}. \end{array} $$

Consider each entry of $\mathrm {E}[(\bar {B}_{l}(c_{1}b)-\bar {B})(\bar {B}_{l+s}(c_{1}b)-\bar {B})^{T}]$. For i≠j,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{l+s}(c_{1}b)-\bar{B}^{(j)} \right] = \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right]\cdot \mathrm{E} \left[\bar{B}^{(j)}_{l+s}(c_{1}b)-\bar{B}^{(j)} \right]=0. $$

(12)

For i = j,

$$ \begin{array}{@{}rcl@{}} &&\mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(i)}_{l+s}(c_{1}b)-\bar{B}^{(i)} \right] \\ & =& \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b) \bar{B}^{(i)}_{l+s}(c_{1}b) \right] + \mathrm{E} \left[\bar{B}^{(i)} \bar{B}^{(i)} \right] - \mathrm{E} \left[ \bar{B}^{(i)} \bar{B}^{(i)}_{l+s}(c_{1}b) \right] - \mathrm{E} \left[ \bar{B}^{(i)} \bar{B}^{(i)}_{l}(c_{1}b) \right] \\ & = &\frac{1}{c_{1}b^{2}} \mathrm{E}\left[ \left( B^{(i)}(l+c_{1}b) - B^{(i)}(l) \right) \left( B^{(i)}(l+s+c_{1}b) - B^{(i)}(l + s) \right) \right] \\ && + \frac{1}{n^{2}}\mathrm{E}\left[ \left( B^{(i)}(n) \right)^{2} \right] - \frac{1}{nc_{1}b}\mathrm{E}\left[B^{(i)}(n) \left( B^{(i)} (l+c_{1}b+s) - B^{(i)}(l+s)\right) \right]\\ && - \frac{1}{nc_{1}b}\mathrm{E}\left[B^{(i)}(n) \left( B^{(i)} (l+c_{1}b) - B^{(i)}(l)\right) \right] \\ & =& \frac{n - c_{1}b}{c_{1}bn} - \frac{s}{n^{2}}. \end{array} $$

(13)

Combine Eqs. 13 and 12,

$$ \text{Cov} \left( \bar{C}_{l}(c_{1}b)-\bar{C}_{n},\ \bar{C}_{l+s}(c_{1}b)-\bar{C}_{n} \right) =\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)\cdot{\Sigma}. $$

Given Eqs. 9, 12 and 13 and let $Z_{1}=\bar {C}_{l}(c_{1}b)^{(i)}-\bar {C}^{(i)}_{n}$, $Z_{2}=\bar {C}_{l}(c_{1}b)^{(j)}-\bar {C}^{(j)}_{n}$, $Z_{3}=\bar {C}_{l+s}(c_{1}b)^{(i)}-\bar {C}^{(i)}_{n}$, $Z_{4}=\bar {C}_{l+s}(c_{1}b)^{(j)}-\bar {C}^{(j)}_{n}$, (Z₁,Z₂,Z₃,Z₄)^T has a 4-dimensional normal distribution with mean 0, and covariance matrix,

$$ \left[\begin{array}{llll}\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{ii}& \left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{ij} &\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right){\Sigma}_{ii}&\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right){\Sigma}_{ij}\\ & \left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{jj}& \left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right){\Sigma}_{ij}&\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right){\Sigma}_{jj}\\ & &\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{ii}&\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{ij} \\&&&\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{jj} \end{array}\right]. $$

Only upper triangle entries are presented due to symmetry of the matrix. By Proposition 2,

$$ \begin{array}{@{}rcl@{}} & \mathrm{E}[Z_{1}Z_{2}Z_{3}Z_{4}] =\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}{\Sigma}_{ij}^{2}+\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)^{2} \left[{\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2} \right] . \end{array} $$

(14)

Plug Eqs. 14 in 7,

$$ \begin{array}{@{}rcl@{}} &&\mathrm{E}[a_{2}] \\ &=&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s} \mathrm{E} \left[ \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)} \right) \right]\\ &=&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s} \mathrm{E}[Z_{1}Z_{2}Z_{3}Z_{4}]\\ &=&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}{\Sigma}_{ij}^{2}+\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)^{2}{\Sigma}_{ii}{\Sigma}_{jj}+\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)^{2}{\Sigma}_{ij}^{2}\right]. \end{array} $$

(15)

Notice that

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)^{2}\\ &=&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\frac{s^{2}}{{c_{1}^{4}}b^{4}}+\left( \frac{2}{{c_{1}^{2}}b^{2}n}-\frac{2}{{c_{1}^{3}}b^{3}}\right)s+\left( \frac{1}{{c_{1}^{2}}b^{2}}+\frac{1}{n^{2}}-\frac{2}{c_{1}bn}\right)\right]\\ &=&\sum\limits_{s=1}^{c_{1}b-1}\left[-\frac{s^{3}}{{c_{1}^{4}}b^{4}}+\left( \frac{n}{{c_{1}^{4}}b^{4}}+\frac{1}{{c_{1}^{3}}b^{3}}+\frac{1}{{c_{1}^{4}}b^{4}}-\frac{2}{{c_{1}^{2}}b^{2}n}\right)s^{2}\right. + \left( \frac{3}{{c_{1}^{2}}b^{2}}-\frac{2n}{{c_{1}^{3}}b^{3}}+\frac{2}{{c_{1}^{2}}b^{2}n}-\frac{2}{{c_{1}^{3}}b^{3}}-\frac{1}{n^{2}}\right)s\\ && \left.+\left( \frac{n}{{c_{1}^{2}}b^{2}}+\frac{3}{n}-\frac{3}{c_{1}b}+\frac{1}{{c_{1}^{2}}b^{2}}+\frac{1}{n^{2}}-\frac{2}{c_{1}bn}-\frac{c_{1}b}{n^{2}}\right)\right]\\ &=&-\frac{1}{{c_{1}^{4}}b^{4}}\left( \frac{{c_{1}^{4}}b^{4}}{4}-\frac{{c_{1}^{3}}b^{3}}{2}+\frac{{c_{1}^{2}}b^{2}}{4}\right)+\left( \frac{n}{{c_{1}^{4}}b^{4}}+\frac{1}{{c_{1}^{3}}b^{3}}+\frac{1}{{c_{1}^{4}}b^{4}}-\frac{2}{{c_{1}^{2}}b^{2}n}\right)\left( \frac{{c_{1}^{3}}b^{3}}{3}-\frac{{c_{1}^{2}}b^{2}}{2}+\frac{c_{1}b}{6}\right)\\ && +\left( \frac{3}{b^{2}}-\frac{2n}{b^{3}}+\frac{2}{b^{2}n}-\frac{2}{b^{3}}-\frac{1}{n^{2}}\right)\left( \frac{b^{2}}{2}-\frac{b}{2}\right)\\ && +\left( \frac{n}{{c_{1}^{2}}b^{2}}+\frac{3}{n}-\frac{3}{c_{1}b}+\frac{1}{{c_{1}^{2}}b^{2}}+\frac{1}{n^{2}}-\frac{2}{c_{1}bn}-\frac{c_{1}b}{n^{2}}\right)\left( c_{1}b-1\right)\\ &=&\frac{n}{{c_{1}^{4}}b^{4}}\cdot\frac{{c_{1}^{3}}b^{3}}{3}-\frac{2n}{{c_{1}^{3}}b^{3}}\cdot\frac{{c_{1}^{2}}b^{2}}{2}+\frac{n}{{c_{1}^{2}}b^{2}}\cdot c_{1}b\\ &=&\frac{1}{3}\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right). \end{array} $$

(16)

Plug Eqs. 16 in 15

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{2}]&={\Sigma}_{ij}^{2}\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}+({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\left[\frac{1}{3}\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right)\right]. \end{array} $$

(17)

Similarly as E[a₂], we calculate E[a₃] by first calculating

$$ \mathrm{E}\left[ \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)} \right) \right], $$

for s = c₁b,..., (n − c₁b). We will show that

$$ \left[ \begin{array}{l} \bar{C}_{l}(b)-\bar{C} \\ \bar{C}_{l+s}(b)-\bar{C} \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}\left( \frac{n-b}{bn}\right){\Sigma}& -\frac{1}{n}{\Sigma}\\ -\frac{1}{n}{\Sigma}& \left( \frac{n-b}{bn}\right){\Sigma} \end{array}\right]\right). $$

(18)

Continuing as in Eq. 12

$$ \text{Cov} \left( \bar{C}_{l}(c_{1}b)-\bar{C},\ \bar{C}_{l+s}(c_{1}b)-\bar{C} \right)=L\cdot\left( -\frac{1}{n}\right)I_{p}\cdot L^{T}=-\frac{1}{n}\cdot{\Sigma}. $$

(19)

The joint distribution in Eq. 18 follows from Eqs. 19 and 9. Denote $Z_{1}=\bar {C}_{l}(c_{1}b)^{(i)}-\bar {C}^{(i)}$, $Z_{2}=\bar {C}_{l}(c_{1}b)^{(j)}-\bar {C}^{(j)}$, $Z_{3}=\bar {C}_{l+s}(c_{1}b)^{(i)}-\bar {C}^{(i)}$, $Z_{4}=\bar {C}_{l+s}(c_{1}b)^{(j)}-\bar {C}^{(j)}$. By Proposition 2,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{3}]&=\sum\limits_{s=b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s} \mathrm{E}[Z_{1}Z_{2}Z_{3}Z_{4}] = \sum\limits_{s=b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}{\Sigma}_{ij}^{2} + \frac{{\Sigma}_{ii}{\Sigma}_{jj}}{n^{2}} + \frac{{\Sigma}_{ij}^{2}}{n^{2}}\right]. \end{array} $$

(20)

Notice

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{s=c_{1}b}^{n-c_{1}b}\sum\limits_{l=1}^{n-c_{1}b+1-s}\frac{1}{n^{2}} \\ &=&\frac{1}{n^{2}}\cdot\sum\limits_{s=c_{1}b}^{n-c_{1}b}(n-c_{1}b-s+1)\\ &=&-\frac{1}{n^{2}}\left( \frac{n^{2}}{2}-c_{1}bn+\frac{n}{2}\right)+\left( \frac{1}{n}-\frac{c_{1}b}{n^{2}}+\frac{1}{n^{2}}\right)(n-2c_{1}b+1)\\ &=&o\left( \frac{n}{b}\right). \end{array} $$

(21)

Plug Eqs. 21 in 20,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{3}]&={\Sigma}_{ij}^{2}\sum\limits_{s=b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}+o\left( \frac{n}{b}\right). \end{array} $$

(22)

Plug Eqs. 11, 17 and 22 in 8,

$$ \begin{array}{@{}rcl@{}} A_{2}& =& \frac{{c_{1}^{2}}b^{2}}{n^{2}}\cdot \left( \mathrm{E} [a_{1}] + 2\mathrm{E} [a_{2}] + 2\mathrm{E} [a_{3}] \right)\\ & = & \frac{c_{1}b^{2}}{n^{2}}\cdot\Bigg[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right)\\ && + {\Sigma}_{ij}^{2}\left( \sum\limits_{l=0}^{n-c_{1}b}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}+2\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}+2\sum\limits_{s=c_{1}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}\right)\Bigg]\\ &=&\frac{{c_{1}^{2}}b^{2}}{n^{2}}\cdot\left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right)+{\Sigma}_{ij}^{2}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}(n-c_{1}b+1)^{2}\right]\\ &= & \frac{{c_{1}^{2}}b^{2}}{n^{2}}\cdot\left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{n}{c_{1}b}+o\left( \frac{n}{c_{1}b}\right)+{\Sigma}_{ij}^{2}\left( \frac{n^{2}}{{c_{1}^{2}}b^{2}}-\frac{4n}{c_{1}b}\right)\right]\\ &=&\left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{c_{1}b}{n}+{\Sigma}_{ij}^{2}-4{\Sigma}_{ij}^{2}\cdot\frac{c_{1}b}{n}\right]+o\left( \frac{b}{n}\right). \end{array} $$

That proves the first part of the lemma. We now move on to term A₃. Let

$$ OL^{(i)} = \left( \bar{C}_{p}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{q}^{(i)}(c_{2}b)-\bar{C}^{(i)} \right) \text{ and } OL^{(j)} = \left( \bar{C}_{p}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}_{q}^{(j)}(c_{2}b)-\bar{C}^{(j)} \right), $$

for p, q satisfying q ≥ p and q + c₁b ≤ p + c₂b. Also, define

$$ a_{4} = \sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}(\bar{C}_{l}(c_{1}b)^{(i)}-\bar{C}^{(i)})(\bar{C}_{l}(c_{1}b)^{(j)}-\bar{C}^{(j)})(\bar{C}^{(i)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C}^{(i)})(\bar{C}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C}^{(j)}), $$

$$ a_{5} = \sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}(\bar{C}_{l}(c_{1}b)^{(i)}-\bar{C}^{(i)})(\bar{C}_{l}(c_{1}b)^{(j)}-\bar{C}^{(j)})(\bar{C}^{(i)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C}^{(i)})(\bar{C}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C}^{(j)}). $$

We have

$$ \begin{array}{@{}rcl@{}} A_{3}&& \\ &=&- \frac{c_{1}c_{2}b^{2}}{n^{2}}\left[\sum\limits_{l=0}^{n-c_{1}b} \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right)\right] \left[\sum\limits_{l=0}^{n-c_{2}b} \left( \bar{C}_{l}^{(i)}(c_{2}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{2}b)-\bar{C}^{(j)} \right) \right]\\ &=& - \frac{c_{1}c_{2}b^{2}}{n^{2}}\cdot\mathrm{E}[((c_{1}-c_{2})b+1)(n-b+1)\cdot OL^{(i)}OL^{(j)} + 2a_{4} + 2a_{5}] \end{array} $$

(23)

First consider E[OL⁽ⁱ⁾OL^(j)] in Eq. 23. We will show that

$$ \left[ \begin{array}{l} \bar{C}_{p}(c_{1}b)-\bar{C}_{n} \\ \bar{C}_{q}(c_{2}b)-\bar{C}_{n} \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}& \left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}\\ \left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}& \left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma} \end{array}\right]\right). $$

(24)

For i≠j,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{p}(c_{1}b) - \bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{q}(c_{2}b) - \bar{B}^{(j)} \right] = \mathrm{E} \left[\bar{B}^{(i)}_{p}(c_{1}b)-\bar{B}^{(i)} \right] \mathrm{E} \left[\bar{B}^{(j)}_{q}(c_{2}b)-\bar{B}^{(j)}\right] = 0. $$

(25)

For i = j and p, q satisfying q ≥ p and q + c₂b ≤ p + c₁b, following steps similar to Eq. 13,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{p}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(i)}_{q}(c_{2}b)-\bar{B}^{(i)} \right] = \frac{n - c_{1}b}{nc_{1}b}. $$

(26)

By Eqs. 25 and 26

$$ \text{Cov} \left( \bar{C}_{p}(c_{1}b)-\bar{C}_{n},\ \bar{C}_{q}(c_{2}b)-\bar{C}_{n} \right)= \frac{n-c_{1}b}{c_{1}bn}\cdot{\Sigma}. $$

(27)

Equation 27 yields the joint distribution in Eq. 24. Denote $Z_{1}=\bar {C}_{p}(c_{1}b)^{(i)}-\bar {C}^{(i)}$, $Z_{2}=\bar {C}_{p}(c_{1}b)^{(j)}-\bar {C}^{(j)}$, $Z_{3}=\bar {C}_{q}(c_{2}b)^{(i)}-\bar {C}^{(i)}$, $Z_{4}=\bar {C}_{q}(c_{2}b)^{(j)}-\bar {C}^{(j)}$. Then

$$ \begin{array}{@{}rcl@{}} &&\mathrm{E} \left[((c_{1}-c_{2})b+1)(n-c_{1}b+1)OL^{(i)}OL^{(j)} \right] \\ & =& ((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \mathrm{E}[Z_{1}Z_{2}Z_{3}Z_{4}] \\ &=&((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}({\Sigma}_{ij}^{2}+{\Sigma}_{ii}{\Sigma}_{jj})\\ && +((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma}_{ij}^{2}. \end{array} $$

(28)

Simplifying further,

$$ \begin{array}{@{}rcl@{}} ((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2} &=(c_{1}-c_{2})\frac{n}{{c_{1}^{2}}b}+o\left( \frac{n}{b}\right) , \text{ and} \end{array} $$

(29)

and

$$ \begin{array}{@{}rcl@{}} ((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right) &=\frac{c_{1}-c_{2}}{c_{1}c_{2}}\frac{n}{b}+o\left( \frac{n}{b}\right). \end{array} $$

(30)

Plug Eqs. 29 and 30 in 28,

$$ \begin{array}{@{}rcl@{}} &&\mathrm{E} \left[((c_{1}-c_{2})b+1)(n-c_{1}b+1)OL^{(i)}OL^{(j)} \right]\\ &=&\left[\frac{(c_{1}-c_{2})}{{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})+\frac{c_{1}-c_{2}}{c_{1}c_{2}}{\Sigma}_{ij}^{2}\right]\cdot\frac{n}{b}+o\left( \frac{n}{b}\right). \end{array} $$

(31)

We calculate E[a₄] by showing that

$$ \left[ \begin{array}{l} \bar{C}_{l}(c_{1}b) - \bar{C} \\ \bar{C}_{l+(c_{1}-c_{2})b+s}(c_{2}b) - \bar{C} \end{array} \right] \!\sim\! N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}& \left( \frac{1}{c_{1}b}-\frac{1}{n}-\frac{s}{{c_{2}^{2}}b^{2}}\right){\Sigma}\\ \left( \frac{1}{c_{1}b}-\frac{1}{n} - \frac{s}{c_{2}b^{2}}\right){\Sigma}& \qquad\left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma} \end{array}\right]\right). $$

(32)

All we need to obtain is the covariance matrix. Continuing as before in Eq. 12, for i≠j,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{B}^{(j)} \right] =0. $$

(33)

For i = j, we need to calculate $\mathrm {E}[\bar {B}^{(i)}_{l}(c_{1}b)-\bar {B}^{(i)}][\bar {B}^{(i)}_{l+(c_{1}-c_{2})b+s}(c_{1}b)-\bar {B}^{(i)}]$ for s = 1,..., (c₂b − 1). Continuing as before in Eq. 13,

$$ \begin{array}{@{}rcl@{}} &\mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(i)}_{l+(c_{1}-c_{2})b+s}(c_{1}b)-\bar{B}^{(i)} \right]= \frac{1}{c_{1}b} - \frac{1}{n} - \frac{s}{c_{1}c_{2}b^{2}}. \end{array} $$

(34)

By Eqs. 33 and 34,

$$ \text{Cov} \left( \bar{C}_{l}(b)-\bar{C},\ \bar{C}_{l+s}(b)-\bar{C} \right) = \left( \frac{1}{c_{1}b} - \frac{1}{n} - \frac{s}{c_{1}c_{2}b^{2}}\right)\cdot{\Sigma}. $$

(35)

Therefore (32) follows from Eqs. 9 and 35. Again denote $Z_{1}=\bar {C}_{l}^{(i)}(c_{1}b)-\bar {C}^{(i)}$, $Z_{2}=\bar {C}_{l}^{(j)}(c_{1}b)-\bar {C}^{(j)}$, $Z_{3}=\bar {C}_{l+(c_{1}-c_{2})b+s}^{(i)}(c_{2}b)-\bar {C}^{(i)}$, $Z_{4}=\bar {C}_{l+(c_{1}-c_{2})b+s}^{(j)}(c_{2}b)-\bar {C}^{(j)}$,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{4}]\!\!\!&&\\ &=&\sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}E[Z_{1}Z_{2}Z_{3}Z_{4}]\\ &=&\sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma}_{ij}^{2}+\left( \frac{1}{c_{1}b} - \frac{1}{n} - \frac{s}{c_{1}c_{2}b^{2}}\right)^{2}{\Sigma}_{ii}{\Sigma}_{jj}+\left( \frac{1}{c_{1}b} - \frac{1}{n} - \frac{s}{c_{1}c_{2}b^{2}}\right)^{2}{\Sigma}_{ij}^{2}\right]. \end{array} $$

(36)

It is easy to show that

$$ \begin{array}{@{}rcl@{}} &\sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right) =\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right). \end{array} $$

(37)

Following the steps in Eq. 16, we get

$$ \begin{array}{@{}rcl@{}} \sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{1}{c_{1}b}-\frac{1}{n}-\frac{s}{c_{1}c_{2}b^{2}}\right)^{2} & =\frac{c_{2}}{{c_{1}^{2}}3}\cdot\frac{n}{b}+o\left( \frac{n}{b}\right). \end{array} $$

(38)

Plug Eqs. 37 and 38 in 36

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{4}]&=\frac{c_{2}}{3{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\frac{n}{b}+{\Sigma}_{ij}^{2}\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right). \end{array} $$

(39)

Finally, we calculate E[a₅]. For s = c₂b,..., (n − c₁b), we will show that

$$ \left[ \begin{array}{l} \bar{C}_{l}(c_{1}b)-\bar{C} \\ \bar{C}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C} \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}& -\frac{1}{n}{\Sigma}\\ -\frac{1}{n}{\Sigma}& \left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma} \end{array}\right]\right). $$

(40)

For i≠j,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{B}^{(j)} \right] = 0. $$

(41)

For i = j, we need to calculate $\mathrm {E}[\bar {B}^{(i)}_{l}(b)-\bar {B}^{(i)}][\bar {B}^{(i)}_{l+(1-c)b+s}(cb)-\bar {B}^{(i)}]$ for s = cb,..., (n − b).. Similar to the steps in Eq. 13, we get

$$ \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{B}^{(j)} \right] = -\frac{1}{n} . $$

(42)

By Eqs. 41 and 42,

$$ \text{Cov} \left( \bar{C}_{l}(c_{1}b)-\bar{C},\ \bar{C}_{l+(c_{1} - c_{2})b + s}(c_{2}b)-\bar{C} \right)= -\frac{1}{n}\cdot{\Sigma}. $$

(43)

Therefore (40) follows from Eqs. 9 and 43. Again denote $Z_{1}=\bar {C}_{l}^{(i)}(c_{1}b)-\bar {C}^{(i)}$, $Z_{2}=\bar {C}_{l}^{(j)}(c_{1}b)-\bar {C}^{(j)}$, $Z_{3}=\bar {C}_{l+(c_{1}-c_{2})b+s}^{(i)}(c_{2}b)-\bar {C}^{(i)}$, $Z_{4}=\bar {C}_{l+(c_{1}-c_{2})b+s}^{(j)}(c_{2}b)-\bar {C}^{(j)}$

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{5}]\!\!\!&&\\ &=&\sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s} \mathrm{E} \left[(\bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)})(\bar{C}_{l+(c_{1}-c_{2})b+s}^{(i)}(c_{2}b)-\bar{C}^{(i)})(\bar{C}_{l+(c_{1}-c_{2})b+s}^{(j)}(c_{2}b)-\bar{C}^{(j)}) \right]\\ &=&\sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma}_{ij}^{2}+\frac{1}{n^{2}}{\Sigma}_{ii}{\Sigma}_{jj}+\frac{1}{n^{2}}{\Sigma}_{ij}^{2}\right]. \end{array} $$

(44)

Straightforward algebra shows,

$$ \begin{array}{@{}rcl@{}} \sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right) &=\frac{1}{2c_{1}c_{2}}\frac{n^{2}}{b^{2}}-\left( \frac{3}{2c_{1}}+\frac{3}{2c_{2}}\right)\frac{n}{b}+o\left( \frac{n}{b}\right), \end{array} $$

(45)

and

$$ \begin{array}{@{}rcl@{}} \sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\frac{1}{n^{2}} &=o\left( \frac{n}{b}\right). \end{array} $$

(46)

Plug Eqs. 45 and 46 in 44,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{5}]&={\Sigma}_{ij}^{2} \left( \frac{1}{2c_{1}c_{2}}\frac{n^{2}}{b^{2}}-\left( \frac{3}{2c_{1}}+\frac{3}{2c_{2}}\right)\frac{n}{b}\right). \end{array} $$

(47)

Replace Eqs. 31, 39 and 47 in 23

$$ \begin{array}{@{}rcl@{}} A_{3}& = & \mathrm{E}\left[-\frac{c_{1}c_{2}b^{2}}{n^{2}}\left[\sum\limits_{l=0}^{n-b}(\bar{C}_{l}^{(i)}(b)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(b)-\bar{C}^{(j)})\right]\left[\sum\limits_{l=0}^{n-cb}(\bar{C}_{l}^{(i)}(cb)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(cb)-\bar{C}^{(j)})\right]\right]\\ &=&- \frac{c_{1}c_{2}b^{2}}{n^{2}}\cdot \mathrm{E} \left[((c_{1}-c_{2})b+1)(n-b+1)\cdot OL^{(i)}OL^{(j)} +2a_{4} + 2a_{5} \right] \\ &=&-c_{1}c_{2}\frac{b}{n}\cdot \left[\frac{(c_{1}-c_{2})}{{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})+\frac{c_{1}-c_{2}}{c_{1}c_{2}}{\Sigma}_{ij}^{2}\right.\\ &&+ \frac{2c_{2}}{3{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})+{\Sigma}_{ij}^{2}\frac{2}{c_{1}} + \left. {\Sigma}_{ij}^{2} \left( \frac{1}{c_{1}c_{2}}\frac{n}{b}-\left( \frac{3}{c_{1}}+\frac{3}{c_{2}}\right)\right) \right]+o\left( \frac{b}{n}\right)\\ &=&-c_{1}c_{2} \frac{b}{n} \left[\frac{3c_{1} - c_{2}}{3{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot - 2\left( \frac{c_{1}+c_{2}}{c_{1}c_{2}}\right)\cdot{\Sigma}_{ij}^{2} +\frac{1}{c_{1}c_{2}} \frac{n}{b}\cdot{\Sigma}_{ij}^{2} \right]+o\left( \frac{b}{n}\right) \\ &=& \frac{(c_{2} - 3c_{1})c_{2}}{3c_{1}}({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})\cdot \frac{b}{n} + 2\left( c_{1}+c_{2}\right)\cdot{\Sigma}_{ij}^{2}\cdot\frac{b}{n} - {\Sigma}_{ij}^{2} +o\left( \frac{b}{n}\right). \end{array} $$

□

With elements $\tilde {\Sigma }_{wL, ij}$, define

$$ \tilde{\Sigma}_{wL}=\frac{1}{n}\sum\limits_{k=1}^{b}\sum\limits_{l=0}^{n-k}k^{2}{\Delta}_{2}w_{n}(k) \left[\bar{C}_{l}(k)-\bar{C}\right]\left[\bar{C}_{l}(k)-\bar{C}\right]^{T}. $$

Lemma 3

If Assumption 1 holds and ${\sum }_{k=1}^{b}({\Delta }_{2}w_{k})^{2}\leq O\left (b^{-2}\right )$ then

$$ Var[\tilde{\Sigma}_{wL,ij}]=({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\bigg[\frac{2}{3}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}k^{3}\cdot\frac{1}{n}+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{\Delta}_{2}w_{u}{\Delta}_{2}w_{t+u}\left( \frac{2}{3}u^{3}+u^{2}t\right)\cdot\frac{1}{n}\bigg]+o\left( \frac{b}{n}\right). $$

Proof

Note $({\Delta }_{2}w_{k})^{2}\leq {\sum }_{k=1}^{b}({\Delta }_{2}w_{k})^{2} = O\left (b^{-2}\right ),$ hence a_k = b ⋅Δ₂w_k = O(1). Consider

$$ \tilde{\Sigma}_{wL,ij} = \frac{1}{n}\sum\limits_{k=1}^{b}\sum\limits_{l=0}^{n-k}k^{2}{\Delta}_{2}w_{n}(k) \left[\bar{C}_{l}^{(i)}(k)-\bar{C}^{(i)} \right] \left[\bar{C}^{(j)}_{l}(k)-\bar{C}^{(j)} \right]. $$

Let c_k = k/b for k = 1,...,b, also denote a_k = b ⋅Δ₂w_k for simplicity. Then,

$$ \begin{array}{@{}rcl@{}} \tilde{\Sigma}_{wL,ij} &=\sum\limits_{k=1}^{b}c_{k}a_{k}\left( \frac{c_{k}b}{n}\sum\limits_{l=0}^{n-c_{k}b} \left[\bar{C}_{l}^{(i)}(k)-\bar{C}^{(i)} \right] \left[\bar{C}^{(j)}_{l}(k)-\bar{C}^{(j)} \right]\right). \end{array} $$

Define $A_{1,ij}^{(k)}$ and $A_{2,ij}^{(ut)}$ below and apply Lemma 2,

$$ \begin{array}{@{}rcl@{}} A_{1,ij}^{(k)} & =& \mathrm{E}\left[\frac{(c_{k}b)^{2}}{n^{2}}\cdot\left( \sum\limits_{k=0}^{n-c_{k}b}(\bar{C}_{l}^{(i)}(c_{k}b)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{k}b)-\bar{C}^{(j)})\right)^{2}\right]\\ &=&\left( \frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-4{\Sigma}_{ij}^{2}\right)\cdot\frac{c_{k}b}{n}+{\Sigma}_{ij}^{2}+o\left( \frac{b}{n}\right), \end{array} $$

(48)

and

$$ \begin{array}{@{}rcl@{}} A_{2,ij}^{(ut)}& = & \mathrm{E}\left[\frac{(c_{u+t}b)^{2}}{n^{2}}\cdot\left( \sum\limits_{p=0}^{n-c_{t}b}(\bar{C}_{p}^{(i)}(c_{t}b)-\bar{C}^{(i)})(\bar{C}_{p}^{(j)}(c_{t}b)-\bar{C}^{(j)})\right)\right.\\ && \left.\cdot\left( \sum\limits_{q=0}^{n-c_{u+t}b}(\bar{C}^{(i)}_{q}(c_{t+u}b)-\bar{C}^{(i)})(\bar{C}^{(j)}_{q}(c_{t+u}b)-\bar{C}^{(j)})\right)\right]\\ &=&\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\frac{b}{n}+\frac{c_{u+t}}{c_{u}}{\Sigma}_{ij}^{2}+o\left( \frac{b}{n}\right). \end{array} $$

(49)

To calculate $\text {Var}[\tilde {\Sigma }_{wL,ij}]$, we will calculate $\mathrm {E}[\tilde {\Sigma }_{wL,ij}^{2}]$ and $(\mathrm {E}[\tilde {\Sigma }_{wL,ij}])^{2}$. Plugging Eqs. 48 and 49 in the expression of $\mathrm {E}[\tilde {\Sigma }_{wL,ij}^{2}]$ results in

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[\tilde{\Sigma}_{wL,ij}^{2}]\!\!&&\\ & =& \mathrm{E}\left[\sum\limits_{k=1}^{b}\left( c_{k}a_{k}\cdot\left[\frac{c_{k}b}{n}\sum\limits_{l=0}^{n-c_{k}b}(\bar{C}_{l}^{(i)}(k)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(k)-\bar{C}^{(j)})\right]\right)^{2}\right.\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}c_{t+u}^{2}a_{t+u}\cdot\frac{b^{2}}{n^{2}}\left( \sum\limits_{p=0}^{n-c_{t}b}(\bar{C}_{p}^{(i)}(c_{t}b)-\bar{C}^{(i)})(\bar{C}_{p}^{(j)}(c_{t}b)-\bar{C}^{(j)})\right)\\ && \left.\cdot\left( \sum\limits_{q=0}^{n-c_{t+u}b}(\bar{C}_{q}^{(i)}(c_{t+u}b)-\bar{C}^{(i)})(\bar{C}_{q}^{(j)}(c_{t+u}b)-\bar{C}^{(j)})\right)\right]\\ &=&\sum\limits_{k=1}^{b}{c_{k}^{2}}{a_{k}^{2}}\cdot E\left[\frac{(c_{k}b)^{2}}{n^{2}}\cdot\left( \sum\limits_{l=0}^{n-c_{k}b}(\bar{C}_{l}^{(i)}(k)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(k)-\bar{C}^{(j)})\right)^{2}\right]\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{t+u}\cdot E\left[\frac{(c_{u+t}b)^{2}}{n^{2}}\cdot\left( \sum\limits_{p=0}^{n-c_{t}b}(\bar{C}_{p}^{(i)}(c_{t}b)-\bar{C}^{(i)})(\bar{C}_{p}^{(j)}(c_{t}b)-\bar{C}^{(j)})\right)\right.\\ && \cdot\left.\left( \sum\limits_{q=0}^{n-c_{t+u}b}(\bar{C}_{q}^{(i)}(c_{t+u}b)-\bar{C}^{(i)})(\bar{C}_{q}^{(j)}(c_{t+u}b)-\bar{C}^{(j)})\right)\right]\\ &=&\sum\limits_{k=1}^{b}{c_{k}^{2}}{a_{k}^{2}}A_{1,ij}^{(k)}+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}A_{2,ij}^{(ut)}\\ &=&o\left( \frac{b}{n}\right)+\sum\limits_{k=1}^{b}{c_{k}^{2}}{a_{k}^{2}}\left[\left( \frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-4{\Sigma}_{ij}^{2}\right)\cdot\frac{c_{k}b}{n}+{\Sigma}_{ij}^{2}\right]\\ &&+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}\left[\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\cdot\frac{b}{n}+\frac{c_{u+t}}{c_{u}}{\Sigma}_{ij}^{2}\right]\\ &=&\left[\sum\limits_{k=1}^{b}{c_{k}^{2}}{a_{k}^{2}}\cdot{\Sigma}_{ij}^{2}+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}\frac{c_{u+t}}{c_{u}}\cdot{\Sigma}_{ij}^{2}\right]+\left[\sum\limits_{k=1}^{b}{c_{k}^{3}}{a_{k}^{2}}\left( \frac{2}{3}[{\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2}]-4{\Sigma}_{ij}^{2}\right)\cdot\frac{b}{n}\right.\\ && +\left.2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\frac{b}{n}\right]+o\left( \frac{b}{n}\right)\\ &=&\left( \sum\limits_{k=1}^{b}a_{k}c_{k} \right)^{2}{\Sigma}_{ij}^{2}+\sum\limits_{k=1}^{b}{c_{k}^{3}}{a_{k}^{2}}\left( \frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-4{\Sigma}_{ij}^{2}\right)\cdot\frac{b}{n}\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\cdot\frac{b}{n}+o\left( \frac{b}{n}\right). \end{array} $$

(50)

By Eq. 9,

$$ \mathrm{E}[(C_{l}^{(i)}(c_{k}b)-\bar{C}^{(i)})(C_{l}^{(j)}(c_{k}b)-\bar{C}^{(j)})]=\frac{n-c_{k}b}{c_{k}bn}{\Sigma}_{ij}. $$

(51)

Using Eq. 51,

$$ \begin{array}{@{}rcl@{}} (\mathrm{E}[\tilde{\Sigma}_{wL,ij}])^{2}) \!\!&&\\ &=&\left( \frac{1}{n}\sum\limits_{k=1}^{b}\sum\limits_{l=0}^{n-k}k^{2}{\Delta}_{2}w_{k} \mathrm{E} \left[(C_{l}^{(i)}(c_{k}b)-\bar{C}^{(i)})(C_{l}^{(j)}(c_{k}b)-\bar{C}^{(j)}) \right]\right)^{2}\\ &=&\left( \sum\limits_{k=1}^{b}c_{k}a_{k}\left[\frac{c_{k}b}{n}\sum\limits_{l=0}^{n-c_{k}b} \mathrm{E} \left[(C_{l}^{(i)}(c_{k}b)-\bar{C}^{(i)})(C_{l}^{(j)}(c_{k}b)-\bar{C}^{(j)}) \right]\right]\right)^{2}\\ &=&\left( \sum\limits_{k=1}^{b}c_{k}a_{k}\left[\frac{c_{k}b}{n}\cdot(n-c_{k}b+1)\cdot\frac{n-c_{k}b}{c_{k}bn}\cdot{\Sigma}_{ij}\right]\right)^{2}\\ &=&{\Sigma}_{ij}^{2}\left[ \left( \sum\limits_{k=1}^{b}a_{k}c_{k} \right)^{2}-\sum\limits_{k=1}^{b}4{a_{k}^{2}}{c_{k}^{3}}\cdot\frac{b}{n}-2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}a_{u}a_{u+t}(2{c_{u}^{2}}c_{u+t}+2c_{u}c_{u+t}^{2})\cdot\frac{b}{n}\right]+o\left( \frac{b}{n}\right). \end{array} $$

(52)

Combine Eqs. 50 and 52,

$$ \begin{array}{@{}rcl@{}} \text{Var}[\tilde{\Sigma}_{wL,ij}] &=& \mathrm{E}[\tilde{\Sigma}_{wL,ij}^{2}] - (\mathrm{E}[\tilde{\Sigma}_{wL,ij}])^{2}\\ &=&\sum\limits_{k=1}^{b}{c_{k}^{3}}{a_{k}^{2}}\Bigg(\left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-4{\Sigma}_{ij}^{2}\right]+4{\Sigma}_{ij}^{2}\Bigg)\cdot\frac{b}{n}\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}\Bigg({c_{u}^{2}}a_{u}a_{u+t}\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\\ && +a_{u}a_{u+t}(2{c_{u}^{2}}c_{u+t}+2c_{u}c_{u+t}^{2}){\Sigma}_{ij}^{2}\Bigg)\cdot\frac{b}{n}+o\left( \frac{b}{n}\right)\\ &=&\sum\limits_{k=1}^{b}\frac{2}{3} {c_{k}^{3}}{a_{k}^{2}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{b}{n}+2\sum\limits_{t=1}^{b}\sum\limits_{u=1}^{b-t}\left( {c_{u}^{2}}c_{u+t}-\frac{1}{3}{c_{u}^{3}}\right)a_{u}a_{u+t}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{b}{n}+o\left( \frac{b}{n}\right)\\ &=&\sum\limits_{k=1}^{b}\frac{2}{3}\left( \frac{k}{b}\right)^{3}(b{\Delta}_{2}w_{k})^{2}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{b}{n}\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}\left[\left( \left( \frac{u}{b}\right)^{2}\frac{u+t}{b}-\frac{1}{3}\left( \frac{u}{b}\right)^{3}\right)b{\Delta}_{2}w_{u}\cdot b{\Delta}_{2}w_{u+t}\right]({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{b}{n}+o\left( \frac{b}{n}\right)\\ &=&({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\bigg[\frac{2}{3}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}k^{3}\cdot\frac{1}{n} + 2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{\Delta}_{2}w_{u}\cdot{\Delta}_{2}w_{t+u}\!\cdot\!\left( \frac{2}{3}u^{3} + u^{2}t\right)\!\cdot\!\frac{1}{n}\bigg] + o\left( \frac{b}{n}\right). \end{array} $$

□

Lemma 4

(Lemma 14 Vats et al. 2018) Suppose (5) holds for f = g and Assumption 1 hold. If, as $n\to \infty$,

$$ b\psi(n)^{2} \log n\left( \sum\limits_{k=1}^{b}|{\Delta}_{2}w_{n}(k)|\right)^{2}\rightarrow 0 \quad \text{ and } \quad \psi(n)^{2}\sum\limits_{k=1}^{b}|{\Delta}_{2}w_{n}(k)|\rightarrow 0, $$

then $\hat {\Sigma }_{w}\rightarrow \tilde {\Sigma }_{wL}$ as $n \to \infty$ w.p. 1.

Lemma 5

Suppose (5) holds for f = g and f = g² (where the square is element-wise) such that $\mathrm {E}_{F}D^{4} < \infty$ and Assumption 1 holds. Further, suppose $\psi ^{2}(n)b^{-1}\log n \rightarrow 0$, then

$$ \mathrm{E} \left[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij} \right]^{2}\rightarrow\ 0\ \text{as}\ n\rightarrow\infty. $$

Proof

An observation of Lemma B.4 of Jones et al. (2006), Lemmas 12, 13 and 14 of Flegal and Jones (2010) and Lemma 5 of Liu and Flegal (2018) show that Lemma 5 hold. □

Appendix C: Proof of Theorem 3

Define

$$ \eta= \text{Var}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]+2\mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})(\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij})]. $$

We show $\eta \rightarrow 0$ as $n\rightarrow \infty$. From Lemma 5, Cauchy-Schwarz inequality, and Var[X] ≤EX².

$$ \begin{array}{@{}rcl@{}} |\eta|&=&|\text{Var}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}] + 2\mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})(\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij})]|\\ &&\leq\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]^{2}+2\sqrt{\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]^{2}\cdot\mathrm{E}[\tilde{\Sigma}_{w,L,ij} - \mathrm{E}\tilde{\Sigma}_{wL, ij}]^{2}}\\ & =& \mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{w,L,ij}]^{2} + 2(\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]^{2})^{1/2}\cdot (\text{Var}[\tilde{\Sigma}_{wL,ij}])^{1/2} \end{array} $$

By the conditions of Lemma 3,

$$ \frac{1}{n}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}k^{3}\leq\frac{b^{3}}{n}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}\leq O \left( \frac{b}{n} \right). $$

Hence we have

$$ \text{Var}[\tilde{\Sigma}_{wL,ij}]=(({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})S \frac{b}{n}+o(1))\cdot\frac{b}{n}. $$

By Lemma 5, $\mathrm {E}[\hat {\Sigma }_{w,ij}-\tilde {\Sigma }_{wL,ij}]^{2}=o(1)$, therefore

$$ \begin{array}{@{}rcl@{}} |\eta|&\leq&\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{,L,ij}]^{2}+2(\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]^{2})^{1/2}\cdot (\text{Var}[\tilde{\Sigma}_{wL,ij}])^{1/2}\\ &=&o(1)+2\sqrt{o(1)\cdot \left[(({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})S+o(1))\cdot\frac{b}{n} \right]} = o(1). \end{array} $$

(53)

Since $b/n\rightarrow 0$ as $n\rightarrow \infty$, plug in Eq. 53

$$ \begin{array}{@{}rcl@{}} \text{Var}[\hat{\Sigma}_{w,ij}]& =& \mathrm{E}[\hat{\Sigma}_{w,ij} - \mathrm{E}\hat{\Sigma}_{w,ij}]^{2}\\ & =& \mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})+(\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij}) - (\mathrm{E}\hat{\Sigma}_{w,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij})]^{2}\\ &&+ 2 \mathrm{E}[[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}) - \mathrm{E}(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})]\cdot [\tilde{\Sigma}_{w,ij} - \mathrm{E}\tilde{\Sigma}_{w,ij}]]\\ & =& \mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{w,L,ij}) - \mathrm{E}(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})]^{2} + \mathrm{E}[\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij}]^{2}\\ && + 2 \mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})\cdot (\tilde{\Sigma}_{w,ij} - \mathrm{E}\tilde{\Sigma}_{w,ij})]\\ & =& \mathrm{E}[\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij}]^{2}+\eta\\ &=&({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})S\cdot\frac{b}{n}+o\left( \frac{b}{n}\right)+o(1). \end{array} $$

Appendix D: Performance Without a Pilot Run

Table 3 Coverage probabilities for 90% confidence regions over 1000 replications with no pilot runs for Bayesian logistic regression example

Table 4 Coverage probabilities for 90% confidence regions over 1000 replications with no pilot runs for the Bayesian dynamic space-time example