Ensemble Kalman inversion: mean-field limit and convergence analysis

Abstract

Ensemble Kalman inversion (EKI) has been a very popular algorithm used in Bayesian inverse problems (Iglesias et al. in Inverse Probl 29: 045001, 2013). It samples particles from a prior distribution and introduces a motion to move the particles around in pseudo-time. As the pseudo-time goes to infinity, the method finds the minimizer of the objective function, and when the pseudo-time stops at 1, the ensemble distribution of the particles resembles, in some sense, the posterior distribution in the linear setting. The ideas trace back further to ensemble Kalman filter and the associated analysis  (Evensen in J Geophys Res: Oceans 99: 10143–10162, 1994; Reich in BIT Numer Math 51: 235–249, 2011), but to today, when viewed as a sampling method, why EKI works, and in what sense with what rate the method converges is still largely unknown. In this paper, we analyze the continuous version of EKI, a coupled SDE system, and prove the mean-field limit of this SDE system. In particular, we will show that 1. as the number of particles goes to infinity, the empirical measure of particles following SDE converges to the solution to a Fokker–Planck equation in Wasserstein 2-distance with an optimal rate, for both linear and weakly nonlinear case; 2. the solution to the Fokker–Planck equation reconstructs the target distribution in finite time in the linear case, as suggested in Iglesias et al. (Inverse Probl 29: 045001, 2013).

Appendices

Appendix A: Moments bound of summation of independent mean-zero random variables

In this section, we prove a lemma which is used in proof of Lemma 3.

Lemma 9

Assume $x_1,\cdots ,x_J$ are i.i.d random variables and satisfy (for $p\ge 2$)

$$\begin{aligned} {\mathbb {E}}x_i=0,\quad {\mathcal {L}}_p={\mathbb {E}}|x_i|^p<\infty \,. \end{aligned}$$

Then, we have

$$\begin{aligned} \left( {\mathbb {E}}\left| \sum ^J_{j=1}x_j\right| ^p\right) ^{1/p}\le CJ^{1/2}\,, \end{aligned}$$

where C is a constant only depends on ${\mathcal {L}}_p$ and p.

Proof

Without loss of generality, we assume p is an even number and $J>p/2$. Then ${\mathbb {E}}\left| \sum ^J_{j=1}x_j\right| ^p={\mathbb {E}}\left( \sum ^J_{j=1}x_j\right) ^{p}$.

Since $\{x_i\}$ are independent with zero mean, we have

$$\begin{aligned} {\mathbb {E}}\left( \sum ^J_{j=1}x_j\right) ^{p}=\sum _{j_1+j_2+\cdots +j_J=p}{\mathbb {E}}\left( x^{j_1}_1x^{j_2}_2\cdots x^{j_J}_J\right) \,, \end{aligned}$$

where $\{j_n\}$ should be nonnegative integers and not equal to 1 (otherwise ${\mathbb {E}}x_i=0$ provides a trivial contribution).

For each term in the summation, using generalization of Hölder’s inequality, we have

$$\begin{aligned} {\mathbb {E}}\left( x^{j_1}_1x^{j_2}_2\cdots x^{j_J}_J\right) \le \varPi ^J_{n=1}({\mathbb {E}}|x_{n}|^p)^{j_n/p}={\mathcal {L}}_p\,, \end{aligned}$$

which implies

$$\begin{aligned} {\mathbb {E}}\left( \sum ^J_{j=1}x_j\right) ^{p}\le {\mathcal {L}}_p\left( \sum _{j_1+j_2+\cdots +j_J=p}1\right) ={\mathcal {L}}_p|I_1| \end{aligned}$$

(61)

where

$$\begin{aligned} I_1=\left\{ \left( j_1,\cdots ,j_J\right) \Bigg |j_n\in {\mathbb {N}}\setminus \{1\},\ \sum ^J_{n=1}j_n=p\right\} \end{aligned}$$

and $|I_1|$ denotes the cardinality of the set $I_1$.

In $I_1$, if $j_n$ does not equal to zero, then $j_n$ is at least 2, meaning there are at most p/2 nontrivial elements in the vector. Therefore, we have the following inequality

$$\begin{aligned} |I_1|\le P(J,{p/2})|I_2|\le J^{p/2}|I_{2}|\le C(p)J^{p/2}\,. \end{aligned}$$

(62)

Here P(J, p/2) denotes the number of p/2-permutations in J and is thus smaller than $J^{p/2}$, and $I_{2}$ is a new set defined by:

$$\begin{aligned} I_{2} = \left\{ \left( i_1,\cdots ,i_{p/2}\right) \Bigg |i_n\in {\mathbb {N}}^+\setminus \{1\},\ \sum ^{p/2}_{n=1}i_n=p\right\} \,. \end{aligned}$$

Its cardinality does not have J dependence, and thus, we bound it by C(p), a constant depending on p only. $\square$

Appendix B: Bound of high moments of $\{u^j\}$

Proof

For convenience, we omit the subscript $'t'$ in $u,\mathbf{u },e,\mathbf{e }$, etc. First, we prove the boundedness of ${\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{j}|e^j|^2\right] ^p$, which we will use later.

$$\begin{aligned} \begin{aligned} {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{j}|e^j|^2\right] ^{p}&\le {\mathbb {E}}\left[ \sum ^{K}_{m=1}\frac{1}{J}\sum ^J_{j}|e^j_m|^2\right] ^{p}\\&\le C_{p}{\mathbb {E}}\left( \sum ^{K}_{m=1}\left[ \frac{1}{J}\sum ^J_{j}|e^j_m|^2\right] ^{p}\right) \\&\le C_{p}V_{2p}(e)\le C\,, \end{aligned} \end{aligned}$$

(63)

which also implies

$$\begin{aligned} {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{j}|\mathbf{e }^j|^2\right] ^{p}\le C{\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{j}|e^j|^2\right] ^{p}\le C\,. \end{aligned}$$

(64)

Then, we first estimate ${\mathbb {E}}|\mathbf{u }^j|^{2p}$. Using Ito’s formula, for fix $1\le j\le J$ and $p\ge 1$, we obtain

$$\begin{aligned} \begin{aligned} d|\mathbf{u }^j|^{2p}=&-2p\left( |\mathbf{u }^j|^{2(p-1)}\left\langle \mathbf{u }^j,\mathrm {Cov}_{\mathbf{u }}\mathbf{u }^j\right\rangle \right) \mathrm{d}t+\mathrm {\mathbf{R }}\ \mathrm{d}W^j_t\\&+p\left( |\mathbf{u }^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle \mathbf{e }^i,\mathbf{e }^k\right\rangle ^2\right] \right) \mathrm{d}t\\&+\frac{2p(p-1)}{J^2}\left( \left| \mathbf{u }^j\right| ^{2(p-2)}\sum ^J_{i,k=1}\left\langle \mathbf{u }^j,\mathbf{e }^i\right\rangle \left\langle \mathbf{u }^j,\mathbf{e }^k\right\rangle \left\langle \mathbf{e }^i,\mathbf{e }^k\right\rangle \right) \mathrm{d}t\\&+2p\left( |\mathbf{u }^j|^{2(p-1)}\left\langle \mathbf{u }^j,\mathrm {Cov}_{\mathbf{u },\mathbf{r }}\varGamma ^{-\frac{1}{2}}\left( r-\mathrm {m}(u)\right) \right\rangle \right) \mathrm{d}t\\&+p\left( |\mathbf{u }^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle \mathbf{e }^i,\mathbf{e }^k\right\rangle \left\langle \mathbf{r }^i,\mathbf{r }^k\right\rangle \right] \right) \mathrm{d}t\\&+\frac{2p(p-1)}{J^2}\left( \left| \mathbf{u }^j\right| ^{2(p-2)}\sum ^J_{i,k=1}\left\langle \mathbf{u }^j,\mathbf{e }^i\right\rangle \left\langle \mathbf{u }^j,\mathbf{e }^k\right\rangle \left\langle \mathbf{r }^i,\mathbf{r }^k\right\rangle \right) \mathrm{d}t\,,\\ \end{aligned} \end{aligned}$$

(65)

where $\mathrm {\mathbf{R }}$ is the coefficient before Brownian motion. The first term is negative. To complete the computation, we need to provide the bound for the rest. The second term is bounded by:

$$\begin{aligned} \begin{aligned}&{\mathbb {E}}\left( |\mathbf{u }^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle \mathbf{e }^i,\mathbf{e }^k\right\rangle ^2\right] \right) \\&\quad \le {\mathbb {E}}\left( |\mathbf{u }^j|^{2(p-1)}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] ^2\right) \\&\quad \le \left( {\mathbb {E}}|\mathbf{u }^j|^{2p}\right) ^{(p-1)/p}\left( {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] ^{2p}\right) ^{1/p}\,. \end{aligned} \end{aligned}$$

The third term is bounded by:

$$\begin{aligned} \begin{aligned}&\frac{1}{J^2}{\mathbb {E}}\left( \left| \mathbf{u }^j\right| ^{2(p-2)}\sum ^J_{i,k=1}\left\langle \mathbf{u }^j,\mathbf{e }^i\right\rangle \left\langle \mathbf{u }^j,\mathbf{e }^k\right\rangle \left\langle \mathbf{e }^i,\mathbf{e }^k\right\rangle \right) \\&\quad \le {\mathbb {E}}\left( |\mathbf{u }^j|^{2(p-1)}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] ^2\right) \\&\quad \le \left( {\mathbb {E}}|\mathbf{u }^j|^{2p}\right) ^{(p-1)/p}\left( {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] ^{2p}\right) ^{1/p}\,. \end{aligned} \end{aligned}$$

And similarly, the rests are bounded by:

$$\begin{aligned} \begin{aligned}&{\mathbb {E}}\left( |\mathbf{u }^j|^{2(p-1)}\left\langle \mathbf{u }^j,\mathrm {Cov}_{\mathbf{u },\mathbf{r }}\varGamma ^{-\frac{1}{2}}\left( r-\mathrm {m}(u)\right) \right\rangle \right) \\&\quad \le C{\mathbb {E}}\left( |\mathbf{u }^j|^{2(p-\frac{1}{2})}\left[ \frac{1}{J}\sum ^J_{k=1}|\mathbf{e }^k|^2\right] ^{\frac{1}{2}}\right) \\&\quad \le C\left( {\mathbb {E}}|\mathbf{u }^j|^{2p}\right) ^{(p-\frac{1}{2})/p}\left( {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] ^{p}\right) ^{1/(2p)} \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned}&{\mathbb {E}}\left( |\mathbf{u }^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle \mathbf{e }^i,\mathbf{e }^k\right\rangle \left\langle \mathbf{r }^i,\mathbf{r }^k\right\rangle \right] \right) \\&\quad \le C{\mathbb {E}}\left( |\mathbf{u }^j|^{2(p-1)}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] \right) \\&\quad \le C\left( {\mathbb {E}}|\mathbf{u }^j|^{2p}\right) ^{(p-1)/p}\left( {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] ^{p}\right) ^{1/p} \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned}&\frac{1}{J^2}{\mathbb {E}}\left( \left| \mathbf{u }^j\right| ^{2(p-2)}\sum ^J_{i,k=1}\left\langle \mathbf{u }^j,\mathbf{e }^i\right\rangle \left\langle \mathbf{u }^j,\mathbf{e }^k\right\rangle \left\langle \mathbf{r }^i,\mathbf{r }^k\right\rangle \right) \\&\quad \le C{\mathbb {E}}\left( |\mathbf{u }^j|^{2(p-1)}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] \right) \\&\quad \le C\left( {\mathbb {E}}|\mathbf{u }^j|^{2p}\right) ^{(p-1)/p}\left( {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{i=1}|\mathbf{e }^i|^2\right] ^{p}\right) ^{1/p}\,. \end{aligned} \end{aligned}$$

Plugging all these inequalities back in (65) and utilizing (64), we have:

$$\begin{aligned} \frac{\mathrm{d}{\mathbb {E}}|\mathbf{u }^j|^{2p}}{\mathrm{d}t}\le 2C\left( {\mathbb {E}}|\mathbf{u }^j|^{2p}\right) ^{(p-1)/p}\Rightarrow {\mathbb {E}}|\mathbf{u }^j|^{2p}\le C\,. \end{aligned}$$

(66)

Then, to deal with ${\mathbb {E}}|u^j|^{2p}$, we use Ito’s formula similarly, for fix $1\le j\le J$ and $p\ge 1$, we obtain

$$\begin{aligned}&\frac{d|u^j|^{2p}}{\mathrm{d}t}=-2p\left( |u^j|^{2(p-1)}\left\langle u^j,\mathrm {Cov}_{u,\mathbf{u }}\mathbf{u }^j\right\rangle \right) \mathrm{d}t+\mathrm {R} \mathrm{d}W^j_t\\&\quad +p\left( |u^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle e^i,e^k\right\rangle \left\langle \mathbf{e }^k,\mathbf{e }^i\right\rangle \right] \right) \mathrm{d}t\\&\quad +\frac{2p(p-1)}{J^2}\left( \left| u^j\right| ^{2(p-2)}\sum ^J_{i,k=1}\left\langle u^j,e^i\right\rangle \left\langle u^j,e^k\right\rangle \left\langle \mathbf{e }^i,\mathbf{e }^k\right\rangle \right) \mathrm{d}t\\&\quad +2p\left( |u^j|^{2(p-1)}\left\langle u^j,\mathrm {Cov}_{u,\mathbf{r }}\varGamma ^{-\frac{1}{2}}\left( r-\mathrm {m}(u)\right) \right\rangle \right) \mathrm{d}t\\&\quad +p\left( |u^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle e^i,e^k\right\rangle \left\langle \mathbf{r }^i,\mathbf{r }^k\right\rangle \right] \right) \mathrm{d}t\\&\quad +\frac{2p(p-1)}{J^2}\left( \left| u^j\right| ^{2(p-2)}\sum ^J_{i,k=1}\left\langle u^j,e^i\right\rangle \left\langle u^j,e^k\right\rangle \left\langle \mathbf{r }^i,\mathbf{r }^k\right\rangle \right) \mathrm{d}t\,, \end{aligned}$$

where $\mathrm {R}$ is the coefficient before Brownian motion. The six terms are considered separately:

• Term 1
  $$\begin{aligned} \begin{aligned}&\left| {\mathbb {E}}\left( |u^j|^{2(p-1)}\left\langle u^j,\mathrm {Cov}_{u,\mathbf{u }}\mathbf{u }^j\right\rangle \right) \right| \\&\quad \le {\mathbb {E}}\left( |u^j|^{2p-\frac{1}{2}}\frac{1}{J}\sum ^J_{k=1}|e^k||\mathbf{e }^k||\mathbf{u }^j|\right) \\&\quad \le \left( {\mathbb {E}}|u^j|^{2p}\right) ^{(2p-\frac{1}{2})/(2p)}\left( {\mathbb {E}}\left( \frac{1}{J}\sum ^J_{k=1}|e^k||\mathbf{e }^k||\mathbf{u }^j|\right) ^{4p}\right) ^{1/(4p)}\\&\quad \le C\left( {\mathbb {E}}|u^j|^{2p}\right) ^{(2p-\frac{1}{2})/(2p)}\,, \end{aligned} \end{aligned}$$

  where in the last inequality we use (63), (64) and (66) with Hölder’s inequality.

• Term 2
  $$\begin{aligned}&\left| {\mathbb {E}}\left( |u^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle e^i,e^k\right\rangle \left\langle \mathbf{e }^k,\mathbf{e }^i\right\rangle \right] \right) \right| \\&\quad \le C{\mathbb {E}}\left( |u^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}|e^i| |e^k||\mathbf{e }^i||\mathbf{e }^k|\right] \right) \\&\quad \le C{\mathbb {E}}\left( |u^j|^{2(p-1)}\left( \frac{1}{J}\sum ^J_{i=1}|e^i|^2\right) \left( \frac{1}{J}\sum ^J_{k=1}|e^k|^2\right) \right) \\&\quad \le C{\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p} \left( {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{i=1}|e^i|^2\right] ^{2p}\right) ^{1/p}\\&\quad \le C{\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p} \left( {\mathbb {E}}\left[ \sum ^K_{m=1}\frac{1}{J}\sum ^J_{i=1}|e^i_m|^2\right] ^{2p}\right) ^{1/p}\\&\quad \le C{\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p} \left( {\mathbb {E}}\sum ^K_{m=1}\left[ \frac{1}{J}\sum ^J_{i=1}|e^i_m|^2\right] ^{2p}\right) ^{1/p}\\&\quad \le CV_{4p}^{1/p}(e_0){\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p}\,. \end{aligned}$$
• Term 3
  $$\begin{aligned} \begin{aligned}&\left| {\mathbb {E}}\left( \left| u^j\right| ^{2(p-2)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle u^j,e^i\right\rangle \left\langle u^j,e^k\right\rangle \left\langle \mathbf{e }^i,\mathbf{e }^k\right\rangle \right] \right) \right| \\&\le \,C{\mathbb {E}}\left( |u^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}|e^i| |e^k||\mathbf{e }^i||\mathbf{e }^k|\right] \right) \\&\le CV_{4p}^{1/p}(e_0){\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p}\,. \end{aligned} \end{aligned}$$
• Term 4
  $$\begin{aligned} \begin{aligned}&\left| {\mathbb {E}}\left( |u^j|^{2(p-1)}\left\langle u^j,\mathrm {Cov}_{u,\mathbf{r }}\varGamma ^{-\frac{1}{2}}\left( r-\mathrm {m}(u)\right) \right\rangle \right) \right| \\ \le&\quad M^2{\mathbb {E}}\left( |u^j|^{2p-\frac{1}{2}}\frac{1}{J}\sum ^J_{k=1}|e^k|\right) \\ \le&\quad \left( {\mathbb {E}}|u^j|^{2p}\right) ^{(2p-\frac{1}{2})/(2p)}\left( {\mathbb {E}}\left( \frac{1}{J}\sum ^J_{k=1}|e^k|\right) ^{4p}\right) ^{1/(4p)}\\ \le&C\left( {\mathbb {E}}|u^j|^{2p}\right) ^{(2p-\frac{1}{2})/(2p)}\,, \end{aligned} \end{aligned}$$

  where in the last inequality we use (63) and (66) with Hölder’s inequality.

• Term 5
  $$\begin{aligned} \begin{aligned}&\left| {\mathbb {E}}\left( |u^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle e^i,e^k\right\rangle \left\langle \mathbf{r }^k,\mathbf{r }^i\right\rangle \right] \right) \right| \\ \le&\ CM^2{\mathbb {E}}\left( |u^j|^{2(p-1)}\left( \frac{1}{J}\sum ^J_{i=1}|e^i|^2\right) \right) \\ \le&\ C{\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p} \left( {\mathbb {E}}\left[ \frac{1}{J}\sum ^J_{i=1}|e^i|^2\right] ^{p}\right) ^{1/p}\\ \le&\ C{\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p} \left( {\mathbb {E}}\left[ \sum ^K_{m=1}\frac{1}{J}\sum ^J_{i=1}|e^i_m|^2\right] ^{p}\right) ^{1/p}\\ \le&\ C{\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p} \left( {\mathbb {E}}\sum ^K_{m=1}\left[ \frac{1}{J}\sum ^J_{i=1}|e^i_m|^2\right] ^{p}\right) ^{1/p}\\ \le&\ CV_{2p}^{1/p}(e_0){\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p}\,. \end{aligned} \end{aligned}$$
• Term 6
  $$\begin{aligned} \begin{aligned}&\left| {\mathbb {E}}\left( \left| u^j\right| ^{2(p-2)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}\left\langle u^j,e^i\right\rangle \left\langle u^j,e^k\right\rangle \left\langle \mathbf{r }^i,\mathbf{r }^k\right\rangle \right] \right) \right| \\ \le&\,C{\mathbb {E}}\left( |u^j|^{2(p-1)}\left[ \frac{1}{J^2}\sum ^J_{i,k=1}|e^i| |e^k||\mathbf{r }^i||\mathbf{r }^k|\right] \right) \\ \le&\,CV_{2p}^{1/p}(e_0){\mathbb {E}}\left( |u^j|^{2p}\right) ^{(p-1)/p}\,. \end{aligned} \end{aligned}$$

By Lemma 4, we obtain the boundedness for ${\mathbb {E}}\left\| u^j\right\| ^{2p}_2$ . Then, to prove the second inequality of (45), it suffices to prove

$$\begin{aligned} \left( {\mathbb {E}}\left\| \mathrm {Cov}_{u_t}\right\| ^p_2\right) ^{1/p}\le C_p\,, \end{aligned}$$

which is a direct result by expansion of $\mathrm {Cov}_{u_t}$ and triangle inequality:

$$\begin{aligned} \begin{aligned} \left( {\mathbb {E}}\left\| \mathrm {Cov}_{u_t}\right\| ^p_2\right) ^{1/p}&\le \frac{1}{J}\sum ^J_{j=1}\left( {\mathbb {E}}\left\| (u^j-{\overline{u}})\otimes (u^j-{\overline{u}})\right\| ^p_2\right) ^{1/p}\\&\le \frac{1}{J}\sum ^J_{j=1}\left( {\mathbb {E}}\left| u^j-{\overline{u}}\right| ^{2p}\right) ^{1/p}\le C\,. \end{aligned} \end{aligned}$$

Here the last inequality that comes from each term of the sum has a bound

$$\begin{aligned} \begin{aligned}&\left( {\mathbb {E}}\left| u^j-{\overline{u}}\right| ^{2p}\right) ^{1/p}\le \left[ \left( {\mathbb {E}}\left| u^j-{\overline{u}}\right| ^{2p}\right) ^{\frac{1}{2}p}\right] ^{2}\\&\quad \le \left[ \frac{J-1}{J}{\mathbb {E}}\left( |u^j|^{2p}\right) ^{\frac{1}{2}p}+\frac{1}{J}\sum ^J_{k\ne j}{\mathbb {E}}\left( |u^k|^{2p}\right) ^{\frac{1}{2}p}\right] ^{2}\le C\,. \end{aligned} \end{aligned}$$

$\square$