Obtaining minimax lower bounds: a review

Abstract

Minimax lower bounds determine the complexity of given statistical problems by providing fundamental limit of any procedures. This paper gives a review on various aspects of obtaining minimax lower bounds focusing on a recent development. We first introduce classical methods, then more involved lower bound constructions such as testing two mixtures, two directional method, and global metric entropy method are provided with various examples including manifold learning, approximation sets and neural nets. In addition, we consider two different types of restrictions on the set of estimators. In particular, we consider the lower bounds when the set of estimators is required to be linear, and a private version of minimax lower bounds is discussed.

Appendix A: Supplementary materials

Lemma Appendix A.1

(Lemma 4 of Cai and Zhou, 2012) Let ${\bar{{{\mathbb {P}}}}}_m = \sum _{i=1}^m w_i {{\mathbb {P}}}_i$ and ${\bar{{{\mathbb {Q}}}}}_m = \sum _{i=1}^m w_i {{\mathbb {Q}}}_i$ where $w_i \ge 0$ and $\sum _{i=1}^m w_i = 1$. Then

$$\begin{aligned} \Vert {\bar{{{\mathbb {P}}}}}_m \wedge {\bar{{{\mathbb {Q}}}}}_m \Vert _1 \ge \sum _{i=1}^m w_i \Vert {{\mathbb {P}}}_i \wedge {{\mathbb {Q}}}_i\Vert _1 \ge \min _{1\le i\le m} \Vert {{\mathbb {P}}}_i \wedge {{\mathbb {Q}}}_i\Vert _1. \end{aligned}$$

Lemma Appendix A.2

Let ${\bar{{{\mathbb {P}}}}}_{k,i} = \frac{1}{2^{m-1}(n/m)^m} \sum _{\alpha \in vec(A_{k,i} \otimes B)} {{\mathbb {P}}}_{\theta _\alpha }$ for $i=0,1$ where ${{\mathbb {P}}}_{\theta _\alpha } = P_{\theta _\alpha }^n$ is a product measure with a density $\theta _\alpha \in \Theta _F$. Then

$$\begin{aligned} \min _{1 \le k \le m} \Vert {\bar{{{\mathbb {P}}}}}_{k,0} \wedge {\bar{{{\mathbb {P}}}}}_{k,1} \Vert _1 > 0. \end{aligned}$$

Proof of Lemma Appendix A.2

By symmetry, we fix $k=1$. By Cauchy–Schwartz inequality,

$$\begin{aligned} V^2({\bar{{{\mathbb {P}}}}}_{k,0}, {\bar{{{\mathbb {P}}}}}_{k,1})&= \left( \int _{[0,1]^n} |d{\bar{{{\mathbb {P}}}}}_{k,0} - d{\bar{{{\mathbb {P}}}}}_{k,1}| \right) ^2 \\&\le \int _{[0,1]^n} |d{\bar{{{\mathbb {P}}}}}_{k,0} - d{\bar{{{\mathbb {P}}}}}_{k,1}|^2 \\&=\int _{[0,1]^n} \left( \frac{1}{2^{m-1}|B|} \sum _{\alpha \in A_{1,1}} \prod _{i=1}^n\theta _\alpha (x_i) - \frac{1}{2^{m-1}|B|} \sum _{\alpha \in A_{1,0}} \prod _{i=1}^n \theta _{\alpha }(x_i) \right) ^2 \\&= \frac{1}{2^{2m-2}|B|^2} \int \left[ (1)^2-(1) \times (2) + (2)^2 \right] \end{aligned}$$

where $(1) := \sum _{\alpha \in A_{1,1}} \prod _{i=1}^n \theta _\alpha (x_i)$ and $(2) := \sum _{\alpha \in A_{1,0} }\prod _{i=1}^n \theta _\alpha (x_i)$. Note that

$$\begin{aligned} \int \left( 1 \right) ^2&= \sum _{\alpha , \alpha ' \in A_{1,1}} \int \prod _{i=1}^n \theta _\alpha (x_i) \theta _{\alpha '}(x_i) \\&= \sum _{\alpha , \alpha ' \in A_{1,1}} \prod _{i=1}^n \int \left( 1+\sqrt{cl_n} \sum _j \alpha _j \phi _j(x_i) \right) \left( 1+\sqrt{cl_n} \sum _j \alpha '_j \phi _j(x_i) \right) \\&= \sum _{\alpha , \alpha ' \in A_{1,1}} \left( 1+ cl_n \sum _j \alpha _j \alpha '_j \right) ^n \end{aligned}$$

since $\int \phi _j(x)^2 = 1$ and $\int \phi _j(x) \phi _{j'}(x) = 0$ when $j \ne j'$. Similarly,

$$\begin{aligned} \int \left( 2\right) ^2 = \sum _{\alpha , \alpha ' \in A_{1,0}} \left( 1+ cl_n \sum _j \alpha _j \alpha '_j \right) ^n \end{aligned}$$

(A.1)

and

$$\begin{aligned} \int \left( 1 \right) \left( 2 \right) =\sum _{\alpha \in A_{1,1}, \alpha ' \in A_{1,0}} \left( 1+cl_n \sum _{j} \alpha _j \alpha '_j \right) ^n. \end{aligned}$$

(A.2)

By construction, (A.1) and (A.2) are equal. This yields

$$\begin{aligned} V^2({\bar{{{\mathbb {P}}}}}_{k,0}, {\bar{{{\mathbb {P}}}}}_{k,1})&\le \frac{1}{2^{2m-2} |B|^2} \left( \sum _{\alpha , \alpha ' \in A_{1,1}} \left( 1+ cl_n \sum _j \alpha _j \alpha '_j \right) ^n \right. \\&\quad \left. - \sum _{\alpha , \alpha ' \in A_{1,0}} \left( 1+ cl_n \sum _j \alpha _j \alpha '_j \right) ^n \right) =(*). \end{aligned}$$

For compuational convenience, we rewrite ${\underline{\alpha }} = (\alpha _1, \ldots , \alpha _n) = (a_1 \cdot b_1, \ldots , a_m \cdot b_m)$ where $a_k \in \{0,1\}$ and $b_k \in \{(1,0,\ldots ,0), (0,1,0, \ldots , 0), \ldots , (0,\ldots ,0,1)\}$ where $(a_1 \cdot b_1) = (a_1 b_{11}, \ldots , a_1 b_{1p})$ with $p=n/m$. When $\alpha \in A_{1,1}$, then $a_1 = 1$ and when $\alpha \in A_{1,0}$, then $a_1 = 0$. This implies

$$\begin{aligned} (*)&\le \frac{1}{2^{2m-2} |B|^2} \sum _{\alpha , \alpha ' \in A_{1,1}} \left\{ \left( 1+cl_n\left( b_1^Tb'_1+\sum _{k=2}^m (a_k \cdot b_k)^T \left( a'_k \cdot b'_k\right) \right) \right) ^n \right. \\&\quad \left. - \left( 1+cl_n \sum _{k=2}^m (a_k \cdot b_k)^T\left( a'_k \cdot b'_k\right) \right) ^n \right\} . \end{aligned}$$

It is clear $(a_k \cdot b_k)^T (a'_k \cdot b'_k) = (a_k a'_k) (b_k^T b'_k)$, and in fact, this is nonzero if and only if $a_k = a'_k = 1$ and $b_k = b'_k$. Based on this observation, we consider new Bernoulli random variables

$$\begin{aligned} b_1^T b'_1 := B_1 \sim Ber\left( \frac{m^2}{4n^2} \right) , \ \ \left( a_ka'_k\right) \left( b_k^T b'_k\right) :=A_k \sim Ber\left( \frac{m^2}{4n^2} \right) . \end{aligned}$$

Then it suffices to show that the following is bounded above by 1.

$$\begin{aligned} \sum _{j=0}^{m-1} \left\{ P\left( B_1 = 1, \sum _{k=2}^m A_k = j\right) \left( (1+(j+1) cl_n)^n - (1+jcl_n)^n \right) \right\} \end{aligned}$$

(A.3)

When $j=m-1$, all $A_k$’s (for $k=2, \ldots , m$) must have value 1. Thus, $P(B_1 = 1, \sum _{k=2}^m A_k = m-1) = (1/4)^{m-1} (m^2/n^2)^m$. Note that $X:= \sum _{k=2}^m A_k \sim B\left( m-1, \frac{m^2}{4n^2}\right)$ where $B(m-1,p)$ is the binomial distribution with a success probability p among $m-1$ trials. Let $p_j = P(X = j)$, then $p_j = {m-1 \atopwithdelims ()j} \left( \frac{m^2}{4n^2} \right) ^j \left( 1- \frac{m^2}{4n^2} \right) ^{m-1-j}$. By the property of moment generating function of X,

$$\begin{aligned} (A.3)&= \frac{m^2}{n^2} \sum _{j=0}^{m-1} p_j \Big ((1+(j+1) cl_n)^n - (1+jcl_n)^n \Big ) \\&\le \frac{m^2}{n^2}\sum _{j=0}^{m-1}p_j \exp \big (c(j+1) \log n \big ) \\&= \frac{m^2}{n^2} n^c E\big (\exp (c\log n X)\big ) = \frac{m^2}{n^2}n^c \left( 1-\frac{m^2}{4n^2} + \frac{m^2}{4n^2}n^c \right) ^m. \end{aligned}$$

Choose $c < \frac{4s-1}{2s+1}$ so that

$$\begin{aligned} \frac{m^2}{n^2}n^c \left( 1-\frac{m^2}{4n^2} + \frac{m^2}{4n^2}n^c \right) ^m \le m^2 n^{c-2} \exp \Big (m^3 n^{c-2} \Big ) \le 1/2. \end{aligned}$$

Thus (A.3) $\le 1/2$.

$\square$

Proof (sketch) of Lemma 4.3

For any P, Q of probability measures where $P\ll Q$ and also $Q \ll P$ with dominating measure $\nu$, note that

$$\begin{aligned} KL(P,Q)+KL(Q,P)&= \int (p-q) \log \frac{p}{q} \le \int \frac{|p-q|^2}{\min (p,q)} \end{aligned}$$

where the inequality follows since for $a\ge 0, b \ge 0$, $|\log (a/b)| \le |a-b|/\min (a,b)$. This gives

$$\begin{aligned} KL(G_{\theta _0}, G_{\theta _1})+KL(G_{\theta _1}, G_{\theta _0}) \le \int _{{\mathcal {Z}}}\frac{ \left\{ g_{\theta _0}(s) - g_{\theta _1}(s)\right\} ^2}{\min \left\{ g_{\theta _0}(s), g_{\theta _1}(s) \right\} }. \end{aligned}$$

(A.4)

Dividing the probability into two parts and by taking the supremum with respect to x first and then by definition of the total variation distance,

$$\begin{aligned} g_{\theta _0}(s)-&g_{\theta _1}(s) = \int _{{\mathcal {X}}}q(s|x) \left\{ \big (dP_{\theta _0}(x)-dP_{\theta _1}(x)\big )_+ + \big (dP_{\theta _0}(x)-dP_{\theta _1}(x)\big )_{-}\right\} \nonumber \\&\le \sup _x q(s|x) \int _{{\mathcal {X}}}\big (dP_{\theta _0}(x)-dP_{\theta _1}(x)\big )_+ + \inf _x q(s|x) \big (dP_{\theta _0}(x)-dP_{\theta _1}(x)\big )_{-} \nonumber \\&\le \left( \sup _x q(s|x)-\inf _x q(s|x) \right) TV(P_{\theta _0}, P_{\theta _1}) \nonumber \\&\le \sup _{x,x'} \big | q(s|x)-q(s|x')\big | TV(P_{\theta _0}, P_{\theta _1}). \end{aligned}$$

(A.5)

Note that

$$\begin{aligned} \sup _{x,x'} \big | q(s|x)-q(s|x')\big | \le 2 \inf _{{\hat{x}}} \sup _{x} \big | q(s|x)-q(s| {\hat{x}})\big | = 2\inf _{{\hat{x}}} q(s|{\hat{x}}) \sup _x \left| \frac{q(s|x)}{q(s|{\hat{x}})} -1 \right| . \end{aligned}$$

Also we have

$$\begin{aligned} \sup _{x,x'} \big | q(s|x)-q(s|x')\big | = \sup _{x,x'} q(s|x') \left| \frac{q(s|x)}{q(s|x')}-1 \right| \le e^\eta \inf _{{\hat{x}}} q(s|{\hat{x}}) \sup _{x,x'}\left| \frac{q(s|x)}{q(s|x')}-1 \right| . \end{aligned}$$

For any $x,x'$, $q(s|x)/q(s|x') \le [e^{-\eta }, e^\eta ]$, which implies that

$$\begin{aligned} \sup _{x,x'} \big | q(s|x)-q(s|x')\big | \le \min (2,e^\eta ) \inf _{x} q(s|x) (e^\eta -1). \end{aligned}$$

(A.6)

Now we consider the denominator in (A.4). That is,

$$\begin{aligned} \frac{1}{\min \{g_{\theta _0}(s), g_{\theta _1}(s) \}}&= \frac{1}{\min \{ \int _{{\mathcal {X}}}q(s|x)dP_{\theta _0}(x), \int _{{\mathcal {X}}}q(s|x) dP_{\theta _1}(x) \}} \nonumber \\&\le \frac{1}{\inf _x q(s|x)}. \end{aligned}$$

(A.7)

Combining (A.5), (A.6), and (A.7), we have the desired bound. $\square$