Phase transitions for detecting latent geometry in random graphs

Abstract

Random graphs with latent geometric structure are popular models of social and biological networks, with applications ranging from network user profiling to circuit design. These graphs are also of purely theoretical interest within computer science, probability and statistics. A fundamental initial question regarding these models is: when are these random graphs affected by their latent geometry and when are they indistinguishable from simpler models without latent structure, such as the Erdős–Rényi graph ${\mathcal {G}}(n, p)$? We address this question for two of the most well-studied models of random graphs with latent geometry—the random intersection and random geometric graph. Our results are as follows: (a) The random intersection graph is defined by sampling n random sets $S_1, S_2, \ldots , S_n$ by including each element of a set of size d in each $S_i$ independently with probability $\delta$, and including the edge $\{i, j\}$ if $S_i \cap S_j \ne \emptyset$. We prove that the random intersection graph converges in total variation to an Erdős–Rényi graph if $d = {\tilde{\omega }}(n^3)$, and does not if $d = o(n^3)$, for both dense and sparse edge densities p. This resolves an open problem in Fill et al. (Random Struct Algorithms 16(2):156–176, 2000), Rybarczyk (Random Struct Algorithms 38(1–2):205–234, 2011) and Kim et al. (Random Struct Algorithms 52(4):662–679, 2018). The same result was obtained simultaneously and independently by Bubeck et al. (When random intersection graphs lose geometry. Manuscript, 2019). (b) We strengthen the preceding argument to show that the matrix of random intersection sizes $|S_i \cap S_j|$ converges in total variation to a symmetric matrix with independent Poisson entries. This yields the first total variation convergence result for $\tau$-random intersection graphs, where the edge $\{i, j\}$ is included if $|S_i \cap S_j| \ge \tau$. More precisely, our results imply that, if p is bounded away from 1, then the $\tau$-random intersection graph with edge density p converges to ${\mathcal {G}}(n, p)$ if $d = \omega (\tau ^3 n^3)$. (c) The random geometric graph on ${\mathbb {S}}^{d - 1}$ is defined by sampling $X_1, X_2, \dots , X_n$ uniformly at random from ${\mathbb {S}}^{d - 1}$ and including the edge $\{i, j\}$ if $\Vert X_i - X_j \Vert _2 \le \tau$. A result of Bubeck et al. (Random Struct Algorithms 49:503–532, 2016) showed that this model converges to ${\mathcal {G}}(n, p)$ in total variation, where p is chosen so that the models have matching edge densities, as long as $d = \omega (n^3)$. It was conjectured in Bubeck et al. (2016) that this threshold decreases drastically for p small. We make the first progress towards this conjecture by showing convergence if $d = {\tilde{\omega }}\left( \min \{ pn^3, p^2 n^{7/2} \} \right)$. Our proofs are a hybrid of combinatorial arguments, direct couplings and applications of information inequalities. Previous upper bounds on the total variation distance between random graphs with latent geometry and ${\mathcal {G}}(n, p)$ have typically not been both combinatorial and information-theoretic, while this interplay is essential to the sharpness of our bounds.

Appendices

Appendix: Random Intersection Graphs and Matrices

1.1 Variance of the Signed Triangle Count in ${\textsc {RIG}}(n, d, p)$

The main purpose of this section is to prove Lemma 3.4, which computes the variance of $T_s(G)$ for $G \sim {\textsc {rig}}(n, d, p)$. The proof follows a similar structure to the proof of Lemma 3.3 but is more computationally involved.

Proof of Lemma 3.4

Let $\tau _{ijk} = (e_{ij} - p)(e_{ik} - p)(e_{jk} - p)$ for each $1 \le i< j < k \le n$. It holds that

$$\begin{aligned} \mathrm{Var}[T_s(G)]&= \sum _{1 \le i< j< k \le n} \sum _{1 \le i'< j' < k' \le n} \text {Cov}\left[ \tau _{ijk}, \tau _{i'j'k'} \right] \nonumber \\&= \left( {\begin{array}{c}n\\ 3\end{array}}\right) \cdot \text {Var}[\tau _{123}] + \frac{4!}{2! \cdot 2!} \cdot \left( {\begin{array}{c}n\\ 4\end{array}}\right) \cdot \text {Cov}\left[ \tau _{123}, \tau _{124} \right] \nonumber \\&\quad + \frac{5!}{2! \cdot 2!} \cdot \left( {\begin{array}{c}n\\ 5\end{array}}\right) \cdot \text {Cov}\left[ \tau _{123}, \tau _{145} \right] \end{aligned}$$

(A.1)

The second equality follows by symmetry among vertex labels and the fact that if $\{i, j, k\} \cap \{i', j', k'\} = \emptyset$ then $\tau _{ijk}$ and $\tau _{i'j'k'}$ are independent. Note that the second coefficient is the number of ways to choose two sets of three vertices that intersect in two elements and the third coefficient is the number of ways to choose these sets so that they intersect in one element. By Lemma 3.3, we have that

$$\begin{aligned} q = {\mathbb {E}}[\tau _{123} = 1] = (1 - p)^3 \cdot \left[ d\delta ^3 + O_n(d\delta ^4) \right] \end{aligned}$$

Now note that

$$\begin{aligned}&\text {Var}[\tau _{123}] = {\mathbb {E}}[\tau _{123}^2] - q^2, \quad \quad \text {Cov}\left[ \tau _{123}, \tau _{124} \right] = {\mathbb {E}}[\tau _{123}\tau _{124}] - q^2 \quad \text {and} \quad \\&\text {Cov}\left[ \tau _{123}, \tau _{145} \right] = {\mathbb {E}}[\tau _{123}\tau _{145} = 1] - q^2 \end{aligned}$$

We will begin by computing ${\mathbb {E}}[\tau _{123}^2]$. Let P and Q be as in Lemma 3.3. Now note that

$$\begin{aligned} {\mathbb {E}}[\tau _{123}^2]&= {\mathbb {E}}\left[ (e_{12} - p)^2(e_{13} - p)^2(e_{23} - p)^2 \right] \\&= {\mathbb {E}}\left[ \prod _{\{i, j\} = \{1, 2\}, \{1, 3\}, \{2, 3\}} \left[ (1 - p)^2 - (1 - 2p)(1 - e_{ij}) \right] \right] \\&= -\sum _{x \in \{0, 1\}^3} (1 - p)^{2|x|}(-1)^{|x|} (1 - 2p)^{3 - |x|} \cdot Q(x_1, x_2, x_3) \\&= (1 - p)^6 - 3(1 - p)^4 (1 - 2p) (1 - \delta )^d(1 + \delta )^d \\&\quad + 3(1 - p)^2 (1 - 2p)^2 (1 - \delta )^d(1 + \delta - \delta ^2)^d \\&\quad - (1 - 2p)^3 (1 + 2\delta )^d (1 - \delta )^{2d} \end{aligned}$$

where the last two equalities follow from the expressions for Q in Lemma 3.3. Further simplifying and applying the estimates in Eqs. 3.19 and 3.20 yields that the above quantity is equal to

$$\begin{aligned}&(1 - p)^6 - 3(1 - p)^5 (1 - 2p) + 3(1 - p)^4 (1 - 2p)^2 \cdot \left( 1 + \frac{\delta ^3}{(1 - \delta ^2)(1 + \delta )} \right) ^d \\&\qquad - (1 - 2p)^3 (1 - p)^3 \cdot \left( 1 + \frac{2\delta ^3 + \delta ^4}{(1 - \delta ^2)(1 + \delta )^2} \right) ^d \\&\quad = (1 - p)^3 \cdot \left[ (1 - p) - (1 - 2p) \right] ^3 + 3(1 - p)^4 (1 - 2p)^2 \\&\qquad \cdot \left[ \left( 1 + \frac{\delta ^3}{(1 - \delta ^2)(1 + \delta )} \right) ^d - 1\right] \\&\qquad - (1 - 2p)^3 (1 - p)^3 \cdot \left[ \left( 1 + \frac{2\delta ^3 + \delta ^4}{(1 - \delta ^2)(1 + \delta )^2} \right) ^d - 1 \right] \\&\quad = p^3 (1 - p)^3 + 3(1 - p)^4 (1 - 2p)^2 d \delta ^3 - (1 - 2p)^3 (1 - p)^3 d\delta ^3 + O_n(d\delta ^4) \\&\quad = p^3 (1 - p)^3 + (2 - p)(1 - p)^3 (1 - 2p)^2 d \delta ^3 + O_n(d\delta ^4) \end{aligned}$$

We now will estimate ${\mathbb {E}}[\tau _{123}\tau _{124}]$ using a similar method to Lemma 3.3. Let $P' : \{0, 1\}^5 \rightarrow [0, 1]$ be such that $P'(x_1, x_2, x_3, x_4, x_5)$ is the probability that $e_{12} = x_1$, $e_{13} = x_2$, $e_{14} = x_3$, $e_{23} = x_4$ and $e_{24} = x_5$. Define $Q' : \{0, 1\}^5 \rightarrow [0, 1]$ as

$$\begin{aligned} Q'(x_1, x_2, x_3, x_4, x_5) = \sum _{y \subseteq x} P'(y_1, y_2, y_3, y_4, y_5) \end{aligned}$$

As in Lemma A.1, the events whose probabilities are given by the values of $Q'$ are each the product of events over the individual elements of [d]. For no edges to be present in the triangles $\{1, 2, 3\}$ or $\{1, 2, 4\}$, each $i \in [d]$ must be in at most one of $S_1, S_2, S_3, S_4$ or is in both of $S_3$ and $S_4$. Thus

$$\begin{aligned} P'(0, 0, 0, 0, 0) = Q'(0, 0, 0, 0, 0)&= \left[ (1 - \delta )^4 + 4\delta (1 - \delta )^3 + \delta ^2(1 - \delta )^2 \right] ^d \\&= \left( 1 + 2\delta - 2\delta ^2 \right) ^d(1 - \delta )^{2d} \end{aligned}$$

Similarly, if $|x| = x_1 + x_2 + x_3 + x_4 + x_5 = 1$, then

$$\begin{aligned} Q'(x_1, x_2, x_3, x_4, x_5)&= \left[ (1 - \delta )^4 + 4\delta (1 - \delta )^3 + 2\delta ^2(1 - \delta )^2 \right] ^d \\&= \left( 1 + 2\delta - \delta ^2 \right) ^d(1 - \delta )^{2d} \end{aligned}$$

If $|x| = 2$ and $x \ne (0, 1, 1, 0, 0), (0, 0, 0, 1, 1)$, then it follows that

$$\begin{aligned} Q'(x_1, x_2, x_3, x_4, x_5)&= \left[ (1 - \delta )^4 + 4\delta (1 - \delta )^3 + 3\delta ^2(1 - \delta )^2 \right] ^d \\&= \left( 1 + 2\delta \right) ^d(1 - \delta )^{2d} \end{aligned}$$

If $x = (0, 1, 1, 0, 0), (0, 0, 0, 1, 1)$, then each $i \in [d]$ can also possibly be in the three sets $S_1, S_3, S_4$ and $S_2, S_3, S_4$, respectively. Therefore

$$\begin{aligned} Q'(0, 1, 1, 0, 0)&= Q'(0, 0, 0, 1, 1) \\&= \left[ (1 - \delta )^4 + 4\delta (1 - \delta )^3 + 3\delta ^2(1 - \delta )^2 + \delta ^3(1 - \delta ) \right] ^d \\&= \left( 1 + \delta - 2\delta ^2 + \delta ^3 \right) ^d(1 - \delta )^{d} \end{aligned}$$

We now consider the cases where $|x| = 3$. When $|x| = 3$, there are always four allowed pairs of sets that any $i \in [d]$ can be in—the three edges of x and $\{3, 4\}$. However, the number of allowed triples varies with x. If $x = (1, 1, 0, 0, 1), (1, 0, 1, 1, 0)$, then there are no allowed triples and

$$\begin{aligned} Q'(1, 1, 0, 0, 1) = Q'(1, 0, 1, 1, 0)&= \left[ (1 - \delta )^4 + 4\delta (1 - \delta )^3 + 4\delta ^2(1 - \delta )^2 \right] ^d \\&= (1 + \delta )^{2d}(1 - \delta )^{2d} \end{aligned}$$

If $|x| = 3$ and $x \ne (1, 1, 0, 0, 1), (1, 0, 1, 1, 0)$, then there is one allowed triple and

$$\begin{aligned} Q'(x_1, x_2, x_3, x_4, x_5)&= \left[ (1 - \delta )^4 + 4\delta (1 - \delta )^3 + 4\delta ^2(1 - \delta )^2 + \delta ^3(1 - \delta ) \right] ^d \\&= \left( 1 + \delta - \delta ^2 \right) ^d(1 - \delta )^{d} \end{aligned}$$

If $|x| = 4$, then there is one forbidden pair of sets and two forbidden triples. Therefore

$$\begin{aligned} Q'(x_1, x_2, x_3, x_4, x_5)&= \left[ (1 - \delta )^4 + 4\delta (1 - \delta )^3 + 5\delta ^2(1 - \delta )^2 + 2\delta ^3(1 - \delta ) \right] ^d \\&= \left( 1 + \delta \right) ^d(1 - \delta )^{d} \end{aligned}$$

Furthermore $Q'(1, 1, 1, 1, 1) = 1$. Now we have that

$$\begin{aligned} {\mathbb {E}}[\tau _{123}\tau _{124}]&= {\mathbb {E}}\left[ (e_{12} - p)^2 (e_{13} - p) (e_{14} - p) (e_{23} - p) (e_{24} - p) \right] \\&= {\mathbb {E}}\left[ \left[ (1 - p)^2 - (1 - 2p)(1 - e_{12}) \right] \right. \\&\qquad \left. \times \prod _{\{i, j\} = \{1, 3\}, \{1, 4\}, \{2, 3\}, \{2, 4\}} \left[ (1 - p) - (1 - e_{ij}) \right] \right] \\&= -\sum _{x \in \{0, 1\}^5} (-1)^{|x|} (1 - p)^{2x_1} (1 - 2p)^{1 - x_1} (1 - p)^{x_2 + x_3 + x_4 + x_5} \\&\quad \cdot Q'(x_1, x_2, x_3, x_4, x_5) \\&= (1 - p)^6 - (1-p)^4 \cdot \left[ 4(1 - p) + (1 - 2p) \right] \cdot \left( 1 + \delta \right) ^d(1 - \delta )^{d} \\&\qquad + (1 - p)^3 \cdot \left[ 4(1 - p) + 4(1 - 2p) \right] \cdot \left( 1 + \delta - \delta ^2 \right) ^d(1 - \delta )^{d} \\&\qquad + 2(1 - p)^4 \cdot (1 + \delta )^{2d}(1 - \delta )^{2d} \\&\qquad - (1 - p)^2 \cdot \left[ 4(1 - p) + 4(1 - 2p) \right] \cdot \left( 1 + 2\delta \right) ^d(1 - \delta )^{2d} \\&\qquad - 2(1 - p)^2 (1 - 2p) \cdot \left( 1 + \delta - 2\delta ^2 + \delta ^3 \right) ^d(1 - \delta )^{d} \\&\qquad + (1 - p) \cdot \left[ (1 - p) + 4(1 - 2p) \right] \cdot \left( 1 + 2\delta - \delta ^2 \right) ^d(1 - \delta )^{2d} \\&\qquad - (1 - 2p)\left( 1 + 2\delta - 2\delta ^2 \right) ^d(1 - \delta )^{2d} \end{aligned}$$

This quantity can be rewritten as the following expression which is more convenient to estimate.

$$\begin{aligned}&(1 - p)^6 - (1-p)^5 (5 - 6p) + 4(1 - p)^5 (2 - 3p) \cdot \left( 1 + \frac{\delta ^3}{(1 + \delta )(1 - \delta ^2)} \right) ^d \\&\qquad + 2(1 - p)^6 - 4(1 - p)^5 (2 - 3p) \cdot \left( 1 + \frac{2\delta ^3 + \delta ^4}{(1 - \delta ^2)(1 + \delta )^2} \right) ^d \\&\qquad - 2(1 - p)^5 (1 - 2p) \cdot \left( 1 + \frac{\delta ^3 - \delta ^4 - \delta ^5}{(1 - \delta ^2)^2(1 + \delta )} \right) ^d \\&\qquad + (1 - p)^5 (5 - 9p) \cdot \left( 1 + \frac{4\delta ^3 + \delta ^4 - 2\delta ^5 - \delta ^6}{(1 - \delta ^2)^2(1 + \delta )^2} \right) ^d \\&\qquad - (1 - p)^5 (1 - 2p) \left( 1 + \frac{6\delta ^3 - 6 \delta ^5 - 2\delta ^6 + 2\delta ^7 + \delta ^8}{(1 - \delta ^2)^3(1 + \delta )^2} \right) ^d \\&\quad = 4(1 - p)^5(2 - 3p) \cdot d\delta ^3 - 4(1 - p)^5(2 - 3p)\\&\qquad \cdot 2d\delta ^3 - 2(1 - p)^5(1 - 2p) \cdot 2\delta ^3 \\&\qquad + (1 - p)^5(5 - 9p) \cdot 4d\delta ^3 - (1 - p)^5(1 - 2p) \cdot 6d\delta ^3 + O_n(d\delta ^4) \\&\quad = 2(1 - p)^5 (1 - 2p) d \delta ^3 + O_n(d\delta ^4) \end{aligned}$$

The second last equality follows from substituting estimates of the form

$$ \left( 1 + \frac{\delta ^3}{(1 + \delta )(1 - \delta ^2)} \right) ^d = 1 + d\delta ^3 + O_n(d\delta ^4) $$

and analogous estimates for the other dth powers in the expression. These estimates can be established using similar bounds to those used to derive Eqs. 3.19 and 3.20. Observe that the terms that are not multiples of $d\delta ^3$ after substituting these estimates sum to zero.

We now will estimate ${\mathbb {E}}[\tau _{123}\tau _{145}]$ using a slightly different method. Note that $\tau _{123}$ is $\sigma (S_1, S_2, S_3)$-measurable and $\tau _{145}$ is $\sigma (S_1, S_4, S_5)$-measurable. Thus conditioned on $S_1$, the random variables $\tau _{123}$ and $\tau _{145}$ are independent. Furthermore, because of symmetry among the elements in [d], $\tau _{123}$ and $\tau _{145}$ are independent conditioned on $|S_1|$. Let $\tau _{123}^m = {\mathbb {E}}\left[ \tau _{123} \big | |S_1| = m\right]$ and observe that conditional independence yields that ${\mathbb {E}}[\tau _{123}\tau _{145}] = {\mathbb {E}}_{m \sim {\mathcal {L}}(|S_1|)} \left[ (\tau _{123}^m)^2 \right]$. We now will compute $\tau ^m_{123}$. Let $P_m : \{0, 1\}^3 \rightarrow [0, 1]$ be such that $P_m(x_1, x_2, x_3)$ is the probability that $e_{12} = x_1, e_{13} = x_2$ and $e_{23} = x_3$ given $|S_1| = m$. Define $Q_m : \{0, 1\}^3 \rightarrow [0, 1]$ to be

$$ Q_m(x_1, x_2, x_3) = \sum _{y \subseteq x} P_m(x_1, x_2, x_3) $$

For no edges in triangle $\{1, 2, 3\}$ to be present, each of the m elements of $S_1$ cannot be in either $S_2$ or $S_3$ and each of the $d - m$ remaining elements must be in at most one of $S_2$ or $S_3$. Therefore

$$ Q_m(0, 0, 0) = (1 - \delta )^{2m}( 1 - \delta ^2)^{d - m} $$

For either no edges or just the edge $\{1, 2\}$ to be present, each element in $S_1$ must not be in $S_3$ and each of the $d - m$ remaining elements must be in at most one of $S_2$ or $S_3$. Similar conditions hold when $\{1, 2\}$ is replaced by $\{1, 3\}$ and thus

$$\begin{aligned} Q_m(1, 0, 0) = Q_m(0, 1, 0) = (1 - \delta )^{m}( 1 - \delta ^2)^{d - m} \end{aligned}$$

For at most the edge $\{2, 3\}$ to be present, each element of $S_1$ cannot be in $S_2$ or $S_3$ and thus

$$\begin{aligned} Q_m(0, 0, 1) = (1 - \delta )^{2m} \end{aligned}$$

For just the edge $\{1, 2\}$ to not be present, each element of $S_1$ cannot be in $S_2$. Similar conditions hold when $\{1, 2\}$ is replaced by $\{1, 3\}$ and thus

$$\begin{aligned} Q_m(0, 1, 1) = Q_m(1, 0, 1) = (1 - \delta )^{m} \end{aligned}$$

For $\{2, 3\}$ to not be present, it must hold that each of the $d - m$ elements not in $S_1$ are in one of $S_2$ or $S_3$. Thus

$$\begin{aligned} Q_m(1, 1, 0) = (1 - \delta ^2)^{d - m} \end{aligned}$$

Furthermore $Q_m(1, 1, 1) = 1$. Now we have that

$$\begin{aligned} \tau _{123}^m&= -{\mathbb {E}}\left[ \left( (1 - e_{12}) - (1 - p) \right) \left( (1 - e_{13}) - (1 - p) \right) \left( (1 - e_{23}) - (1 - p) \right) \big | |S_1| = m\right] \\&= - \sum _{x \in \{0, 1\}^3} (-1)^{|x|} (1 - p)^{|x|} \cdot Q_m(x_1, x_2, x_3) \\&= (1 - p)^3 - 2(1 - p)^2(1 - \delta )^{m} - (1 - p)^2(1 - \delta ^2)^{d - m} \\&\quad + 2(1 - p) \cdot (1 - \delta )^{m}( 1 - \delta ^2)^{d - m} \\&\quad + (1 - p) (1 - \delta )^{2m} - (1 - \delta )^{2m}( 1 - \delta ^2)^{d - m} \end{aligned}$$

Now note that $|S_1| \sim \text {Bin}(d, \delta )$ and thus ${\mathbb {E}}_{m \sim {\mathcal {L}}(|S_1|)} \left[ x^m \right] = \left( 1 - \delta + \delta x \right) ^d$ for any $x > 0$, by the form of the moment generating function of the binomial distribution. Expanding $(\tau _{123}^m)^2$ and applying this identity now yields that

$$\begin{aligned} {\mathbb {E}}_{m \sim {\mathcal {L}}(|S_1|)} \left[ (\tau _{123}^m)^2 \right]&= (1 - p)^6 - 4(1 - p)^5 (1 - \delta ^2)^d - 2(1 - p)^5 (1 - \delta ^2)^d \\&\quad \left( 1 - \delta + \frac{\delta }{1 - \delta ^2} \right) ^d \\&\quad + 6(1 - p)^4 (1 - 2\delta ^2 + \delta ^3)^d + (1 - p)^4 (1 - \delta ^2)^{2d} \\&\quad \left( 1 - \delta + \frac{\delta }{(1 - \delta ^2)^2} \right) ^d \\&\quad + 8(1 - p)^4 (1 - \delta ^2)^d \left( 1 - \delta + \frac{\delta (1 - \delta )}{1 - \delta ^2} \right) ^d \\&\quad - 12(1 - p)^3 (1 - \delta ^2)^d \left( 1 - \delta + \frac{\delta (1 - \delta )^2}{1 - \delta ^2} \right) ^d \\&\quad - 4(1 - p)^3 (1 - 3\delta ^2 + 3\delta ^3 - \delta ^4)^d \\&\quad - 4(1 - p)^3 (1 - \delta ^2)^{2d} \left( 1 - \delta + \frac{\delta (1 - \delta )}{(1 - \delta ^2)^2} \right) ^d \\&\quad + 6(1 - p)^2 (1 - \delta ^2)^{2d} \left( 1 - \delta + \frac{\delta (1 - \delta )^2}{(1 - \delta ^2)^2} \right) ^d \\&\quad + (1 - p)^2 (1 - 4\delta ^2 + 6\delta ^3 - 4\delta ^4 + \delta ^5)^d \\&\quad + 8(1 - p)^2 (1 - \delta ^2)^d \left( 1 - \delta + \frac{\delta (1 - \delta )^3}{1 - \delta ^2} \right) ^d \\&\quad - 4(1 - p) (1 - \delta ^2)^{2d} \left( 1 - \delta + \frac{\delta (1 - \delta )^3}{(1 - \delta ^2)^2} \right) ^d \\&\quad - 2(1 - p) (1 - \delta ^2)^d \left( 1 - \delta + \frac{\delta (1 - \delta )^4}{1 - \delta ^2} \right) ^d \\&\quad + (1 - \delta ^2)^{2d} \left( 1 - \delta + \frac{\delta (1 - \delta )^4}{(1 - \delta ^2)^2} \right) ^d \end{aligned}$$

This quantity can be rewritten as the following expression which is more convenient to estimate.

$$\begin{aligned}&(1 - p)^6 - 4(1 - p)^6 - 2(1 - p)^6 \left( 1 + \frac{\delta ^3}{1 - \delta ^2} \right) ^d \\&\quad + 14(1 - p)^6 \left( 1 + \frac{\delta ^3}{(1 - \delta ^2)(1 + \delta )} \right) ^d + (1 - p)^6 \left( 1 + \frac{2\delta ^3 - \delta ^5}{(1 - \delta ^2)^2} \right) ^d \\&\quad - 16(1 - p)^6 \left( 1 + \frac{2\delta ^3 + \delta ^4}{(1 - \delta ^2)(1 + \delta )^2} \right) ^d \\&\quad - 4(1 - p)^6 \left( 1 + \frac{3\delta ^3 - \delta ^4 - \delta ^5}{(1 - \delta ^2)^2(1 + \delta )}\right) ^d \\&\quad + 6(1 - p)^6 \left( 1 + \frac{3\delta ^3 + \delta ^4 - 2\delta ^5 - \delta ^6}{(1 - \delta ^2)^2(1 + \delta )^2} \right) ^d \\&\quad + (1 - p)^6 \left( 1 + \frac{6\delta ^3 - 4\delta ^4 - 3\delta ^5 + \delta ^6 + \delta ^7}{(1 - \delta ^2)^3 (1 + \delta )} \right) ^d \\&\quad + 8(1 - p)^6 \left( 1 + \frac{4\delta ^3 + \delta ^4 - 2\delta ^5 - \delta ^6}{(1 - \delta ^2)^2(1 + \delta )^2} \right) ^d \\&\quad - 4(1 - p)^6 \left( 1 + \frac{5\delta ^3 + 5\delta ^4 - \delta ^5 - 3\delta ^6 - \delta ^7}{(1 - \delta ^2)^2(1 + \delta )^3} \right) ^d \\&\quad - 2(1 - p)^6 \left( 1 + \frac{7\delta ^3 - 6\delta ^4 - 2\delta ^5 + 2\delta ^7 + \delta ^8}{(1 - \delta ^2)^3(1 + \delta )^2} \right) ^d \\&\quad + (1 - p)^6 \left( 1 + \frac{8\delta ^3 + 6\delta ^4 - 6\delta ^5 - 8\delta ^7 + 3\delta ^8 + \delta ^9}{(1 - \delta ^2)^3(1 + \delta )^3} \right) ^d \end{aligned}$$

Now substitute estimates for each of the dth powers of the form $1 + cd\delta ^3 + O_n(d\delta ^4)$ for constants c varying per term. For example, the first power can be estimated to be

$$ \left( 1 + \frac{\delta ^3}{1 - \delta ^2} \right) ^d = 1 + d\delta ^3 + O_n(d\delta ^4) $$

These estimates can be established using the same bounding argument used to derive Eqs. 3.19 and 3.20. Observe that the sum of the constant terms and the multiples of $d\delta ^3$ are zero after substituting these estimates into the expression above for ${\mathbb {E}}_{m \sim {\mathcal {L}}(|S_1|)} \left[ (\tau _{123}^m)^2 \right]$. Thus we obtain that

$$ {\mathbb {E}}_{m \sim {\mathcal {L}}(|S_1|)} \left[ (\tau _{123}^m)^2 \right] = O_n(d\delta ^4) $$

Now note that $q = O_n(d\delta ^3)$ and therefore we have that

$$\begin{aligned} \text {Var}[\tau _{123}]&= {\mathbb {E}}[\tau _{123}^2] - q^2 = p^3 (1 - p)^3 + O_n(d\delta ^3) \\ \text {Cov}\left[ \tau _{123}, \tau _{124} \right]&= {\mathbb {E}}[\tau _{123}\tau _{124}] - q^2 = O_n(d\delta ^3) \\ \text {Cov}\left[ \tau _{123}, \tau _{145} \right]&= {\mathbb {E}}[\tau _{123}\tau _{145} = 1] - q^2 = O_n(d\delta ^4) \end{aligned}$$

since $d\delta ^2 = O_n(1)$. Substituting into Eq. A.1 completes the proof of the lemma. $\square$

For the sake of completeness, we show how to apply the approach in Lemma 3.3 to compute ${\mathbb {E}}[T(G)]$ where $G \sim {\textsc {rig}}(n, d, p)$ in the following lemma.

Lemma A.1

If $G \sim {\textsc {rig}}(n, d, p)$ where $1 - p = (1 - \delta ^2)^d = \Omega _n(1)$, then it follows that

$$ {\mathbb {E}}\left[ T(G) \right] = \left( {\begin{array}{c}n\\ 3\end{array}}\right) \cdot \left[ p^3 + d \delta ^3 (1 + 2p)(1 - p)^2 + O_n\left( d\delta ^4 \right) \right] $$

Proof

Given three distinct vertices in $i, j, k \in [n]$, let $T_{ijk}$ denote the indicator for the event that i, j and k form a triangle in G. Linearity of expectation yields that

$$\begin{aligned} {\mathbb {E}}\left[ T(G) \right] = \sum _{1 \le i< j < k \le n} {\mathbb {E}}[T_{ijk}] = \left( {\begin{array}{c}n\\ 3\end{array}}\right) \cdot {\mathbb {P}}[T_{123} = 1] \end{aligned}$$

(A.2)

where the second equality holds by symmetry. Let P and Q be as in Lemma 3.3. The principle of inclusion-exclusion now yields that

$$\begin{aligned} {\mathbb {P}}[T_{123} = 1]&= P(1, 1, 1) = \sum _{x \in \{0, 1\}^3} (-1)^{3 - |x|} \cdot Q(x_1, x_2, x_3) \\&= 1 - 3 (1 - \delta )^d(1 + \delta )^d + 3 (1 - \delta )^d(1 + \delta - \delta ^2)^d \\&\quad - (1 + 2\delta )^d (1 - \delta )^{2d} \\&= \left( 1 - (1 - \delta ^2)^d \right) ^3 + 3\left( 1 - 2 \delta ^2 + \delta ^3 \right) ^d - \left( 1 - 3\delta ^2 + 2\delta ^3 \right) ^d \\&\quad - 3(1 - \delta ^2)^{2d} + (1 - \delta ^2)^{3d} \end{aligned}$$

since $p = 1 - (1 - \delta ^2)^d$. Now observe that

$$\begin{aligned} 3\left( 1 - 2 \delta ^2 + \delta ^3 \right) ^d - 3(1 - \delta ^2)^{2d}&= 3(1 - \delta ^2)^{2d} \cdot \left[ \left( 1 + \frac{\delta ^3}{(1 - \delta ^2)(1 + \delta )} \right) ^d - 1 \right] \nonumber \\ \left( 1 - 3\delta ^2 + 2\delta ^3 \right) ^d - \left( 1 - \delta ^2 \right) ^{3d}&= (1 - \delta ^2)^{3d} \cdot \left[ \left( 1 + \frac{2\delta ^3 + \delta ^4}{(1 - \delta ^2)(1 + \delta )^2} \right) ^d - 1\right] \end{aligned}$$

(A.3)

The same bounds as in Lemma 3.3 now yield that

$$\begin{aligned} 3\left( 1 - 2 \delta ^2 + \delta ^3 \right) ^d - 3(1 - \delta ^2)^{2d}&= 3d \delta ^3 (1 - \delta ^2)^{2d} + O_n\left( d\delta ^4 (1 - \delta ^2)^{2d} \right) \\ \left( 1 - 3\delta ^2 + 2\delta ^3 \right) ^d - \left( 1 - \delta ^2 \right) ^{3d}&= 2d \delta ^3 \left( 1 - \delta ^2 \right) ^{3d} + O_n\left( d\delta ^4 \left( 1 - \delta ^2 \right) ^{3d} \right) \end{aligned}$$

Substituting $1 - p = (1 - \delta ^2)^d$ and these bounds into Eq. A.3, we have that

$$\begin{aligned} {\mathbb {P}}[T_{123} = 1]&= \left( 1 - (1 - \delta ^2)^d \right) ^3 + 3d \delta ^3 \left( 1 - \delta ^2 \right) ^{2d} - 2d \delta ^3 \left( 1 - \delta ^2 \right) ^{3d} \\&\quad + O_n\left( d\delta ^4 (1 - \delta ^2)^{2d} \right) \\&= p^3 + 3d\delta ^3 \left( 1 - \delta ^2 \right) ^{2d} - 2d\delta ^3 \left( 1 - \delta ^2 \right) ^{3d} + O_n\left( d\delta ^4 (1 - p)^2 \right) \\&= p^3 + d \delta ^3 (1 + 2p)(1 - p)^2 + O_n\left( d\delta ^4 \right) \end{aligned}$$

Substituting this into Eq. A.2 now completes the proof of the lemma. $\square$

1.2 Testing for Planted Poisson Matrices

In this section, we prove Lemma 4.1. The proof uses a similar second moment method computation of $\chi ^2$ divergence as in the proof of Lemma 3.1.

Proof of Lemma 4.1

Let $\tau = \left( {\begin{array}{c}t\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1}$. We first carry out several preliminary computations with the laws of $\text {Poisson}(\lambda )$ and $\text {Poisson}(\lambda + \tau )$ that will be useful in simplifying subsequent $\chi ^2$ divergences. Observe that the following sum has a simple closed form expression.

$$\begin{aligned} \sum _{k = 0}^\infty \frac{{\mathbb {P}}\left[ \text {Poisson}(\lambda ) = k - 1 \right] ^2}{{\mathbb {P}}\left[ \text {Poisson}(\lambda + \tau ) = k \right] }&= \lambda ^{-2} e^{-\lambda + \tau } \sum _{k = 1}^\infty \frac{k}{(k-1)!} \left( \frac{\lambda ^2}{\lambda + \tau } \right) ^k \nonumber \\&= \frac{e^{-\lambda + \tau }}{\lambda + \tau } \sum _{k = 1}^\infty \frac{1}{(k-1)!} \left( \frac{\lambda ^2}{\lambda + \tau } \right) ^{k-1} \nonumber \\&\quad + \frac{\lambda ^2 e^{-\lambda + \tau }}{(\lambda + \tau )^2} \sum _{k = 2}^\infty \frac{1}{(k-2)!} \left( \frac{\lambda ^2}{\lambda + \tau } \right) ^{k-2} \nonumber \\&= \frac{e^{-\lambda + \tau }}{\lambda + \tau } \cdot e^{\frac{\lambda ^2}{\lambda + \tau }} + \frac{\lambda ^2 e^{-\lambda + \tau }}{(\lambda + \tau )^2} \cdot e^{\frac{\lambda ^2}{\lambda + \tau }} \nonumber \\&= e^{\frac{\tau ^2}{\lambda + \tau }} \cdot \frac{\lambda ^2 + \lambda + \tau }{(\lambda + \tau )^2} \end{aligned}$$

(A.4)

The following two sums can be evaluated similarly.

$$\begin{aligned}&\sum _{k = 0}^\infty \frac{{\mathbb {P}}\left[ \text {Poisson}(\lambda ) = k - 1 \right] \cdot {\mathbb {P}}\left[ \text {Poisson}(\lambda ) = k \right] }{{\mathbb {P}}\left[ \text {Poisson}(\lambda + \tau ) = k \right] } \nonumber \\&\quad = \lambda ^{-1} e^{-\lambda + \tau } \sum _{k = 1}^\infty \frac{1}{(k-1)!} \left( \frac{\lambda ^2}{\lambda + \tau } \right) ^k \nonumber \\&\quad = \frac{\lambda }{\lambda + \tau } \cdot e^{-\lambda + \tau } \cdot e^{\frac{\lambda ^2}{\lambda + \tau }} = e^{\frac{\tau ^2}{\lambda + \tau }} \cdot \frac{\lambda }{\lambda + \tau } \end{aligned}$$

(A.5)

$$\begin{aligned}&\sum _{k = 0}^\infty \frac{{\mathbb {P}}\left[ \text {Poisson}(\lambda ) = k \right] ^2}{{\mathbb {P}}\left[ \text {Poisson}(\lambda + \tau ) = k \right] } \nonumber \\&\quad = e^{-\lambda + \tau } \sum _{k = 1}^\infty \frac{1}{k!} \left( \frac{\lambda ^2}{\lambda + \tau } \right) ^k = e^{-\lambda + \tau } \cdot e^{\frac{\lambda ^2}{\lambda + \tau }} = e^{\frac{\tau ^2}{\lambda + \tau }} \end{aligned}$$

(A.6)

Given a fixed set $S' \subseteq [n]$ of size t, let ${\textsc {poim}}_P\left( n, S', \lambda \right)$ denote the distribution of ${\textsc {poim}}_P\left( n, t, \lambda \right)$ conditioned on the event $S = S'$. If ${\mathcal {U}}_t$ denotes the uniform distribution on the size t subsets of [n], then in particular ${\textsc {poim}}_P\left( n, t, \lambda \right) =_d {\mathbb {E}}_{S \sim {\mathcal {U}}_t} {\textsc {poim}}_P\left( n, S, \lambda \right)$. Let ${\mathcal {M}}_n$ denote the set of all symmetric matrices in ${\mathbb {Z}}_{\ge 0}^{n \times n}$ with diagonal entries equal to zero and X denote an arbitrary $X \in {\mathcal {M}}_n$. Let ${\mathcal {P}}_S$, ${\mathcal {P}}$ and ${\mathcal {Q}}$ be shorthands for ${\textsc {poim}}_P\left( n, S, \lambda \right)$, ${\textsc {poim}}_P\left( n, t, \lambda \right)$ and ${\textsc {poim}}\left( n, \lambda + \tau \right)$, respectively. Observe that these are each product distributions. Following a similar second moment method computation as in Lemma 3.1, we have that

$$\begin{aligned}&1 + \chi ^2\left( {\textsc {poim}}_P\left( n, t, \lambda \right) , {\textsc {poim}}\left( n, \lambda + \tau \right) \right) \\&\quad = \sum _{X \in {\mathcal {M}}_n} \frac{{\mathbb {P}}_{{\mathcal {P}}}[X]^2}{{\mathbb {P}}_{{\mathcal {Q}}}[X]} = {\mathbb {E}}_{S, T \sim {\mathcal {U}}_t} \left[ \sum _{X \in {\mathcal {M}}_n} \frac{{\mathbb {P}}_{{\mathcal {P}}_S}[X] \cdot {\mathbb {P}}_{{\mathcal {P}}_T}[X]}{{\mathbb {P}}_{{\mathcal {Q}}}[X]} \right] \\&\quad = {\mathbb {E}}_{S, T \sim {\mathcal {U}}_t} \left[ \prod _{1 \le i < j \le n} \left( \sum _{k = 0}^\infty \frac{{\mathbb {P}}_{{\mathcal {P}}_S}[X_{ij} = k] \cdot {\mathbb {P}}_{{\mathcal {P}}_T}[X_{ij} = k]}{{\mathbb {P}}_{{\mathcal {Q}}}[X_{ij} = k]} \right) \right] \end{aligned}$$

The second equality holds by linearity of expectation and because S and T are independent. The marginal distributions of ${\mathcal {P}}_S, {\mathcal {P}}_T$ and ${\mathcal {Q}}$ combined with Eqs. A.4, A.5 and A.6 now imply that

$$\begin{aligned}&1 + \chi ^2\left( {\textsc {poim}}_P\left( n, t, \lambda \right) , {\textsc {poim}}\left( n, \lambda + \tau \right) \right) \nonumber \\&\quad = {\mathbb {E}}_{S, T \sim {\mathcal {U}}_t} \nonumber \\&\qquad \left[ \left( e^{\frac{\tau ^2}{\lambda + \tau }} \cdot \frac{\lambda ^2 + \lambda + \tau }{(\lambda + \tau )^2} \right) ^{\left( {\begin{array}{c}|S \cap T|\\ 2\end{array}}\right) } \left( e^{\frac{\tau ^2}{\lambda + \tau }} \cdot \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) - \left( {\begin{array}{c}|S \cap T|\\ 2\end{array}}\right) } \right. \nonumber \\&\qquad \quad \left. \left( e^{\frac{\tau ^2}{\lambda + \tau }} \right) ^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) - 2 \left( {\begin{array}{c}t\\ 2\end{array}}\right) + \left( {\begin{array}{c}|S \cap T|\\ 2\end{array}}\right) } \right] \nonumber \\&\quad = e^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \cdot \frac{\tau ^2}{\lambda + \tau }} \cdot {\mathbb {E}}_{S, T \sim {\mathcal {U}}_t} \left[ \left( \frac{\lambda ^2 + \lambda + \tau }{(\lambda + \tau )^2} \right) ^{\left( {\begin{array}{c}|S \cap T|\\ 2\end{array}}\right) } \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) - \left( {\begin{array}{c}|S \cap T|\\ 2\end{array}}\right) } \right] \nonumber \\&\quad = e^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \cdot \frac{\tau ^2}{\lambda + \tau }} \cdot \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \cdot {\mathbb {E}}_{S, T \sim {\mathcal {U}}_t} \left[ \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) ^{\left( {\begin{array}{c}|S \cap T|\\ 2\end{array}}\right) } \right] \end{aligned}$$

(A.7)

Now fix two subsets $S, T \subseteq [n]$ of size t and note that $|S \cap T| \le t = O_n(1)$. Note that $e^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \cdot \frac{\tau ^2}{\lambda + \tau }} = e^{\left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau }} \le e^{\left( {\begin{array}{c}t\\ 2\end{array}}\right) } = O_n(1)$. As in Lemma 3.1, $|S \cap T|$ is distributed as $\text {Hypergeometric}(n, t, t)$ since $S, T \sim {\mathcal {U}}_t$ are independent. Furthermore, ${\mathbb {P}}[|S \cap T| = k] = \left( {\begin{array}{c}t\\ k\end{array}}\right) \left( {\begin{array}{c}n - t\\ t - k\end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} = O_n(n^{-k})$. Observe that

$$\begin{aligned}&\sum _{k = 3}^t {\mathbb {P}}[|S \cap T| = k] \cdot e^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \cdot \frac{\tau ^2}{\lambda + \tau }} \cdot \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \cdot \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) ^{\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \nonumber \\&\quad \le \sum _{k = 3}^t e^{\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \cdot \left( {\begin{array}{c}t\\ k\end{array}}\right) \left( {\begin{array}{c}n - t\\ t - k\end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) ^{\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \nonumber \\&\quad = O_n \left( \max _{2 < k \le t} n^{-k} \cdot \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) ^{\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \right) \end{aligned}$$

(A.8)

since $t = O_n(1)$. Also observe that

$$\begin{aligned}&\sum _{k = 0}^2 {\mathbb {P}}[|S \cap T| = k] \cdot e^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \cdot \frac{\tau ^2}{\lambda + \tau }} \cdot \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \cdot \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) ^{\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \\&\quad = e^{\left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau }} \cdot \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \\&\qquad \cdot \left[ \left( {\begin{array}{c}n - t\\ t\end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} + t \left( {\begin{array}{c}n - t\\ t - 1\end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} + \left( {\begin{array}{c}t\\ 2\end{array}}\right) \left( {\begin{array}{c}n - t\\ t - 2\end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) \right] \end{aligned}$$

Using the fact that $\sum _{\ell = 0}^t \left( {\begin{array}{c}t\\ \ell \end{array}}\right) \left( {\begin{array}{c}n - t\\ t - \ell \end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} = 1$, this quantity simplifies to

$$\begin{aligned}&e^{\left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau }} \cdot \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \nonumber \\&\qquad \cdot \left[ 1 + \left( {\begin{array}{c}t\\ 2\end{array}}\right) \left( {\begin{array}{c}n - t\\ t - 2\end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } - 1 \right) - \sum _{\ell = 3}^t \left( {\begin{array}{c}t\\ \ell \end{array}}\right) \left( {\begin{array}{c}n - t\\ t - \ell \end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} \right] \nonumber \\&\quad = e^{\left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau }} \cdot \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \nonumber \\&\qquad \cdot \left[ 1 + \left( {\begin{array}{c}t\\ 2\end{array}}\right) ^2 \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } - 1 \right) + O_n\left( (1 + \lambda ^{-1})n^{-3}\right) \right] \end{aligned}$$

(A.9)

The equality above follows from: (1) $\left( {\begin{array}{c}n - t\\ t - 2\end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} = \left( {\begin{array}{c}t\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} + O_n(n^{-3})$, as established in Eq. 3.9; (2) from the fact that $\frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \le 1 + \lambda ^{-1}$; and (3) from $\left( {\begin{array}{c}t\\ \ell \end{array}}\right) \left( {\begin{array}{c}n - t\\ t - \ell \end{array}}\right) \left( {\begin{array}{c}n\\ t\end{array}}\right) ^{-1} = O_n(n^{-3})$ for each $3 \le \ell \le t$ and the fact that the sum contains $t - 2 = O_n(1)$ terms. Note that $\lambda = \omega _n(n^{-2})$ and thus $\frac{\tau }{\lambda + \tau } = o_n(1)$. Since $2\left( {\begin{array}{c}t\\ 2\end{array}}\right) = O_n(1)$, we have by Taylor expanding that

$$\begin{aligned} e^{\left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau }} \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) }&= e^{\left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau }} \left( 1 - \frac{\tau }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \nonumber \\&= \left[ 1 + \left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau } + O_n\left( \frac{\tau ^2}{(\lambda + \tau )^2} \right) \right] \nonumber \\&\quad \left[ 1 - 2 \left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau } + O_n\left( \frac{\tau ^2}{(\lambda + \tau )^2} \right) \right] \nonumber \\&= 1 - \left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau } + O_n\left( \frac{\tau ^2}{(\lambda + \tau )^2} \right) \nonumber \\&= 1 - \left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau } + O_n\left( \lambda ^{-2} n^{-4} \right) \end{aligned}$$

(A.10)

Substituting $\tau = \left( {\begin{array}{c}t\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1}$ and the estimate in Eq. A.10 into Eq. A.9 yields that

$$\begin{aligned}&\sum _{k = 0}^2 {\mathbb {P}}[|S \cap T| = k] \cdot e^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \cdot \frac{\tau ^2}{\lambda + \tau }} \cdot \left( \frac{\lambda }{\lambda + \tau } \right) ^{2\left( {\begin{array}{c}t\\ 2\end{array}}\right) } \cdot \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) ^{\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \nonumber \\&\quad = \left[ 1 - \left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau } + O_n\left( \lambda ^{-2} n^{-4} \right) \right] \nonumber \\&\qquad \left[ 1 + \left( {\begin{array}{c}t\\ 2\end{array}}\right) \tau \cdot \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } - 1 \right) + O_n\left( (1 + \lambda ^{-1})n^{-3}\right) \right] \nonumber \\&\quad = 1 - \left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau }{\lambda + \tau } + \left( {\begin{array}{c}t\\ 2\end{array}}\right) \tau \cdot \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } - 1 \right) \nonumber \\&\qquad + O_n\left( \lambda ^{-2} n^{-4} \right) + O_n\left( (1 + \lambda ^{-1})n^{-3}\right) \nonumber \\&\quad = 1 + \left( {\begin{array}{c}t\\ 2\end{array}}\right) \cdot \frac{\tau ^2(1 - \lambda )}{\lambda ^2 + \lambda \tau } + O_n\left( \lambda ^{-2} n^{-4} \right) + O_n\left( (1 + \lambda ^{-1})n^{-3}\right) \nonumber \\&\quad = 1 + O_n\left( (1 + \lambda ^{-2}) n^{-4} \right) + O_n\left( (1 + \lambda ^{-1})n^{-3}\right) \end{aligned}$$

(A.11)

Substituting Eqs. A.11 and A.8 into Eq. A.7 now yields that

$$\begin{aligned} \chi ^2\left( {\textsc {poim}}_P\left( n, t, \lambda \right) , {\textsc {poim}}\left( n, \lambda + \tau \right) \right)&= O_n\left( (1 + \lambda ^{-2}) n^{-4} \right) + O_n\left( (1 + \lambda ^{-1})n^{-3}\right) \\&\quad + O_n \left( \max _{2 < k \le t} n^{-k} \cdot \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) ^{\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \right) \end{aligned}$$

Observe that $\frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } = \frac{\lambda }{\lambda + \tau } + \lambda ^{-1} \le 1 + \lambda ^{-1}$ and, when $k = 3$ in the third term above, the bound is $n^{-3} \cdot \left( \frac{\lambda ^2 + \lambda + \tau }{\lambda ^2 + \lambda \tau } \right) ^3 = \Omega _n\left( (1 + \lambda ^{-1})n^{-3}\right)$. Now applying Cauchy–Schwarz as in Lemma 3.1 completes the proof of the lemma. $\square$

1.3 Total Variation Convergence of RIM and POIM

In this section, we complete the proof of Theorem 2.3. We first deduce the following elementary upper bound on the total variation between two univariate Poisson distributions using some of the calculations in Lemma 4.1.

Lemma A.2

If $\lambda _1 \ge \lambda _2 > 0$, then it follows that

$$ d_{\mathrm{TV}}\left( \text {Poisson}(\lambda _1), \text {Poisson}(\lambda _2) \right) \le \sqrt{\frac{1}{2} \left( e^{\lambda _1^{-1} (\lambda _1 - \lambda _2)^2} - 1 \right) } $$

which is $O\left( \lambda _1^{-1} (\lambda _1 - \lambda _2)^2 \right)$ if $(\lambda _1 - \lambda _2)^2 \le \lambda _1$.

Proof

By the same computation in Eq. A.6, we have that

$$\begin{aligned} 1 + \chi ^2\left( \text {Poisson}(\lambda _2), \text {Poisson}(\lambda _1) \right)= & {} \sum _{k = 0}^\infty \frac{{\mathbb {P}}\left[ \text {Poisson}(\lambda _2) = k \right] ^2}{{\mathbb {P}}\left[ \text {Poisson}(\lambda _1) = k \right] } \\= & {} e^{\lambda _1 - 2\lambda _2} \sum _{k = 0}^\infty \frac{1}{k!} \left( \frac{\lambda _2^2}{\lambda _1} \right) ^k = e^{\lambda _1^{-1} (\lambda _1 - \lambda _2)^2} \end{aligned}$$

Applying Cauchy–Schwarz to obtain $d_{\mathrm{TV}}\le \sqrt{\frac{1}{2} \cdot \chi ^2}$ now proves the lemma. $\square$

Observe that $\lambda _1 \ge \frac{1}{4} \left( \sqrt{\lambda _1} + \sqrt{\lambda _2} \right) ^2$, from which we obtain that

$$ d_{\mathrm{TV}}\left( \text {Poisson}(\lambda _1), \text {Poisson}(\lambda _2) \right) \le \sqrt{\frac{1}{2} \left( e^{4(\sqrt{\lambda _1} - \sqrt{\lambda _2})^2} - 1 \right) } $$

This implies that if $|\lambda _1 - \lambda _2| = o(1)$, then $d_{\mathrm{TV}}\left( \text {Poisson}(\lambda _1), \text {Poisson}(\lambda _2) \right) = o(1)$. Applying the triangle inequality now yields that $d_{\mathrm{TV}}\left( \text {Poisson}(\lambda _1), \text {Poisson}(\lambda _2) \right) = o(1)$ if $\lambda _2 = \lambda _2' + o(1)$ and $(\lambda _1 - \lambda _2')^2 \ll \lambda _1$. We will use this fact in the proof of Theorem 2.3.

We now will prove Theorem 2.3, referencing parts of the proof of Theorem 3.1 where details are identical or similar.

Proof of Theorem 2.3

First observe that $n\delta \ll n^{1/2} d^{-1/3} \ll 1$. We first summarize several observations and definitions from Theorem 3.1 as they apply to random intersection matrices.

• Let $p_k = {\mathbb {P}}[| \{ j : i \in S_j\}| = k] = \left( {\begin{array}{c}n\\ k\end{array}}\right) \delta ^k (1 - \delta )^{n - k}$ be the probability some $i \in [d]$ is in k sets $S_j$ and let $M_k$ be the number of $i \in [d]$ in exactly k sets $S_k$. Note that $(M_0, M_1, \dots , M_n) \sim \text {Multinomial}(d, p_0, p_1, \dots , p_n)$.

• A random matrix $X \sim {\textsc {rim}}(n, d, \delta )$ can now be generated through the procedure ${\mathcal {P}}_{\text {gen}}$ by first setting all entries of X to be zero, sampling $(M_0, M_1, \dots , M_n) \sim \text {Multinomial}(d, p_0, p_1, \dots , p_n)$ and then, for each $2 \le k \le n$, independently sampling a subset S of size k from [n] uniformly at a random a total of $M_k$ times and increasing $X_{ij}$ by 1 for each $i, j \in S$.

• Let ${\mathcal {L}}_P$ denote the distribution on $(M_0, M_1, \dots , M_n)$ where the $M_k$ are mutually independent and $M_k \sim \text {Poisson}(dp_k)$. Let ${\textsc {rim}}_P(n, d, \delta )$ be the distribution on matrices X generated through ${\mathcal {P}}_{\text {gen}}$, generating $(M_0, M_1, \dots , M_n) \sim {\mathcal {L}}_P$ instead of from a multinomial distribution.

• Poisson splitting implies that, after sampling $(M_0, M_1, M_2)$ from ${\mathcal {L}}_P$ and applying ${\mathcal {P}}_{\text {gen}}$ for $k = 2$, the resulting matrix $X_2$ is distributed as ${\textsc {poim}}\left( n, \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} dp_2 \right)$.

• Let ${\textsc {rim}}_P(n, d, \delta , m_3, m_4, \dots , m_K)$ denote ${\textsc {rim}}_P(n, d, \delta )$ conditioned on the event that $M_k = m_k$ for $3 \le k \le K$ and $M_k = 0$ for $K < k \le n$. Note that $X \sim {\textsc {rim}}_P(n, d, \delta , m_3, m_4, \dots , m_K)$ is distributed as ${\textsc {poim}}\left( n, \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} dp_2 \right)$ with $m_k$ random planted increased subsets of size k for each $3 \le k \le K$ as in ${\mathcal {P}}_{\text {gen}}$.

The argument in Proposition 3.1 implies that if $n \delta \ll 1$, then

$$\begin{aligned} d_{\mathrm{TV}}\left( {\textsc {rim}}(n, d, \delta ), {\textsc {rim}}_P(n, d, \delta ) \right) = O_n \left( n^2 \delta ^2 \right) \end{aligned}$$

(A.12)

as $n \rightarrow \infty$. Now let

$$\begin{aligned} E_t= & {} \min \left\{ 1, C_t \left( \left( 1 +d^{-1}\delta ^{-2}(1 - \delta )^{2-n} \right) n^{-2} \right. \right. \\&\qquad \left. \left. + \max _{2 < k \le t} n^{-k/2} \left( 1 + d^{-1}\delta ^{-2}(1 - \delta )^{2-n} \right) ^{\frac{1}{2} \left( {\begin{array}{c}k\\ 2\end{array}}\right) } \right) \right\} \end{aligned}$$

for a sufficiently large constant $C_t > 0$ so that $E_t$ is an upper bound in Lemma 4.1 when $\lambda = \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} dp_2 = d\delta ^2(1 - \delta )^{n - 2}$. As observed above, we have that X after the step of ${\mathcal {P}}_{\text {gen}}$ with $k = 2$ is distributed as ${\textsc {rim}}_P(n, d, \delta , 0, 0, \dots , 0) \sim {\textsc {poim}}\left( n, d\delta ^2(1 - \delta )^{n - 2} \right)$. The same induction as in Proposition 3.2 yields that

$$\begin{aligned}&d_{\mathrm{TV}}\left( {\textsc {rim}}_P(n, d, \delta , m_3, m_4, \dots , m_K), {\textsc {poim}}\left( n, \lambda (n, d, \delta , m_3, m_4, \dots , m_K)\right) \right) \nonumber \\&\quad \le \sum _{t = 3}^K m_t E_t \end{aligned}$$

(A.13)

for each $K \ge 1$ and $(m_3, m_4, \dots , m_K) \in {\mathbb {Z}}_{\ge 0}^K$, where $\lambda (n, d, \delta , m_3, m_4, \dots , m_K)$ is given by

$$ \lambda (n, d, \delta , m_3, m_4, \dots , m_K) = d\delta ^2(1 - \delta )^{n - 2} + \sum _{t = 3}^K m_t \left( {\begin{array}{c}t\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} $$

We now apply the bounding argument from the end of Proposition 3.2 and the conditioning argument in the beginning of Theorem 3.1 to reduce the proof to comparing a ${\textsc {poim}}$ to a mixture of ${\textsc {poim}}$ distributions. Fix some function $w = w(n) \rightarrow \infty$ as $n \rightarrow \infty$ such that $d \gg w^2 n^3$ and $w\delta \ll d^{-1/3} n^{-1/2}$. Let E be the event that $(M_3, M_4, \dots , M_n) \sim {\mathcal {L}}_P$ satisfy all of the following inequalities

$$\begin{aligned} dp_k - \sqrt{wdp_k} \le M_k \le dp_k + \sqrt{wdp_k} \quad&\text {for } k \ge 3 \text { with } dp_k > w^{-1/2} \\ M_k = 0 \quad&\text {for } k \ge 3 \text { with } dp_k \le w^{-1/2} \end{aligned}$$

Now note that if $k \ge 6$, since $w\delta \ll d^{-1/3} n^{-1/2}$ and $d \gg n^3$, it follows that

$$ dp_k = d\left( {\begin{array}{c}n\\ k\end{array}}\right) \delta ^k (1 - \delta )^{n - k} \le dn^k \delta ^k \ll \frac{n^{k/2}}{w^6 d^{k/3 - 1}} = o_n\left( w^{-1} \right) $$

Repeating the concentration inequalities and bounds used to establish Eq. 3.15, we have that

$$\begin{aligned} {\mathbb {P}}_{{\mathcal {L}}_P}\left[ E^c \right]&\lesssim 3w^{-1} + 3w^{-1/2} + \sum _{k = 6}^n dp_k \\&\le 3w^{-1} + 3w^{-1/2} + \sum _{k = 6}^n dn^k \delta ^k \\&= 3w^{-1} + 3w^{-1/2} + \frac{dn^6 \delta ^6}{1 - n\delta } = o_n(1) \end{aligned}$$

We now bound $wdp_k E_k$ for $3 \le k \le 5$ in a similar way to Proposition 3.2. First consider the case where $d\delta ^2 \ge 1$. Note that since $n \delta \ll 1$, it follows that $(1 - \delta )^{n - 2} \ge 1 - (n - 2) \delta = 1 - o_n(1)$. Therefore it follows that $d^{-1} \delta ^{-2} (1 - \delta )^{2 - n} = O_n(1)$ and hence $E_k = O_n(n^{-3/2})$ for each $3 \le k \le 5$. Therefore since $n \delta \ll 1$, we have that

$$ wdp_k E_k \le wd \cdot (n \delta )^k \cdot n^{-3/2} \lesssim wd \cdot (n \delta )^3 \cdot n^{-3/2} = o_n(w^{-2}) $$

for each $3 \le k \le 5$, since $w\delta \ll d^{-1/3} n^{-1/2}$. Now consider the case where $d\delta ^2 < 1$ and let $\delta = \gamma /\sqrt{d}$ where $\gamma < 1$. It follows that $1 + d^{-1} \delta ^{-2} (1 - \delta )^{2 - n} = O_n(\gamma ^{-2})$ and thus for $3 \le t \le 5$, we have that

$$\begin{aligned} wdp_t E_t\lesssim & {} w \cdot \min \left\{ d n^t \delta ^t, \sum _{k = 3}^t d n^t \delta ^t \cdot n^{-k/2} \gamma ^{-\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \right\} \\= & {} w \cdot \min \left\{ d^{1 - t/2} n^t \gamma ^t, \sum _{k = 3}^t d^{1 - t/2} n^{t-k/2} \gamma ^{t -\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \right\} \end{aligned}$$

Since $\frac{2}{t - 1} \in (0, 1]$ if $3 \le t \le 5$, we have that

$$\begin{aligned} wdp_t E_t&\lesssim w \cdot \sum _{k = 3}^t \left( d^{1 - t/2} n^t \gamma ^t \right) ^{\frac{t - 3}{t - 1}} \left( d^{1 - t/2} n^{t-k/2} \gamma ^{t -\left( {\begin{array}{c}k\\ 2\end{array}}\right) } \right) ^{\frac{2}{t - 1}} \\&= w \cdot \sum _{k = 3}^t d^{1 - t/2} n^{t - \frac{k}{t - 1}} \gamma ^{t - \frac{2}{t - 1} \cdot \left( {\begin{array}{c}k\\ 2\end{array}}\right) } \\&\le w \cdot \sum _{k = 3}^t d^{1 - t/2} n^{t - \frac{k}{t - 1}} \end{aligned}$$

where the last inequality follows from $\gamma < 1$ and $t - \frac{2}{t - 1} \cdot \left( {\begin{array}{c}k\\ 2\end{array}}\right) \ge 0$ if $k \le t$. Hence,

$$\begin{aligned} wdp_3 E_3&\lesssim w d^{-1/2} n^{3/2} = o_n(1) \\ wdp_4 E_4&\lesssim w d^{-1} n^{3} + w d^{-1} n^{8/3} = o_n(w^{-1}) \\ wdp_5 E_5&\lesssim wd^{-3/2} n^{17/4} + w d^{-3/2} n^{4} + w d^{-3/2} n^{15/4} = o_n(w^{-3/2}) \end{aligned}$$

since $d \gg w^2 n^3$. In summary, $wdp_k E_k = o_n(1)$ for each $3 \le k \le 5$.

Now let ${\textsc {rim}}_E(n, d, \delta )$ and ${\mathcal {L}}_E$ denote the distributions of ${\textsc {rim}}(n, d, \delta )$ and ${\mathcal {L}}_P$ conditioned on the event E holding. Note that if E holds, then it follows that $M_k \le dp_k + \sqrt{wdp_k} = O_n(wdp_k)$ for each $3 \le k \le 5$ with $M_k \ne 0$ and $M_k = 0$ for all other $k \ge 3$. Combining Eq. A.13, the conditioning property of total variation, the triangle inequality and $wdp_k E_k = o_n(1)$ for $3 \le k \le 5$ yields that

$$\begin{aligned}&d_{\mathrm{TV}}\left( {\textsc {rim}}_P(n, d, \delta ), {\mathbb {E}}_{(m_3, m_4, m_5) \sim {\mathcal {L}}_E} \, {\textsc {poim}}\left( n, \lambda (n, d, \delta , m_3, m_4, m_5) \right) \right) \\&\quad \le {\mathbb {P}}\left[ E^c \right] + d_{\mathrm{TV}}\\&\qquad \left( {\textsc {rim}}_E(n, d, \delta ), {\mathbb {E}}_{(m_3, m_4, m_5) \sim {\mathcal {L}}_E} \, {\textsc {poim}}\left( n, \lambda (n, d, \delta , m_3, m_4, m_5) \right) \right) \\&\quad \le {\mathbb {P}}\left[ E^c \right] + \sup _{(m_3, m_4, m_5) \in \text {supp}({\mathcal {L}}_E)} d_{\mathrm{TV}}\\&\qquad \left( {\textsc {rim}}_E(n, d, \delta , m_3, m_4, m_5), {\textsc {poim}}\left( n, \lambda (n, d, \delta , m_3, m_4, m_5) \right) \right) \\&\quad \le {\mathbb {P}}\left[ E^c \right] + \sup _{(m_3, m_4, m_5) \in \text {supp}({\mathcal {L}}_E)} \sum _{k = 3}^5 m_k E_k \\&\quad \lesssim {\mathbb {P}}\left[ E^c \right] + \sum _{k = 3}^5 wdp_k E_k = o_n(1) \end{aligned}$$

The triangle inequality and Eq. A.12 now imply that it suffices to show

$$\begin{aligned}&d_{\mathrm{TV}}\left( {\textsc {poim}}(n, d\delta ^2), {\mathbb {E}}_{(m_3, m_4, m_5) \sim {\mathcal {L}}_E} \, {\textsc {poim}}\left( n, \lambda (n, d, \delta , m_3, m_4, m_5) \right) \right) \nonumber \\&\quad = o_n(1) \end{aligned}$$

(A.14)

Now consider a matrix X sampled from either ${\mathbb {E}}_{(m_3, m_4, m_5) \sim {\mathcal {L}}_E} \, {\textsc {poim}}(n, \lambda (n, d, \delta , m_3, m_4, m_5) )$ or ${\textsc {poim}}(n, d\delta ^2)$. Conditioned on the event $s = \sum _{1 \le i < j \le n} X_{ij}$, the entries $(X_{ij} : 1 \le i < j \le n)$ are distributed according to $\text {Multinomial}\left( s, \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} \right)$ under either distribution, by Poisson splitting. To show Eq. A.14, the conditioning property of total variation thus implies that it suffices to bound the total variation between $\sum _{1 \le i < j \le n} X_{ij}$ under the two distributions. In other words, it suffices to show the following total variation bound

$$\begin{aligned}&d_{\mathrm{TV}}\left( \text {Poisson}\left( \left( {\begin{array}{c}n\\ 2\end{array}}\right) d\delta ^2 \right) , {\mathbb {E}}_{(m_3, m_4, m_5) \sim {\mathcal {L}}_E} \, \text {Poisson}\left( \left( {\begin{array}{c}n\\ 2\end{array}}\right) \lambda (n, d, \delta , m_3, m_4, m_5) \right) \right) \nonumber \\&\quad = o_n(1) \end{aligned}$$

(A.15)

As in the proof of Theorem 3.1, let $A \subseteq \{3, 4, 5\}$ be the set of indices k such that $dp_k > w^{-1/2}$ and define

$$\begin{aligned}&\lambda _1 = d\delta ^2(1 - \delta )^{n - 2} + \sum _{k \in A} dp_k \left( {\begin{array}{c}k\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} \quad \text {and} \\&\lambda _2 = d\delta ^2(1 - \delta )^{n - 2} + \sum _{k = 3}^5 dp_k \left( {\begin{array}{c}k\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} \end{aligned}$$

Observe that

$$ \left| \lambda _1 - \lambda _2 \right| = \sum _{k \in A^c \cap \{3, 4, 5\}} dp_k \left( {\begin{array}{c}k\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} \le 3w^{-1/2} n^{-2} $$

Also note that

$$\begin{aligned} \lambda _2&= d\delta ^2(1 - \delta )^{n - 2} + \sum _{k = 3}^5 d\left( {\begin{array}{c}n\\ k\end{array}}\right) \left( {\begin{array}{c}k\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} \delta ^k (1 - \delta )^{n - k} \\&= d\delta ^2 \left[ (1 - \delta )^{n - 2} + \sum _{k = 3}^5 \left( {\begin{array}{c}n - 2\\ k - 2\end{array}}\right) \delta ^{k - 2} (1 - \delta )^{n - k - 2} \right] \\&= d\delta ^2 \left[ 1 - \sum _{\ell = 4}^{n - 2} \left( {\begin{array}{c}n - 2\\ \ell \end{array}}\right) \delta ^{\ell } (1 - \delta )^{n - 2 - \ell } \right] \end{aligned}$$

Since $n \delta \ll 1$ and $\delta \ll w^{-1} d^{-1/3} n^{-1/2}$, it therefore follows that

$$ \left| d\delta ^2 - \lambda _2 \right| = d\delta ^2 \sum _{\ell = 4}^{n - 2} \left( {\begin{array}{c}n - 2\\ \ell \end{array}}\right) \delta ^{\ell } (1 - \delta )^{n - 2 - \ell } \le \sum _{\ell = 4}^\infty n^\ell \delta ^\ell \lesssim dn^4 \delta ^6 \ll \frac{n}{w^6 d} $$

Finally note that if $(m_3, m_4, m_5) \in \text {supp}({\mathcal {L}}_E)$, then the triangle inequality yields that

$$\begin{aligned} \left| \lambda _1 - \lambda (n, d, \delta , m_3, m_4, m_5) \right|&\le \sum _{k \in A} |m_k - dp_k| \cdot \left( {\begin{array}{c}k\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} \\&\le \sum _{k = 3}^5 \sqrt{wdp_k} \cdot \left( {\begin{array}{c}k\\ 2\end{array}}\right) \left( {\begin{array}{c}n\\ 2\end{array}}\right) ^{-1} \\&\lesssim \sum _{k = 3}^5 w^{1/2} d^{1/2} n^{k/2 - 2} \delta ^{k/2} \lesssim w^{1/2} d^{1/2} n^{-1/2} \delta ^{3/2} \end{aligned}$$

since $n \delta \ll 1$. The triangle inequality now yields that

$$\begin{aligned} \left| \left( {\begin{array}{c}n\\ 2\end{array}}\right) d\delta ^2 - \left( {\begin{array}{c}n\\ 2\end{array}}\right) \lambda (n, d, \delta , m_3, m_4, m_5) \right|&\lesssim w^{-1/2} + \frac{n^3}{w^6 d} + w^{1/2} d^{1/2} n^{3/2} \delta ^{3/2} \\&= o_n(1) + w^{1/2} d^{1/2} n^{3/2} \delta ^{3/2} \end{aligned}$$

Furthermore note that

$$ \frac{\left( w^{1/2} d^{1/2} n^{3/2} \delta ^{3/2} \right) ^2}{\left( {\begin{array}{c}n\\ 2\end{array}}\right) d\delta ^2} \lesssim w n \delta \ll 1 $$

Thus by the earlier remark on total variation distances between Poisson distributions, it follows that

$$ d_{\mathrm{TV}}\left( \text {Poisson}\left( \left( {\begin{array}{c}n\\ 2\end{array}}\right) d\delta ^2 \right) , \text {Poisson}\left( \left( {\begin{array}{c}n\\ 2\end{array}}\right) \lambda (n, d, \delta , m_3, m_4, m_5) \right) \right) = o_n(1) $$

for any $(m_3, m_4, m_5) \in \text {supp}({\mathcal {L}}_E)$. The conditioning property of total variation then implies Eq. A.15, completing the proof of the theorem. $\square$

Appendix: Random Geometric Graphs on ${\mathbb {S}}^{d - 1}$

1.1 Estimates for $\psi _d$

In this section, we prove Lemma 5.1 which gives key estimates for quantities in terms of $\psi _d$ and $t_{p, d}$ in our analysis of random geometric graphs.

Proof of Lemma 5.1

As mentioned previously, first item is shown in Section 2 of [55] and the second item is Lemma 2 in Section 2 of [8]. We now prove the remaining three items.

• 3. From the first item in this lemma, we have that

  $$\begin{aligned} \frac{\psi _d(t -\delta )}{\psi _d(t)}= & {} \left( \frac{1-(t-\delta )^2}{1-t^2}\right) ^{\frac{d-3}{2}} = \left( 1 + \frac{2t\delta - \delta ^2}{1-t^2}\right) ^{\frac{d-3}{2}} \\\le & {} \left( 1+ \frac{8t\delta }{3}\right) ^{\frac{d-3}{2}} \le e^{2td\delta } \end{aligned}$$

• 4. Let $\delta _1 = \min \left\{ \tfrac{1}{\sqrt{d}},\tfrac{1}{dt_{p,d}}\right\}$. Since $\psi _d$ is decreasing, we have that

  $$\begin{aligned} p= & {} \int _{t_{p,d}}^{1} \psi _d(x)dx \ge \int _{t_{p,d} }^{t_{p,d}+\delta _1}\psi _d(x)dx \ge \delta _1 \psi _d(t_{p,d} + \delta _1) \\\ge & {} \delta _1 \psi _d(t_{p,d})e^{-2d(t_{p,d}+\delta _1)\delta _1} \end{aligned}$$

  Note $2d(t_{p,d}+\delta _1)\delta _1 \le C$ for some universal constant $C > 0$, from which the result follows.

• 5. Since $\psi _d$ is symmetric, we have that ${\mathbb {P}}\left( |T| \ge t \right) = 2\Psi _d(t)$. Combining the facts that $\Psi _d(t_{p,d}) = p$, there is a constant $C > 0$ such that $t_{p,d} \le C\sqrt{\frac{\log p^{-1}}{d}}$ and the fact that $\Psi _d$ is a decreasing function, we now have $\Psi _d\left( C\sqrt{\frac{\log p^{-1}}{d}}\right) \le p$. Taking $t = C\sqrt{\frac{\log p^{-1}}{d}}$ in ${\mathbb {P}}\left( |T| \ge t \right) = 2\Psi _d(t)$, the result follows.

This completes the proof of the lemma. $\square$

1.2 Deferred Proofs from the Coupling Argument

In this section, we prove Lemmas 5.3, 5.4, 5.5 and 5.6 deferred from our coupling argument analysis of random geometric graphs on ${\mathbb {S}}^{d - 1}$.

Proof of Lemma 5.3

We prove the two items of the lemma separately.

• 1. We will show this item in the case where $a = (1, 0, 0, \dots , 0)$. The statement for any other unit vector $a \in {\mathbb {S}}^{d - 1}$ will follow after applying a rotation to the $a = (1, 0, 0, \dots , 0)$ case. The isotropy of the d-dimensional Gaussian distribution implies that a random vector $W \sim {\mathsf {unif}}({\mathbb {S}}^{d-1})$ can be generated as $W = Z/\Vert Z\Vert _2$ where $Z = (Z_1, Z_2, \dots , Z_d) \sim {\mathcal {N}}(0, I_d)$. Now let $Z_{\sim 1} = (0, Z_2, Z_3, \dots , Z_d)$ and note that

  $$ W = \frac{Z_1}{\Vert Z\Vert _2} \cdot a + \sqrt{1 - \frac{Z_1^2}{\Vert Z\Vert _2^2}} \cdot \frac{Z_{\sim 1}}{\Vert Z_{\sim 1}\Vert _2} $$

  Note that $Z_{\sim 1} \sim {\mathcal {N}}(0, I_{d-1})$ by definition. The rotational invariance of ${\mathcal {N}}(0, I_{d-1})$ implies that $Z_{\sim 1}/\Vert Z_{\sim 1}\Vert _2$ and $\Vert Z_{\sim 1}\Vert _2$ are independent. Now note that $Z_1/\Vert Z\Vert _2 = Z_1/\sqrt{Z_1^2 + \Vert Z_{\sim 1}\Vert _2^2}$ is in the $\sigma$-algebra $\sigma (Z_1, \Vert Z_{\sim 1}\Vert _2)$ and thus independent of $Z_{\sim 1}/\Vert Z_{\sim 1}\Vert _2$. Furthermore, by definition we have that $Z_1/\Vert Z\Vert _2 = W_1 \sim \psi _d$ and the isotropy of ${\mathcal {N}}(0, I_{d-1})$ implies that $Z_{\sim 1}/\Vert Z_{\sim 1}\Vert _2 \sim {\mathsf {unif}}({\mathbb {S}}^{a^\perp })$. This implies that W is equal in distribution to

  $$ W =_d Ta + \sqrt{1 - T^2} \cdot Y $$

  where $T \sim \psi _d$, $Y \sim {\mathsf {unif}}({\mathbb {S}}^{a^\perp })$ and T and Y are independent. This proves the if direction of the item of the lemma. We now prove the only if direction. If $X = Ta + \sqrt{1 - T^2} \cdot Y$ is uniformly distributed on ${\mathbb {S}}^{d - 1}$, then it can be coupled to $(Z_1, Z_2, \dots , Z_d) \sim {\mathcal {N}}(0, I_d)$ so that $X = Z/\Vert Z\Vert _2$. Now note that T and Y are deterministic functions of X with $T = X_1 = Z_1/\Vert Z\Vert _2$ and $Y = X_{\sim 1}/\Vert X_{\sim 1}\Vert _2 = Z_{\sim 1}/\Vert Z_{\sim 1}\Vert _2$. The discussion above now shows that (T, Y) satisfy the three desired conditions.

• 2. Note that $Z_{1}, Z_2 \dots ,Z_{m}$ can be completed to an orthonormal basis $Z_1, Z_2, \dots , Z_d$. Fix a procedure to do this as a deterministic function of $Z_1, Z_2,\dots ,Z_m$. Let $\alpha _i = \langle X, Z_i\rangle$ and note that $X = \sum _{i=1}^{d} \alpha _i Z_i$. Now consider conditioning on $Z_1,\dots , Z_m$. Given this conditioning, we have that X is uniformly distributed on ${\mathbb {S}}^{d-1}$ and, by rotational invariance, also that $(\alpha _1,\dots ,\alpha _d) \sim {\mathsf {unif}}({\mathbb {S}}^{d-1})$. The result now follows by repeatedly applying the first item of this lemma with the last $d - m$ coordinates of $(\alpha _1,\dots ,\alpha _d)$ as the choices of a.

This completes the proof of the lemma. $\square$

Proof of Lemma 5.4

We again proceed item by item.

• 1. Let $\xi = (a_{23}, a_{24}, \dots ,a_{2n}) \in {\mathbb {R}}^{n-2}$ and let ${\hat{\xi }} = \xi /\Vert \xi \Vert _2$. By item 2 in Lemma 5.3, we have that ${\hat{\xi }}$ is uniformly distributed over ${\mathbb {S}}^{n-3}$. Similarly, let $\zeta = (T_3, T_4 \dots , T_n)$ and ${\hat{\zeta }} = \zeta /\Vert \zeta \Vert _2$. Observe that

  $$\begin{aligned} \sum _{j=3}^n a_{2j}T_j = \langle \xi ,\zeta \rangle = \Vert \xi \Vert _2 \cdot \Vert \zeta \Vert _2 \cdot \langle {\hat{\xi }},{\hat{\zeta }}\rangle \end{aligned}$$

  (B.1)

  Note that $\xi$ is in $\sigma (X_2, X_3, \dots , X_n)$ and $\zeta$ is in $\sigma (\Gamma _3, \Gamma _4, \dots , \Gamma _n)$, which implies that $\xi$ and $\zeta$ are independent. Therefore, it holds that $\langle {\hat{\xi }},{\hat{\zeta }}\rangle \sim \psi _{n-2}$. By item 5 in Lemma 5.1, there is a constant $C_1 > 0$ depending only on s such that

  $$ {\mathbb {P}}\left[ \left| \langle {\hat{\xi }},{\hat{\zeta }}\rangle \right| \le C_1 \sqrt{\frac{\log {n}}{n-2}}\right] \ge 1- \frac{1}{9n^s} $$

  Since $a_{2j} = \langle X_2, Y_j \rangle$, it follows that $a_{2j} \sim \psi _d$ for each $3 \le j \le n$. Thus for some for some constant $C_2 > 0$ depending only on s, item 5 of Lemma 5.1 again implies that

  $$ {\mathbb {P}}\left[ a_{2j}^2 > \frac{C^2_2\log n}{d}\right] \le \frac{1}{9n^{s+1}} $$

  for each $3 \le j \le n$. Since $\Vert \xi \Vert ^2_2 = \sum _{j=3}^n a_{2j}^2$, if $\Vert \xi \Vert _2 > \sqrt{\tfrac{(n-2)\log {n}}{d}}$, then it must follow that $a_{2j}^2 > C^2_2 \cdot \tfrac{\log n}{d}$ for some j. A union bound now yields that

  $$\begin{aligned} {\mathbb {P}}\left[ \Vert \xi \Vert _2> C_2 \sqrt{\frac{(n-2)\log {n}}{d}}\right] \le \sum _{j = 3}^n {\mathbb {P}}\left[ a_{2j}^2 > \frac{C^2_2\log n}{d}\right] \le \frac{1}{9n^s} \end{aligned}$$

  (B.2)

  By item 2 of Proposition 5.1, we have that $T_j \sim \psi _d$ for each $3 \le j \le n$. Repeating the same union bound argument above yields that there is a constant $C_3 > 0$ depending only on s such that

  $$ {\mathbb {P}}\left[ \Vert \zeta \Vert _2 > C_3 \sqrt{\frac{(n-2)\log {n}}{d}}\right] \le \frac{1}{9n^s} $$

  Therefore each of the following events have probability at least $1 - \frac{1}{9n^s}$ for some constants $C_1,C_2$ and $C_3$ which depend only on s.

  $$\begin{aligned}&\left\{ \left| \langle {\hat{\xi }},{\hat{\zeta }}\rangle \right| \le C_1 \sqrt{\frac{\log {n}}{n-2}}\right\} , \quad \left\{ \Vert \xi \Vert _2 \le C_2 \sqrt{\frac{(n-2)\log {n}}{d}}\right\} \quad \text {and} \\&\left\{ \Vert \zeta \Vert _2 \le C_3 \sqrt{\frac{(n-2)\log {n}}{d}}\right\} \end{aligned}$$

  The result follows from union bound and combining these inequalities with Eq. B.1.

• 2. By the definition of $a_{22}$, we have that $a_{22} = \sqrt{1 - \Vert \xi \Vert ^2_2}$. If $C_2$ is as in Eq. B.2, then the two events $\left\{ a_{22} > \sqrt{1 - C_2^2 \cdot \frac{(n-2)\log {n}}{d}} \right\}$ and $\left\{ \Vert \xi \Vert _2 \le C_2 \sqrt{\frac{(n-2)\log {n}}{d}}\right\}$ coincide. The result now follows from Eq. B.2.

• 3. By definition, we have that $\Gamma _i \sim \psi _{d-n+i}$. Item 5 of Lemma 5.1 implies that

  $$ {\mathbb {P}}\left[ |\Gamma _i| > C_4\sqrt{\frac{\log {n}}{d-n+i}}\right] \le \frac{1}{3n^{s+1}}\,. $$

  Using the fact that $d \gg n\log {n}$ and a union bound, we conclude the result.

Now taking $C_s = \max (C_1C_2C_3, C_2^2, C_4)$ completes the proof of the lemma. $\square$

Proof of Lemma 5.5

From Eq. 5.3, we have that

$$ Q_0 = {\mathbb {P}}\left[ \Gamma _2 \ge t^{\prime }_{p, d} \biggr |{\mathcal {F}}\right] \quad \text {where} \quad t_{p,d}^{\prime } = \frac{t_{p,d} - \sum _{j=3}^n a_{2j} T_j}{a_{22} \cdot \prod _{j=3}^{n} \sqrt{1-\Gamma _j^2}} $$

Since $\Gamma _2$ is independent of ${\mathcal {F}}$ and $t_{p,d}^{\prime }$ is ${\mathcal {F}}$-measurable, we conclude by Fubini’s theorem that

$$ Q_0 = \Psi _{d-n+2}\left( t_{p, d}' \right) $$

Note that by definition, $p = \Psi _d(t_{p,d})$. By the triangle inequality, we have that

$$\begin{aligned} |Q_0 - p|\le & {} \left| \Psi _{d-n+2}\left( t_{p, d}'\right) - \Psi _{d-n+2}(t_{p,d})\right| \nonumber \\&+ \left| \Psi _{d-n+2}(t_{p,d}) -\Psi _d(t_{p,d})\right| \end{aligned}$$

(B.3)

We first will apply Lemma 5.2 to bound $\left| \Psi _{d-n+2}(t_{p,d}) -\Psi _d(t_{p,d})\right|$. By monotonicity, we have ${\bar{\Phi }}(t\sqrt{d}) \le {\bar{\Phi }}(t\sqrt{d-n+2})$. Now observe that

$$\begin{aligned} {\bar{\Phi }}\left( t\sqrt{d-n+2}\right)&= {\bar{\Phi }}\left( t\sqrt{d}\right) + \frac{1}{\sqrt{2\pi }} \int _{t\sqrt{d-n+2}}^{t\sqrt{d}}e^{-\frac{x^2}{2}} dx \nonumber \\&\le {\bar{\Phi }}\left( t\sqrt{d}\right) + \frac{1}{\sqrt{2\pi }} \cdot t\left( \sqrt{d}-\sqrt{d-n+2}\right) \cdot e^{-\frac{(d-n+2)t^2}{2}} \nonumber \\&\le {\bar{\Phi }}\left( t\sqrt{d}\right) + \frac{C_1nt}{\sqrt{d}} \cdot e^{-\frac{t^2(d-n)}{2}} \end{aligned}$$

(B.4)

Here, we have used the fact that $d \gg n\log ^3{n}$. Applying the standard estimate for the Gaussian CDF when $x \ge 1$ given by ${\bar{\Phi }}(x) \ge \frac{1}{\sqrt{2\pi }}\left( \frac{1}{x} - \frac{1}{x^3}\right) e^{-\frac{x^2}{2}}$, we now have that

$$\begin{aligned} {\bar{\Phi }}\left( t\sqrt{d}\right) \ge {\left\{ \begin{array}{ll} {\bar{\Phi }}(2) &{}\quad \text {if } t\sqrt{d} \le 2 \\ \frac{1}{2t\sqrt{2\pi d}} \cdot e^{-\frac{dt^2}{2}} &{}\quad \text {otherwise} \end{array}\right. } \end{aligned}$$

Combining these inequalities with Eq. B.4 and the fact that $d\gg n$ yields

$$\begin{aligned} 1 \le \frac{{\bar{\Phi }}\left( t\sqrt{d-n+2}\right) }{{\bar{\Phi }}\left( t\sqrt{d}\right) } \le {\left\{ \begin{array}{ll} 1 + \frac{Cn}{d} &{}\quad \text {if } t\sqrt{d} \le 2 \\ 1 + Cnt^2 \cdot e^{\frac{nt^2}{2}} &{}\quad \text {otherwise} \end{array}\right. } \end{aligned}$$

(B.5)

for an absolute constant $C > 0$. Let $C_{\mathsf {est}}$ be the positive constant given in Lemma 5.2. Since $p \gg n^{-3}$ and $d \gg n \log n$, we have that $t_{p,d} < C_{\mathsf {est}}$ for sufficiently large n by item 2 of Lemma 5.1. Using the distributional approximation in Lemma 5.2, we can bound $\Psi _d$ and $\Psi _{d-n+2}$ as follows in terms of ${\bar{\Phi }}$. Since $p = \Psi _d(t_{p,d})$, we have

$$\begin{aligned}&\bigr |\Psi _{d-n+2}(t_{p,d}) -\Psi _d(t_{p,d})\bigr | \nonumber \\&\quad = p \cdot \left| \frac{\Psi _{d-n+2}(t_{p,d})}{\Psi _d(t_{p,d})}-1\right| \nonumber \\&\quad = p \cdot \left| \left( 1+ O_n(d^{-1}) \right) \cdot e^{O_n(dt^4_{p,d})} \cdot \frac{{\bar{\Phi }}\left( t_{p,d} \sqrt{d-n+2}\right) }{{\bar{\Phi }}\left( t_{p,d}\sqrt{d}\right) } -1\right| \nonumber \\&\quad = p \cdot \left| \left( 1+ O_n(d^{-1}) \right) \cdot e^{O_n(dt^4_{p,d})} \cdot \left( 1 + O_n\left( nt_{p,d}^2\right) \right) -1\right| \nonumber \\&\quad = O_n\left( pdt_{p,d}^4+pnt_{p,d}^2 + \frac{p}{d}\right) \end{aligned}$$

(B.6)

We now will bound the term $|\Psi _{d-n+2}(t_{p, d}') - \Psi _{d-n+2}(t_{p,d})|$ by approximating the density $\psi _{d-n+2}$ in the neighborhood of $t_{p,d}$. First observe that combining the items in Lemma 5.4 with $d \gg n \log n$ implies that

$$\begin{aligned} a_{22} \cdot \prod _{j=3}^{n} \sqrt{1-\Gamma _j^2} = 1 - O_n\left( \frac{n \log n}{d} \right) \end{aligned}$$

(B.7)

on the event $E_{\mathsf {rem}}$. Combining this bound with the expression for $t_{p, d}'$, the fact that $p \gg n^{-3}$ and the bounds in item 2 of Lemma 5.1 now yields that

$$\begin{aligned} |t_{p,d}-t_{p,d}^{\prime }| \cdot {\mathbb {1}}(E_{\mathsf {rem}}) = O_n\left( \frac{\sqrt{n}\log ^{3/2}{n}}{d} \right) \end{aligned}$$

(B.8)

Observe that this difference is $O_n\left( \sqrt{\tfrac{\log n}{d}} \right)$ on the event $E_{\mathsf {rem}}$ since $d \gg n \log ^2 n$. Let

$$ u = \underset{x \in \left[ t_{p, d}, t_{p, d}'\right] }{\text {argmin}} \, |x| $$

Note that $u = O_n\left( \sqrt{\tfrac{\log n}{d}} \right)$ conditioned on $E_{\mathsf {rem}}$. Thus given $E_{\mathsf {rem}}$ holds,

$$\begin{aligned} \bigr |\Psi _{d-n+2}\left( t^{\prime }_{p,d}\right) - \Psi _{d-n+2}(t_{p,d})\bigr |&= \left| \int _{t_{p,d}}^{t'_{p,d}} \psi _{d-n+2}(x)dx \right| \\&\le \psi _{d-n+2}\left( u \right) \cdot \left| t^{\prime }_{p,d}-t_{p,d}\right| \\&= \frac{\psi _{d-n+2}\left( u \right) }{\psi _d(u)} \cdot \frac{\psi _d(u)}{\psi _d(t_{p,d})} \cdot \psi _d(t_{p,d}) \cdot \left| t^{\prime }_{p,d}-t_{p,d}\right| \end{aligned}$$

where the inequality follows from the fact that $\psi _{d - n + 2}(t)$ is monotonically decreasing in |t|. Furthermore, we have that

$$\begin{aligned}&\bigr |\Psi _{d-n+2}\left( t^{\prime }_{p,d}\right) - \Psi _{d-n+2}(t_{p,d})\bigr | \\&\quad \lesssim \left( \sqrt{\frac{d - n + 2}{d}} \cdot \left( 1 - u^2 \right) ^{-n/2} \right) e^{2dt_{p,d} | u - t_{p, d} |} \cdot \psi _d(t_{p,d}) \cdot \left| t^{\prime }_{p,d}-t_{p,d}\right| \\&\quad \lesssim \sqrt{\frac{d - n}{d}} \cdot \left( 1 + O_n\left( \frac{n \log n}{d} \right) \right) e^{2dt_{p,d} | t'_{p,d} - t_{p, d} |} \cdot \psi _d(t_{p,d}) \cdot \left| t^{\prime }_{p,d}-t_{p,d}\right| \\&\quad = \left( 1 + O_n\left( \frac{n \log n}{d} \right) \right) \exp \left( O_n\left( d \cdot \sqrt{ \frac{\log p^{-1}}{d}} \cdot \frac{\sqrt{n}\log ^{3/2}{n}}{d} \right) \right) \\&\qquad \cdot \psi _d(t_{p,d}) \cdot \left| t^{\prime }_{p,d}-t_{p,d}\right| \\&\quad = \left( 1 + O_n\left( \frac{n \log n}{d} + \sqrt{\frac{n \log ^4 n}{d}} \right) \right) \cdot \psi _d(t_{p,d}) \cdot \left| t^{\prime }_{p,d}-t_{p,d}\right| \\&\quad = \left( 1+o_n(1)\right) \cdot \psi _d(t_{p,d}) \cdot \left| t^{\prime }_{p,d}-t_{p,d}\right| \end{aligned}$$

The second inequality follows from items 1 and 3 of Lemma 5.1 and using $\Gamma \left( \frac{d}{2}\right) /\Gamma \left( \frac{d-1}{2}\right) \sqrt{\pi } = \Theta (\sqrt{d})$. The third inequality follows from the fact that Bernoulli’s inequality implies that $(1 - u^2)^{-n/2} \le 1 + nu^2$ if $nu^2 \le 1$. The third last equality follows from item 2 of Lemma 5.1, the fact that $p \gg n^{-3}$ and Eq. B.8. The final estimate follows from the fact that $d \gg n\log ^4 n$. Let $C > 0$ be the constant in the $\lesssim$ above. Substituting this bound into Eq. B.3, we have

$$ |Q_0-p| \cdot {\mathbb {1}}(E_{\mathsf {rem}})\le O_n\left( pdt_{p,d}^4+pnt_{p,d}^2 + \frac{p}{d}\right) + C(1+o_n(1)) \cdot \psi _{d}\left( t_{p,d}\right) \cdot \bigr |t^{\prime }_{p,d}-t_{p,d}\bigr | $$

Now note that

$$ pdt_{p,d}^4+pnt_{p,d}^2 + \frac{p}{d} = O_n\left( \frac{pn}{d}\log { p^{-1}}\right) $$

Therefore we have that for sufficiently large n,

$$ |Q_0-p| \cdot {\mathbb {1}}(E_{\mathsf {rem}})\le O_n\left( \frac{pn}{d}\log { p^{-1}}\right) +2 C\psi _{d}\left( t_{p,d}\right) \cdot \bigr |t^{\prime }_{p,d}-t_{p,d}\bigr | $$

This proves the first claim in the lemma. Using the fact that $\psi _d(t_{p,d})\le Cp\sqrt{d\log p^{-1}}$ and Eq. B.8, we conclude that

$$ |Q_0-p| \cdot {\mathbb {1}}(E_{\mathsf {rem}})\le O_n\left( \frac{pn}{d}\log { p^{-1}}\right) + O_n\left( p\sqrt{\frac{n\log { p^{-1}}}{d}} \cdot \log ^{3/2}{n} \right) $$

which proves the second claim in the lemma. $\square$

Proof of Lemma 5.6

Given the event $E_{\mathsf {rem}}$, Eq. B.7 and the expression for $t_{p,d}'$ imply that

$$\begin{aligned} \bigr |t^{\prime }_{p,d}-t_{p,d}\bigr | \cdot {\mathbb {1}}(E_{\mathsf {rem}})&\le O_n\left( t_{p,d} \cdot \frac{n\log {n}}{d}\right) + (1+o_n(1)) \cdot \left| \sum _{j=3}^n T_j a_{2j}\right| \\&\le O_n\left( n\left( \frac{\log {n}}{d}\right) ^{\frac{3}{2}}\right) + (1+o_n(1)) \cdot \left| \sum _{j=3}^n T_j a_{2j}\right| \end{aligned}$$

using the upper bound on $t_{p,d}$ in item 2 of Lemma 5.1. The inequality $(x+y)^2 \le 2x^2 + 2y^2$ yields

$$\begin{aligned} {\mathbb {E}}\left[ \bigr |t^{\prime }_{p,d}-t_{p,d}\bigr |^2 \cdot {\mathbb {1}}(E_{\mathsf {rem}}) \right] \lesssim n^2\left( \frac{\log {n}}{d}\right) ^{3}+ {\mathbb {E}}\left[ \left| \sum _{j=3}^n T_j a_{2j}\right| ^2 \right] \end{aligned}$$

(B.9)

Now recall that $Y_j$ is a unit norm random vector in the $\sigma$-algebra $\sigma (X_j, \dots , X_n)$. Therefore, for $j \ge 3$, $Y_j$ is independent of $X_2$. Also note that $a_{2j} = \langle X_2,Y_j\rangle$. Furthermore, the $T_j$ are independent of $X_2,\dots ,X_n$ and, since the random variable $T_j T_k$ is symmetric about zero, we have that ${\mathbb {E}}[T_jT_k] = 0$ for $k\ne j$. Thus ${\mathbb {E}}[T_jT_k a_{2j}a_{2k}] = {\mathbb {E}}[T_jT_k] \cdot {\mathbb {E}}[a_{2j}a_{2k}] = 0$ if $j\ne k$, and hence

$$ {\mathbb {E}}\left[ \left| \sum _{j=3}^n T_j a_{2j}\right| ^2 \right] = \sum _{j=3}^{n} {\mathbb {E}}\left[ T_j^2 a_{2j}^2\right] = \sum _{j=3}^{n} {\mathbb {E}}\left[ T_j^2\right] \cdot {\mathbb {E}}\left[ a_{2j}^2\right] = \frac{n - 2}{d^2} \le \frac{n}{d^2} $$

Where the equality holds because ${\mathbb {E}}\left[ a_{2j}^2\right] = {\mathbb {E}}\left[ T_j^2\right] = \frac{1}{d}$ for $3 \le j \le n$, since $a_{2j}, T_j \sim \psi _d$ by item 2 in Proposition 5.1. Substituting this into Eq. B.9 completes the proof of the lemma. $\square$