1 Introduction and statement of results

The arithmetic Kakeya conjecture, sometimes known as the sums-differences conjecture, was formulated by Katz and Tao around fifteen years ago. It is a purely additive-combinatorial statement which, if true, would have a deep geometric consequence—that the Minkowski dimension of Besicovitch sets in \({\mathbf {R}}^n\) is n. This is the celebrated Kakeya conjecture, discussed at length in many places: for an introduction see [21].

The arithmetic Kakeya conjecture is mentioned explicitlyFootnote 1 in [20]. One of the main aims of this paper is to give a number of equivalent forms of the conjecture. Here is probably the simplest formulation. It is not the original one of Katz and Tao, which is Conjecture 1.3 below.

Conjecture 1.1

Let k, N be positive integers. Write \(F_k(N)\) for the size of the smallest set of integers containing, for each \(d \in \{1,\dots , N\}\), a k-term arithmetic progression with common difference d. Then

$$\begin{aligned} \lim _{k \rightarrow \infty } \lim _{N \rightarrow \infty } \frac{\log F_k(N)}{\log N} = 1. \end{aligned}$$

This conjecture was raised by the second author as [17, Conjecture 4.2], but no links to the Kakeya problem were mentioned there.

We turn now to arguably the most natural of our formulations, concerning the entropy of random variables. As usual, the entropy \({\mathbf {H}}\) of a random variable \(\mathsf {X}\) with finite range is defined by

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}) := -\sum _{x} {\mathbf {P}}(\mathsf {X}= x) \log {\mathbf {P}}(\mathsf {X}= x), \end{aligned}$$

where x ranges over all values taken by \(\mathsf {X}\).

Conjecture 1.2

For any \(\varepsilon > 0\) there areFootnote 2\(r_1,\dots , r_k \in {\mathbf {Q}}\), none equal to \(-1\), such that for any two real-valued random variables \(\mathsf {X}\) and \(\mathsf {Y}\) taking only finitely many values we have

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}- \mathsf {Y}) \leqslant (1 + \varepsilon ) \sup _j {\mathbf {H}}(\mathsf {X}+ r_j \mathsf {Y}). \end{aligned}$$

Next we give the original form of the conjecture discussed by Katz and Tao. Let \(A \subset {\mathbf {Z}}\times {\mathbf {Z}}\) be a finite set. For rational r we write \(\pi _r(A) := \{ x + ry : (x,y) \in A\}\). We also write \(\pi _{\infty }(A) := \{ y : (x,y) \in A\}\).

Conjecture 1.3

Let \(\varepsilon > 0\) be arbitrary. Then there are \(r_1,\dots , r_k \in {\mathbf {Q}}\cup \{\infty \}\), none equal to \(-1\), such that \(\#\pi _{-1}(A) \leqslant \sup _{i} \# \pi _{r_i}(A)^{1 + \varepsilon }\) for all finite sets \(A \subset {\mathbf {Z}}\times {\mathbf {Z}}\).

Our fourth conjecture has not, so far as we are aware, appeared explicitly in the literature before. It is in fact a whole family of conjectures, one for each natural number n; however, we will later show that all of these are equivalent.

Conjecture 1.4

(n) Let k be a positive integer. If p is a prime, let \(f_{k,n}(p)\) denote the size of the smallest set containing, for every \(d \in {\mathbf {F}}_p^n {\setminus } \{0\}\), a k-term progression with common difference d. Then

$$\begin{aligned} \lim _{k \rightarrow \infty } \lim _{p \rightarrow \infty } \frac{\log f_{n,k}(p)}{\log p} = n. \end{aligned}$$

Remarks

Note that \(f_{p,n}(p)\) is the size of the smallest Besicovitch set in \({\mathbf {F}}_p^n\), that is to say a set containing a full line in every direction. Since \(f_{p,n}(p) \geqslant f_{k,n}(p)\) whenever \(p \geqslant k\), Conjecture 1.4 (n) trivially implies that

$$\begin{aligned} \lim _{p \rightarrow \infty } \frac{f_{p,n}(p)}{\log p} = n, \end{aligned}$$

i.e. any Besicovitch set in \({\mathbf {F}}_p^n\) has size \(p^{n - o_{p\rightarrow \infty }(1)}\). This is known to be true, a celebrated result of Dvir [5]. However, the only known arguments use the “polynomial method” (see, for example, [11, 22] for modern introductions). This very strongly hints that any proof of Conjecture 1.4 (and hence, by our main theorem, of the other conjectures) would have to use some form of the polynomial method.

Our fifth and final conjecture is included mainly for historical interest, as it relates very closely to a question asked by Erdős and Selfridge in the 1970s, well before the current wave of interest in the Kakeya problem and related matters.

Conjecture 1.5

Fix a positive integer k. Then, uniformly for all positive integers N, all finite sets \(p_1< \dots < p_N\) of primes and all intervals \(I \subset {\mathbf {N}}\) of length \(kp_N\), we have

$$\begin{aligned} \# \big ( I \cap \bigcup _{i = 1}^N p_i {\mathbf {Z}}\big ) \gg _k N^{1 - \gamma _k}. \end{aligned}$$

where \(\gamma _k \rightarrow 0\) as \(k \rightarrow \infty \).

Remark

Erdős and Selfridge [8, §6] in fact asked whether or not one can take \(\gamma _k = 0\). The second-named author [16] showed that the answer is negative, and in fact we must have \(\gamma _k \geqslant \frac{1}{k}\). We note that Proposition 4.1 and Theorem 1.7 combine to give the much better bound \(\gamma _k \gg \frac{1}{\log \log k}\).

As previously stated, our main result is the equivalence of the five conjectures stated above.

Theorem 1.6

Conjectures 1.1, 1.2, 1.3, 1.4(n) (for each \(n = 1,2,3,\dots \)) and 1.5 are all equivalent.

Let us make some further remarks.

  1. (1)

    Once Theorem 1.6 is proven, it seems natural to use the term “arithmetic Kakeya conjecture” to refer to any one of the five conjectures.

  2. (2)

    It is known that Conjecture 1.3 (and hence all the other conjectures) implies that the upper Minkowski dimension of any Besicovitch setFootnote 3 in \({\mathbf {R}}^n\) is n, a statement often referred to as the Kakeya conjecture. This follows by a straightforward generalisation of the “slicing” argument of Bourgain [4]: a sketch of this may be found in [21]. However, Bourgain [2, 3] observed that, in the notation of Conjecture 1.1, if the statement

    $$\begin{aligned} \lim _{N \rightarrow \infty } \frac{\log F_{N^{\eta }}(N)}{\log N} \geqslant 1 \end{aligned}$$
    (1.1)

    is true for all \(\eta > 0\) then the Kakeya conjecture follows. Since \(F_k(N)\) is a nondecreasing function of k, (1.1) is immediately implied by Conjecture 1.1, whilst an implication in the reverse direction seems very unlikely without resolving both conjectures. In this sense, the arithmetic Kakeya conjecture should be considered a strictly harder problem than the Kakeya conjecture.

  3. (3)

    The equivalence of Conjectures 1.2 and 1.3 was proven by the second author in [18] (see also [14]). We are not aware of any references for the other implications.

Now we discuss the other results in the paper. First, we establish a lower bound showing that the convergence in Theorem 1, if it occurs, is very slow.

Theorem 1.7

In the notation of Conjecture 1.1, we have

$$\begin{aligned} \lim _{N \rightarrow \infty } \frac{\log F_k(N)}{\log N} \leqslant 1- \frac{c}{\log \log k}, \end{aligned}$$

where the constant \(c > 0\) is absolute.

Second, we show that a finite field variant of Conjecture 1.2is true. Write \({\mathbf {F}}_p^{\infty } := \bigcup _{n = 1}^{\infty } {\mathbf {F}}_p^n\).

Theorem 1.8

Suppose that \(\mathsf {X}\) and \(\mathsf {Y}\) are two \({\mathbf {F}}_p^{\infty }\)-valued random variables, both taking only finitely many values. Then

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}- \mathsf {Y}) \leqslant \left( 1 + O\left( \frac{1}{\log p}\right) \right) \sup _{r \in {\mathbf {F}}_p \cup \{\infty \} {\setminus } \{-1\}} {\mathbf {H}}(\mathsf {X}+ r \mathsf {Y}). \end{aligned}$$

Here, the constant in the O() notation is absolute.

The \(O(\frac{1}{\log p})\) term is best possible, as we remark in Sect. 6.

We neither discuss nor make progress on partial results towards any of Conjectures 1.1, 1.2, 1.3, 1.4 or 1.5. We believe that the best value of \(\varepsilon \) for which Conjecture 1.2 is known is \(\varepsilon \approx 0.67513\dots \), which is equivalent to a result obtained in [13]. (The precise value here is \(\alpha - 1\), where \(\alpha \) solves \(\alpha ^3 - 4\alpha + 2 = 0\).) This bound is now 15 years old.

Notation. Most of our notation is quite standard. We use \(\# X\) for the cardinality of a set X. Occasionally, if A is a set in some abelian group and k is an integer we will write \(k \cdot A\) to mean \(\{ ka : a \in A\}\).

2 Progressions, projections and entropy

In this section we establish around half of Theorem 1.6 by proving that the first three conjectures mentioned in the introduction are equivalent. Whilst at a local level the arguments are a mix of fairly unexciting linear algebra and standard tools such as Freiman isomorphisms, random projections and taking tensor powers, the large number of them makes the proof of Theorem 1.6 somewhat lengthy.

It is convenient to proceed by first showing that Conjectures 1.1, 1.3 and 1.2 are equivalent. In the course of doing so, and for later use, it is convenient to introduce a further conjecture, apparently stronger than Conjecture 1.1 but, as it turns out, equivalent to it.

Conjecture 1’. Let k be a positive integer. Write \(F'_k(N)\) for the cardinality of the smallest set \(A \subset {\mathbf {Z}}\) which contains an arithmetic progression of length k and common difference d, for N different values of d. Then

$$\begin{aligned} \lim _{k \rightarrow \infty }\lim _{N \rightarrow \infty } \frac{\log F'_k(N)}{\log N} = 1. \end{aligned}$$

It is obvious that Conjecture 1’ implies Conjecture 1.1, because \(F'_k(N) \leqslant F_k(N)\). It turns out that the reverse implication holds as well. In fact, we claim that the following is true.

Proposition 2.1

We have \(F_k(N) \ll k^3 \log N \cdot F'_k(N)\).

Proof

Suppose we have a set

$$\begin{aligned} A_0 = \bigcup _{i = 1}^N \bigcup _{j = 0}^{k-1}\{ a_i + jd_i\}, \end{aligned}$$

where the \(d_i\) are distinct. We claim that there is a set \(A_1\), \(\# A_1 \ll k^3 \log N \cdot \# A_0\), containing an arithmetic progression of length k and common difference d for all \(d \in \{1,\dots , N\}\). This obviously implies the result.

Pick \(\theta \in (0,1)\) uniformly at random, and define the function

$$\begin{aligned} \phi _{\theta } : {\mathbf {Z}}\rightarrow \{0,1,\dots ,N-1\} \end{aligned}$$

by

$$\begin{aligned} \phi _{\theta }(x) := \lfloor N \{ \theta x\} \rfloor . \end{aligned}$$

Here, \(\{t\} = t - \lfloor t \rfloor \), so \(0 \leqslant \{t \} < 1\).

Note that if \(i \ne j\) then

$$\begin{aligned} {\mathbf {P}}_{\theta }(\phi _{\theta }(d_i) = \phi _{\theta }(d_j)) \leqslant {\mathbf {P}}_{\theta }(\theta (d_i - d_j) \in \left( -\frac{1}{N}, \frac{1}{N}\right) ({\text {mod}}\, 1)) = \frac{2}{N}. \end{aligned}$$

It follows that the expected number of pairs (i, j) with \(i < j\) for which \(\phi _{\theta }(d_i) = \phi _{\theta }(d_j)\) is at most \(\frac{2}{N} \left( {\begin{array}{c}N\\ 2\end{array}}\right) = N - 1\). By linearity of expectation, there is some choice of \(\theta \) for which, setting \(d'_i := \phi _{\theta }(d_i)\), there are at most \(N - 1\) pairs (i, j) with \(i < j\) and \(d'_i = d'_j\). If \(n \in \{0,1,\dots , N-1\}\), write f(n) for the number of i with \(d'_i = n\). Then it follows that \(\sum _n \left( {\begin{array}{c}f(n)\\ 2\end{array}}\right) \leqslant N - 1\), from which we obtain, since \(\sum _n f(n) = N\), that \(\sum _n f(n)^2 \leqslant 3N\). By Cauchy–Schwarz,

$$\begin{aligned} N^2 = \left( \sum _n f(n)\right) ^2 \leqslant \# \{n : f(n) \ne 0\} \sum _n f(n)^2, \end{aligned}$$

and therefore there are at least N / 3 values of n for which \(f(n) \ne 0\), or in other words there are at least N / 3 distinct values amongst the \(d'_i\).

Now consider the set \(A_2 := \phi _{\theta }(A_0)\). Obviously \(\# A_2 \leqslant \# A_0\). Whilst \(A_2\) itself does not obviously contain any long progressions, we observe that

$$\begin{aligned} \phi _{\theta }(a_i + (j+1) d) - \phi _{\theta }(a_i + j d) - d'_i \in \{0,1\} - \{0,N\} \end{aligned}$$

(In fact, \(\phi _{\theta }(x + y) - \phi _{\theta }(x) - \phi _{\theta }(y) \in \{0,1\} - \{ 0,N\}\) for every x, y.) By a simple induction,

$$\begin{aligned} \phi _{\theta }(a_i) + jd'_i - \phi _{\theta }(a_i + jd) \in \{0,1,\dots , k-1\} - \{ 0, N, \dots , (k-1) N\} \end{aligned}$$

for \(j = 0,1,\dots , k-1\), and so the set \(A_3 := A_2 + \{0,1,\dots , k-1\} - \{ 0, N, \dots , (k-1) N\}\) contains a progression of length k and common difference \(d'_i\), for all i. Note that \(\# A_3 \leqslant k^2 \# A_0\).

By taking random translates (see Lemma 6.2 for details) and the fact that there are \(\geqslant N/3\) distinct \(d'_i\), there is some set T of integers, \(\# T \ll \log N\), such that every element of \(\{1,\dots , N\}\) can be written as \(d'_i + t\) with \(t \in T\). Set

$$\begin{aligned} A_1 := A_3 + \{0,1,\dots , k-1\} \cdot T. \end{aligned}$$

We have \(\# A_1 \leqslant k \cdot \# T \cdot \# A_3 \ll k^3 \log N \cdot \# A_0\). It is easy to see that \(A_1\) contains an arithmetic progression of length k and common difference \(d'_i + t\), for all i and for all \(t \in T\), and hence contains an arithmetic progression of length k and common difference d for all \(d \in \{1,\dots , N\}\). This concludes the proof of Proposition 2.1. \(\square \)

Now we turn to the proof that Conjectures 1’, 1.2 and 1.3 are equivalent.

Conjecture 1’ implies Conjecture 1.3. Suppose that Conjecture 1.3 is false. Then there is some \(\varepsilon > 0\) such that, for every k, there is a set \(A_k \subset {\mathbf {Z}}\times {\mathbf {Z}}\) such that

$$\begin{aligned} \# \pi _{-1}(A_k) > \max _{r \in H_k {\setminus } \{-1\}} \# \pi _r(A_k)^{1 + \varepsilon }, \end{aligned}$$
(2.1)

where \(H_k\) denotes the set of rationals with height at most k, that is to say

$$\begin{aligned} H_k := \left\{ \frac{a}{b} : |a|, |b| \leqslant k\right\} \cup \{\infty \}. \end{aligned}$$

Our first step is to use a “tensor power” argument to show that there are arbitrarily large sets with the same property; in fact, we shall argue that for every j there is a set \(A_{k,j} \subset {\mathbf {Z}}\times {\mathbf {Z}}\) such that

$$\begin{aligned} \# \pi _{-1}(A_{k,j}) \geqslant j\max _{r \in H_k {\setminus } \{-1\}} \# \pi _r(A_{k,j})^{1 + \varepsilon }. \end{aligned}$$
(2.2)

This is simple if the \(A_{k,j}\) are allowed to be subsets of \({\mathbf {Z}}^n\). Indeed we may define \(A_k^{(n)}\) to be the set

$$\begin{aligned} \left\{ \big ( (a_1, a_2,\dots , a_n), (a^{\prime }_1,a^{\prime }_2,\dots , a^{\prime }_n) \big ) \in {\mathbf {Z}}^n \times {\mathbf {Z}}^n : (a_i, a^{\prime }_i) \in A_k \; \text{ for } \text{ all } i\right\} . \end{aligned}$$

Then, writing \(\pi _r^{(n)} : {\mathbf {Z}}^n \times {\mathbf {Z}}^n \rightarrow {\mathbf {Z}}^n\), for the map sending (x, y) to \(x + ry\) (or, when \(r = \infty \), to y) we have

$$\begin{aligned} \# \pi _r^{(n)}(A_k^{(n)}) = \big ( \# \pi _r(A_k) \big ) ^n \end{aligned}$$

for all r, n. In particular, by choosing n large enough (depending on j) we have, since the inequality (2.1) is strict,

$$\begin{aligned} \# \pi _{-1}^{(n)}(A_k^{(n)}) \geqslant j \max _{r \in H_k {\setminus } \{-1\}} \# \pi _r^{(n)} \left( A_k^{(n)}\right) ^{1 + \varepsilon }. \end{aligned}$$
(2.3)

To create a subset of \({\mathbf {Z}}\times {\mathbf {Z}}\) from \(A_k^{(n)}\), we take an integer t and apply a map \(\psi _t : {\mathbf {Z}}^n \times {\mathbf {Z}}^n \rightarrow {\mathbf {Z}}\times {\mathbf {Z}}\) of the form

$$\begin{aligned} \psi _t(x,y) = ((t, t^2,\dots , t^n) \cdot x, (t, t^2,\dots , t^n) \cdot y)), \end{aligned}$$

where the dot denotes the usual inner product. Set \(A := \psi _t (A_k^{(n)})\). Choose t such that for \(r \in H_k\) and \((x,y), (x',y') \in A_k^{(n)}\) we have

$$\begin{aligned} (\pi _r^{(n)}(x,y) - \pi _r^{(n)}(x',y')) \cdot (t, t^2,\dots , t^n) \ne 0 \end{aligned}$$
(2.4)

unless \(\pi _r^{(n)}(x,y) = \pi _r^{(n)}(x',y')\). There is such a t, because for each of the finite number of choices of \(x,y,x',y',r\) the left-hand side of (2.4) is a nontrivial polynomial equation in t. For such a choice of t it follows that \(\pi _r (\psi _t(x,y)) = \pi _r(\psi _t(x',y'))\) if and only if \(\pi _r^{(n)}(x,y) = \pi _r^{(n)}(x', y')\), and so

$$\begin{aligned} \# \pi _r(A) = \# \pi _r^{(n)}(A_k^{(n)}) \end{aligned}$$

for all r. This establishes the existence of the sets \(A_{k,j}\) satisfying (2.2).

For each j, k, consider the set \(S_{k,j} \subset {\mathbf {Q}}\) defined by

$$\begin{aligned} S_{k,j} := \bigcup _{1 \leqslant i \leqslant k} \bigcup _{r \in H_k {\setminus } \{-1\}} \frac{i}{k} \cdot \pi _r(A_{k,j}). \end{aligned}$$

Then

$$\begin{aligned} \# S_{k,j} \leqslant k \cdot \# H_k \cdot \max _{r \in H_k^+} \# \pi _r(A_{k,j}) \ll _k \left( j^{-1}\# \pi _{-1}(A_{k,j})\right) ^{1/(1 + \varepsilon )}. \end{aligned}$$

On the other hand, suppose that \(d \in -\pi _{-1}(A_{k,j})\). This means that \(d = y - x\) for some \((x,y) \in A_{k,j}\). If \(0 \leqslant i \leqslant k - 1\), we have

$$\begin{aligned} x + \frac{id}{k} = \frac{k-i}{k}\left( x + \frac{i}{k-i} y\right) . \end{aligned}$$

Since \(x + \frac{i}{k-i} y \in \pi _{i/(k-i)}(A_{k,j}) \subset \bigcup _{r \in H_k^+} \pi _r(A_{k,j})\), it follows that \(x + \frac{id}{k} \in S_{k,j}\) for \(i = 0,1,\dots , k-1\), that is to say \(S_{k,j}\) contains a progression of length k and common difference \(\frac{d}{k}\). Thus, writing \(N_j := \# \pi _{-1}(A_{k,j})\), we see that \(S_{k,j}\) is a set of size \(\ll (j^{-1} N_j)^{1/(1 + \varepsilon )}\) containing progressions of length k with at least \(N_j\) distinct common differences. Since, evidently, \(\# S_{k,j} \geqslant k\), the presence of the factor \(j^{-1}\) forces \(N_j \rightarrow \infty \) as \(j \rightarrow \infty \). By multiplying through by an appropriate integer, we may find sets \({\tilde{S}}_{k,j} \subset {\mathbf {Z}}\) with the same property, contrary to Conjecture 1’.

Conjecture 1.3implies Conjecture 1.2. This implication is essentially given in [18]. The notation there takes a little unpicking and the proof is short, so we repeat the argument.

Let \(\varepsilon > 0\), and suppose that \(r_1,\dots , r_k \in {\mathbf {Q}}_{\geqslant 0} \cup \{\infty \} {\setminus } \{-1\}\) are such that

$$\begin{aligned} \# \pi _{-1}(A) \leqslant \sup _i \# \pi _{r_i}(A)^{1 + \varepsilon } \end{aligned}$$
(2.5)

for all finite sets \(A \subset {\mathbf {Z}}\times {\mathbf {Z}}\). We claim that

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}- \mathsf {Y}) \leqslant (1 + \varepsilon )\sup _j {\mathbf {H}}(\mathsf {X}+ r_j \mathsf {Y}). \end{aligned}$$
(2.6)

for all \({\mathbf {Z}}\)-valued random variables \(\mathsf {X}, \mathsf {Y}\), both taking only finitely many values. (Let us remind the reader that, by convention, \({\mathbf {H}}(\mathsf {X}+ \infty \mathsf {Y}) = {\mathbf {H}}(\mathsf {Y})\).)

We begin with a couple of observations. The first is that (2.5) is automatically true for sets \(A \subset {\mathbf {Z}}^n \times {\mathbf {Z}}^n\), for any n. This follows from the case \(n = 1\) by applying a suitable map \(\psi _t : {\mathbf {Z}}^n \times {\mathbf {Z}}^n \rightarrow {\mathbf {Z}}\times {\mathbf {Z}}\), exactly as in the argument following (2.3) above.

The second observation is that, by a simple limiting argument, we may assume that there is some q such that \(q{\mathbf {P}}((\mathsf {X}, \mathsf {Y}) = (x,y)) \in {\mathbf {Z}}\) for all (x, y): if we can prove the result for such \((\mathsf {X}, \mathsf {Y})\) the same inequality for arbitrary \((\mathsf {X}, \mathsf {Y})\) with finite range follows by letting \(q \rightarrow \infty \).

Now let m be very large, and construct a set \(A \subset {\mathbf {Z}}^{mq} \times {\mathbf {Z}}^{mq}\) as follows. Let it consist of all pairs \(((x_1,\dots , x_{mq}), (y_1,\dots , y_{mq})) \in {\mathbf {Z}}^{mq} \times {\mathbf {Z}}^{mq}\) for which

$$\begin{aligned} \# \{ i : (x_i, y_i) = (x,y)\} = mq{\mathbf {P}}((\mathsf {X}, \mathsf {Y}) = (x,y)). \end{aligned}$$

Let us calculate \(\# \pi _{r}(A)\). After a moment’s thought we see that

$$\begin{aligned} \pi _r(A) = \big \{ (z_1,\dots , z_{mq}) : \# \{i : z_i = z\} = mq {\mathbf {P}}(\mathsf {X}+ r\mathsf {Y}= z)\big \}. \end{aligned}$$

(Here, we interpret \({\mathbf {P}}(X + \infty \mathsf {Y}= z)\) as \({\mathbf {P}}(\mathsf {Y}= z)\).) Writing \(n = mq\) and \(p_z = {\mathbf {P}}(\mathsf {X}+ r\mathsf {Y}= z)\) for short, it follows that

$$\begin{aligned} \# \pi _r (A) = \frac{n!}{\prod _{z} (n p_z)!}. \end{aligned}$$

Note that the product over z is finite, and that each \(n p_z\) is an integer. Taking logs and using the fact that \(\log N! = N \log N - N + o(N)\), we have

$$\begin{aligned} \log \pi _r(A) = -n \sum _z p_z \log p_z + o(n) = n {\mathbf {H}}(\mathsf {X}+ r\mathsf {Y}) + o(n). \end{aligned}$$

We may assume that the o(n) term is uniform in \(r \in \{r_1,\dots , r_k\}\) (since this is a finite set); of course, it also depends on \(\mathsf {X}, \mathsf {Y}\), but we are thinking of these as fixed for the duration of the argument.

Taking logs of (2.5) (which is valid for \(A \subset {\mathbf {Z}}^{n} \times {\mathbf {Z}}^n\), as remarked), we conclude that

$$\begin{aligned} n{\mathbf {H}}(\mathsf {X}- \mathsf {Y}) \leqslant (1 + \varepsilon )n \sup _i {\mathbf {H}}(\mathsf {X}+ r_i\mathsf {Y}) + o(n). \end{aligned}$$

Now we may simply divide through by n and let \(n \rightarrow \infty \) to conclude the claim (2.6).

Conjecture 1.2implies Conjecture 1.1. This is relatively easy. Assume Conjecture 1.2. Let \(\varepsilon > 0\) be arbitrary, and select \(r_1,\dots , r_m \in {\mathbf {Q}}\cup \{\infty \} {\setminus } \{-1\}\) so that we have

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}- \mathsf {Y}) \leqslant (1 + \varepsilon ) \sup _i {\mathbf {H}}(\mathsf {X}+ r_i \mathsf {Y}). \end{aligned}$$
(2.7)

Let Q, M be positive integers to be specified later (depending on \(r_1,\dots \), \(r_m\)) and suppose that \(A \subset {\mathbf {Z}}\) contains an arithmetic progression of length \(k = 2MQ\) and common difference d, for every \(d \in \{1,\dots , N\}\). Define \({\mathbf {Z}}\)-valued random variables \(\mathsf {X}\), \(\mathsf {Y}\) as follows: pick d uniformly at random, and let \(\{a(d),\dots , a(d) + (k-1) d\}\) be the progression in A for which a(d) is minimal (choosing a(d) minimal is not important, but is one way of making a definite choice). Set \(\mathsf {X}= a(d) + MQd\) and \(\mathsf {Y}= a(d) + (M+1)Qd\).

Then \(\mathsf {X}- \mathsf {Y}\) is uniformly distributed on the set \(\{-Q, -2Q, \dots , -NQ\}\), and so

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}- \mathsf {Y}) = \log N. \end{aligned}$$
(2.8)

On the other hand,

$$\begin{aligned} {\mathbf {H}}(X + r_j \mathsf {Y}) = {\mathbf {H}}\left( \frac{\mathsf {X}+ r_j \mathsf {Y}}{1 + r_j} \right) = {\mathbf {H}}\left( a(d) + (QM + \frac{Qr_j}{1 + r_j}) d \right) . \end{aligned}$$

By choosing Q and then M suitably, we may ensure that all the \(Qr_j/(1 + r_j)\) are integers of magnitude \(< QM\), which means that

$$\begin{aligned} a(d) + \left( QM + \frac{Qr_j}{1 + r_j}\right) d \in \{a(d),\dots , a(d) + (k-1) d\} \subset A. \end{aligned}$$

That is, \(\mathsf {X}+ r_j \mathsf {Y}\) takes values in \((1 + r_j) \cdot A\). Since \({\mathbf {H}}(\mathsf {W}) \leqslant \log m\) for any random variable \(\mathsf {W}\) taking values in a set of size m, this implies that

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}+ r_j \mathsf {Y}) \leqslant \log \# A. \end{aligned}$$

Combining this with (2.7) and (2.8) we obtain

$$\begin{aligned} \log N \leqslant (1 + \varepsilon ) \log \# A, \end{aligned}$$

or in other words

$$\begin{aligned} \# A \geqslant N^{1/(1 + \varepsilon )}. \end{aligned}$$

Since \(\varepsilon \) was arbitrary, the implication follows.

This completes the proof that Conjectures 1.1, 1’, 1.2 and 1.3 are equivalent.

3 Finite fields

Next we turn to Conjecture 1.4 (n). To demonstrate its equivalence to the first three conjectures, it suffices to show that for each n we have Conjecture 1’ \(\Rightarrow \) Conjecture 1.4 (n) \(\Rightarrow \) Conjecture 1.1.

Conjecture 1’ implies Conjecture 1.4 (n). Suppose that \(A_1 \subset {\mathbf {F}}_p^n\) is a set containing a k-term arithmetic progression with common difference d, for every \(d \in {\mathbf {F}}_p^n\). Define the “unwrapping” map \(\psi : {\mathbf {F}}_p \rightarrow {\mathbf {Z}}\) to be the inverse of the natural projection map from \(\{0,\dots ,p-1\}\) to \({\mathbf {F}}_p\). Define a map \(\psi ^{(n)} : {\mathbf {F}}_p^n \rightarrow {\mathbf {Z}}^n\) by setting \(\psi ^{(n)}(x_1,\dots , x_n) := (\psi (x_1),\dots , \psi (x_n))\).

For each \(d \in {\mathbf {F}}_p^n\), select a progression \(\{ x(d) + \lambda d, \lambda = 0,1,\dots , k-1\}\), lying in \(A_1\). Let \(A_2 \subset {\mathbf {Z}}^n\) be the union of all progressions \(\{ \psi ^{(n)}(x(d)) + \lambda \psi ^{(n)}(d) : \lambda = 0,1,\dots , k-1\}\). By construction, \(A_2 \subset \{0,1,\dots , k(p-1)\}^n\), and \(\pi ^{(n)}(A_2) \subset A_1\), where \(\pi ^{(n)} : {\mathbf {Z}}^n \rightarrow {\mathbf {F}}_p^n\) is the natural map. Since \(\{ 0,1,\dots , k(p-1)\}\) is covered by k discrete intervals of length p, on each of which the projection map \(\pi : {\mathbf {Z}}\rightarrow {\mathbf {F}}_p\) is injective, we see that \(\# A_2 \leqslant k^n \# A_1\).

By construction, \(A_2\) contains a progression of length k and common difference d for \(p^n\) distinct values of d. Whilst \(A_2\) is a subset of \({\mathbf {Z}}^n\), we can create a subset of \({\mathbf {Z}}\) with the same properties by looking at the image of \(A_2\) under the map \(f : {\mathbf {Z}}^n \rightarrow {\mathbf {Z}}\) defined by \(f(x_1,\dots , x_n) = \sum _{i = 1}^n (10 kp)^i x_i\). It follows that \(\# A_2 \geqslant F'_k(p^n)\), and hence \(\# A_1 \geqslant k^{-n} F'_k(p^n)\). In the notation of Conjecture 1.4, this means that \(f_{n,k}(p) \geqslant k^{-n} F'_k(p^n)\). It follows that

$$\begin{aligned} \lim _{p \rightarrow \infty } \frac{\log f_{n,k}(p)}{\log p} \geqslant n \lim _{p \rightarrow \infty } \frac{\log F'_k(p^n)}{\log p^n}, \end{aligned}$$

and so

$$\begin{aligned} \lim _{k \rightarrow \infty }\lim _{p \rightarrow \infty } \frac{\log f_{n,k}(p)}{\log p} \geqslant n \lim _{k \rightarrow \infty }\lim _{p \rightarrow \infty } \frac{\log F'_k(p^n)}{\log p^n}. \end{aligned}$$

Assuming Conjecture 1’ (taking \(N = p^n\)), the right hand side here is precisely n. This implies Conjecture 1.4.

Conjecture 1.4implies Conjecture 1.1. Suppose we have a set \(A_1 \subset {\mathbf {Z}}\) containing a progression of length k and common difference d for each \(d \in \{1,\dots , N\}\). Partition \({\mathbf {Z}}\) into intervals \(I_j := 10 kj N + \{1,\dots , 10k N\}\), \(j \in {\mathbf {Z}}\). Any progression of length k and common difference \(d \in \{1,\dots ,N\}\) is either wholly contained in some \(I_j\), or else is split into two progressions, one in \(I_j\) and the other in \(I_{j+1}\), with one of these having length at least k / 2. It follows that the set \(A_2 \subset {\mathbf {Z}}\) defined byFootnote 4

$$\begin{aligned} A_2 = \bigcup _j \{ (A_1 \cap I_j) - 10 kj N\} \end{aligned}$$

contains a progression of length at least k / 2 and common difference d, for all \(d \in \{1,\dots , N\}\). Manifestly \(\# A_2 \leqslant \# A_1\), and by construction \(A_2\) has the additional property that

$$\begin{aligned} A_2 \subset \{1,\dots , 10k N\}. \end{aligned}$$
(3.1)

Using \(A_2\), we construct a set \(A_3 \subset {\mathbf {Z}}^n\). We will later use this to construct a further set \(A_4 \subset {\mathbf {F}}_p^n\), for a suitable prime p, by projection. To define \(A_3\), let \(M := \lfloor N^{1/n}\rfloor \). Select \(t \in \{-10kN,\dots , 20kN - 1\}\) uniformly at random, and define

$$\begin{aligned} A_3(t) := \left\{ (x_1,\dots , x_n) \in \{0,\dots , M-1\}^n : \sum _{i=1}^n M^{i - 1} x_i \in A_2 + t\right\} . \end{aligned}$$

Suppose that \(d = \sum _{i = 1}^n M^{i - 1} d_i\) with \(0 \leqslant d_i \leqslant M/2k\) for all i. There are at least \((M/4k)^n\) such values of d, and all lie in \(\{0,\dots , N\}\). For each such d there is, by assumption, a progression \(\{ x(d) + \lambda d : \lambda = 0,1,\dots , \lfloor k/2\rfloor - 1\}\) lying in \(A_2\). The progression \(\{ x(d) + t + \lambda d : \lambda = 0,1,\dots , \lfloor k/2\rfloor - 1\}\) then lies in \(A_2 + t\). Write

$$\begin{aligned} S := \left\{ \sum _{i = 1}^n M^{i - 1} s_i : 0 \leqslant s_i < M/2 \; \text{ for } \text{ all } i\right\} . \end{aligned}$$

If it so happens that \(t \in - x(d) + S\) then \(A_3(t)\) contains a progression of length k and common difference \((d_1,\dots , d_n)\), namely \(\{ (s_1,\dots , s_n) + \lambda (d_1,\dots , d_n) : \lambda \in \{0,1,\dots , k - 1\}\}\), where \(x(d) + t = \sum _{i = 1}^n M^{i - 1} s_i\).

Since \(0 \leqslant x(d) \leqslant 10kN\) and \(S \subset \{0,1,\dots , M^n\}\), \(-x(d) + S \subset \{-10k N,\dots , 20k N - 1\}\). It follows that

$$\begin{aligned} {\mathbf {P}}(t \in -x(d) + S) = \frac{1}{30kN} \# S \geqslant \frac{1}{30kN}\left( \frac{M}{2}\right) ^n \gg _{k,n} 1. \end{aligned}$$

(Recall here that t is chosen uniformly at random on \(\{-10kN,\dots , 20kN - 1\}\).) Summing over the \((M/2k)^n \gg _{k,n} N\) choices of d, we see that the expected number of d for which \(t \in -x(d) + S\) is \(\gg _{k,n} N\). Fix some choice of t such that \(t \in -x(d) + S\) for \(\gg _{k,n} N \gg _{k,n} M^n\) values of d, and write \(A_3 := A_3(t)\). Then by construction we have

$$\begin{aligned} \# A_3 \leqslant \# A_2 \leqslant \# A_1, \end{aligned}$$
(3.2)

whilst \(A_3\) contains a progression of length \(\geqslant k/2\) and common difference d for all d in some set \({\mathscr {D}} \subset \{0,\dots , M-1\}^n\), \(\# {\mathscr {D}} \gg _{k,n} M^n\).

Now choose a prime p with \(M \leqslant p < 2M\), and let \(A_4 \subset {\mathbf {F}}_p^n\) be the image of \(A_3\) under the natural projection \( \pi ^{(n)} : {\mathbf {Z}}^n \rightarrow {\mathbf {F}}_p^n\). We have

$$\begin{aligned} \# A_4 = \# A_3, \end{aligned}$$
(3.3)

and moreover \(A_4\) contains a progression of length k and common difference d for all \(d \in \pi ^{(n)}({\mathscr {D}})\), that is to say for \(\gg _{n,k} N \gg _{n,k} p^n\) values of d. By a standard argument (taking random translations of \(\pi ^{(n)}({\mathscr {D}})\), see Corollary 6.4 for details) there is a further set \(A_5 \subset {\mathbf {F}}_p^n\),

$$\begin{aligned} \# A_5 \ll _{n,k} (\log p) \# A_4, \end{aligned}$$
(3.4)

containing a progression of length k and common difference d, for all\(d \in {\mathbf {F}}_p^n {\setminus } \{0\}\). Tracing back through (3.4), (3.3), (3.2) we see that

$$\begin{aligned} F_k(N) \gg _{k,n} \frac{1}{\log p} f_{n,k}(p), \end{aligned}$$

where \(p = p(N) \sim N^{1/n}\) is some prime. It follows that

$$\begin{aligned} \lim _{N \rightarrow \infty }\frac{\log F_k(N)}{\log N} \geqslant \lim _{N \rightarrow \infty }\frac{\log f_{n,k}(p(N))}{n \log p(N)}. \end{aligned}$$

Assuming Conjecture 1.4 (n), the limit on the right is 1. This concludes the proof that Conjecture 1.4 (n) implies Conjecture 1.1.

Before leaving this topic, we remark that it is quite possible that very strong bounds such as

$$\begin{aligned} f_{p^{\eta }, 1}(p) \geqslant p/2 \end{aligned}$$
(3.5)

are true, provided \(p \geqslant p_0(\eta )\) is large enough. This issue is strongly hinted at, if not explicitly conjectured, in [1]. It is pointed out there that such bounds imply vastly more than is currently known about the purely arithmetic problem of bounding the least quadratic nonresidue modulo p.

Whilst a bound of the form (3.5) is not known to imply the arithmetic Kakeya conjecture (the progressions are of length \(p^{\eta }\), rather than of bounded size), the arguments of Bourgain may be adapted to show that it does imply the Euclidean Kakeya conjecture. Further details may be found in lecture notes of the first author [9, Section 10].

It is quite interesting that the innocent-looking statement (3.5) implies two famous unsolved problems in completely different mathematical areas.

4 A problem of Erdős and Selfridge

Finally, we turn to Conjecture 1.5. In fact, we prove the following rather tight connection between Conjecture 1’ and Conjecture 1.5.

Proposition 4.1

Write \(G_k(N)\) for the minimum, over all choices \(p_1< \dots < p_N\) of primes and all intervals I of length \(k p_N\), of \(\# \big ( I \cap \bigcup _{i = 1}^N p_i {\mathbf {Z}}\big )\). Then \(F'_k(N) \leqslant G_k(N) \leqslant k F'_k(N)\). In particular, Conjectures 1’ and 1.5 are equivalent.

Proof

Suppose first we have a set of primes \(p_1< \dots < p_N\) and an interval I of length \(kp_N\) so that \(\# A = G_k(N)\), where \(A = \bigcup _{i = 1}^N\{ x \in I : p_i | x \}\). Note that A obviously contains a progression of length k and common difference \(p_i\), for each i, and therefore \(F'_k(N) \leqslant G_k(N)\).

In the other direction, suppose we have a set A attaining the bound \(F'_k(N)\), that is to say \(\# A = F'_k(N)\) and A contains, for \(i = 1,\dots , N\), a progression \(\{ a_i + jd_i : j = 0,1,\dots , k-1\}\). By translating if necessary, we may assume that A consists of positive integers. Let \(\delta \in (0,\frac{1}{2})\) be a quantity to be specified shortly. By the theorem of the first author and T. Tao [10, Theorem 1.2], we may find positive u and v such that all of the numbers \(v, u+v, \dots d_N u + v\) are prime and lie in some interval \([(1 - \delta ) X, X]\), \(X \geqslant 100\). Set \(p_i := d_i u + v\). Note that \(\frac{v}{u+v} \geqslant 1 - \delta \), which rearranges as \(\frac{v}{u} \geqslant \frac{1}{\delta } - 1\), hence

$$\begin{aligned} \frac{v}{u} > 4\max A \end{aligned}$$
(4.1)

provided that \(\delta \) is chosen sufficiently small. Note also that

$$\begin{aligned} \frac{p_i}{p_N}\geqslant \frac{v}{v + ud_N} = \frac{1}{1 + \frac{u}{v} d_N} \geqslant 1 - \frac{1}{4k} \end{aligned}$$
(4.2)

if \(\delta \) is small enough. In particular if \(\delta \) is small enough then we have

$$\begin{aligned} p_i> \frac{3}{4}p_N \geqslant \frac{1}{2} p_N + \frac{1}{4}v > \frac{1}{2}p_N + u \max A \end{aligned}$$
(4.3)

by (4.1).

Define \(A' := u \cdot A + \{0,v,2v,\dots , (k-1)v\}\). The cardinality of \(A'\) satisfies \(\# A' \leqslant k F'_k(N)\), and

$$\begin{aligned} A' \supset \bigcup _{i = 1}^N\{ ua_i + jp_i : j = 0,1,\dots , k-1\} \end{aligned}$$
(4.4)

for \(i = 1,\dots , N\). By the Chinese remainder theorem we may find w so that \(p_i | w + ua_i\) for \(i = 1,\dots , N\).

Set \(I := w - \lfloor \frac{1}{2} p_N\rfloor + \{1,2,\dots , kp_N\}\). Obviously I is an interval of length \(kp_N\). Let \(i \in \{1,\dots , N\}\). We claim that \(w + ua_i + jp_i \in I\) for an integer j if and only if \(j \in \{0,1,\dots , k-1\}\). Since \(w + ua_i + p_i{\mathbf {Z}}= p_i {\mathbf {Z}}\), this implies that

$$\begin{aligned} I \cap p_i {\mathbf {Z}}= \{ w + ua_i + jp_i : j = 0,1,\dots , k-1\}, \end{aligned}$$

and hence by (4.4)

$$\begin{aligned} I \cap \bigcup _{i = 1}^N p_i {\mathbf {Z}}\subset w + A', \end{aligned}$$

whence

$$\begin{aligned} G_k(N) \leqslant \# \left( I \cap \bigcup _{i = 1}^N p_i {\mathbf {Z}}\right) \leqslant \# A' \leqslant k F'_k(N). \end{aligned}$$

It remains to prove the claim. To prove the if implication, it suffices in view of (4.4) to show that \(w + A' \subset I\). However it is obvious that \(\min (w + A') \geqslant \min I\) (since all elements of \(A'\) are positive) and moreover

$$\begin{aligned} \max (w + A')&\leqslant w + u \max A + (k-1) v \\&< w + \left( k- \frac{1}{2}\right) v \;\; \text{ by } (4.1) \\&\leqslant w + \left( k - \frac{1}{2}\right) p_N \\&\leqslant \max I. \end{aligned}$$

This establishes the if direction of the claim. To establish the only if direction, it suffices to show that

$$\begin{aligned} w + ua_i - p_i < \min I \end{aligned}$$
(4.5)

and that

$$\begin{aligned} w + ua_i + kp_i > \max I. \end{aligned}$$
(4.6)

However by (4.3) we have

$$\begin{aligned} w + ua_i - p_i < w + u(a_i - \max A) - \frac{1}{2}p_N \leqslant w - \frac{1}{2} p_N \leqslant \min I, \end{aligned}$$

so (4.5) does hold. Also,

$$\begin{aligned} w + ua_i + kp_i&> w + kp_i \; \; \text{ since } A \subset {\mathbf {N}}\\&\geqslant w + \left( k - \frac{1}{4}\right) p_N \; \; \text{ by } (4.2) \\&\geqslant w + kp_N - \left\lfloor \frac{1}{2} p_N \right\rfloor = \max I, \end{aligned}$$

the last step being a consequence of the fact that \(p_N \geqslant (1-\delta ) X \geqslant 50\). Thus (4.6) also holds, and this completes the proof of the claim. \(\square \)

Remark

The use of the theorem of the first author and Tao is a little excessive. One could do without it using simpler arguments if one was prepared to settle for logarithmic losses.

5 Small unions of progressions

In this section we prove Theorem 1.7. Write \(3 = p_1< p_2 < \dots \) for the odd primes, and set \(Q := \prod _{i = 1}^m p_i\), where \(m = \lceil 10 \log k\rceil \). Note that \(Q = k^{O(\log \log k)}\).

Define a set S to be the union of all progressions \(\{ x_d + j d : j = 0,1,\dots , k-1\}\) where, for \(d \in \{1,\dots , Q-1\}\), \(x_d\) is the unique element of \(\{1,\dots ,Q\}\) congruent to \(d^2 ({\text {mod}}\, Q)\). Evidently, S contains a progression of length k and common difference d, for all \(d \in \{0,1,\dots , Q-1\}\).

Fix \(j \in \{0,1\dots , k-1\}\). For each i we have

$$\begin{aligned} x_d + jd \equiv d^2 + jd \equiv \left( d + \frac{j}{2}\right) ^2 - \frac{j^2}{4} ({\text {mod}}\, p_i), \end{aligned}$$

and so \(x_d + jd ({\text {mod}}\, p_i)\) takes values in a set of size \(\frac{1}{2}(p_i + 1)\) as d varies. Therefore \(x_d + jd ({\text {mod}}\, Q)\) takes values in a set of size \(\prod _{i = 1}^m \frac{1}{2}(p_i + 1)\). Since, additionally, \(0 < x_d + jd \leqslant kQ\), \(x_d + jd\) takes values in a set of size \(k \prod _{i = 1}^m\frac{1}{2}(p_i + 1)\). Therefore

$$\begin{aligned} \#S \leqslant k^2 \prod _{i = 1}^m \frac{1}{2}(p_i + 1) = k^2 2^{-m} Q \prod _{i = 1}^m \left( 1 + \frac{1}{p_i}\right) . \end{aligned}$$

Recalling that \(m \sim 10\log k\), and using the bound \(\prod _{i = 1}^m (1 + \frac{1}{p_i}) \ll \log m \ll k\), we see that

$$\begin{aligned} \# S \ll k^{-7} Q \end{aligned}$$

and so

$$\begin{aligned} \# S \leqslant Q^{1 - \frac{c}{\log \log k}} \end{aligned}$$

if k is sufficiently large, for some absolute \(c > 0\).

Now let n be an arbitrary positive integer, set \(N_n := Q^n\), and consider the set

$$\begin{aligned} A_n := \{ s_0 + s_1 Q + \dots + s_{n-1} Q^{n-1} : s_0,\dots , s_{n-1} \in S\}. \end{aligned}$$

Then \(\# A_n \leqslant (\# S)^n \leqslant N_n^{1 - \frac{c}{\log \log k}}\). The set \(A_n\) contains a progression of length k and common difference \(d_0 + d_1 Q + \dots + d_{n-1}Q^{n-1}\) for any choice of \(d_i \in \{0,1,\dots , Q-1\}\), or in other words for all \(d \in \{0,\dots , N_n - 1\}\).

Finally, suppose N is an arbitrary positive integer. Choose n minimal so that \(N_n > N\), and set \(A := A_n\). Then A contains a progression of length k and common difference d, for all \(d \in \{1,\dots , N\}\). Moreover,

$$\begin{aligned} \# A \leqslant N_n^{1 - \frac{c}{\log \log k}} \leqslant (QN)^{1 - \frac{c}{\log \log k}} \ll _k N^{1 - \frac{c}{\log \log k}}. \end{aligned}$$

The result follows.

6 Entropy inequalities in positive characteristic

In this section we give the proof of Theorem 1.8. Suppose that \(\mathsf {X}\) and \(\mathsf {Y}\) are two \({\mathbf {F}}_p^{\infty }\)-valued random variables, both taking finitely many values. Suppose that

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}- \mathsf {Y}) \geqslant (1 + \varepsilon ) \sup _{r \ne -1} {\mathbf {H}}(\mathsf {X}+ r\mathsf {Y}). \end{aligned}$$
(6.1)

Our aim is to prove that \(\varepsilon = O(\frac{1}{\log p})\), which immediately implies Theorem 1.8.

The initial phases of the argument mirror the deduction of Conjecture 1.2 from Conjecture 1.3. We may assume that there is some q such that \(q {\mathbf {P}}((\mathsf {X}, \mathsf {Y}) = (x, y)) \in {\mathbf {Z}}\) for all (x, y); if (6.1) can be established in this case, uniformly in q, then the general result follows by an easy approximation argument on letting \(q \rightarrow \infty \).

Now let m be very large, write \(n = mq\), and construct a set \(B^{(n)} \subset ({\mathbf {F}}_p^{\infty })^{qm} \times ({\mathbf {F}}_p^{\infty })^{qm}\) as follows. Let it consist of all pairs \(((x_1,\dots , x_{mq})\), \((y_1,\dots , y_{mq}))\) for which

$$\begin{aligned} \# \{ i : (x_i, y_i) = (x,y)\} = mq {\mathbf {P}}((\mathsf {X}, \mathsf {Y}) = (x,y)). \end{aligned}$$

By arguments essentially the same as we saw before,

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}+ r\mathsf {Y}) = \frac{1}{n} \log \pi _r\left( B^{(n)}\right) + o_{n \rightarrow \infty }(1). \end{aligned}$$

Hence, taking m sufficiently large (and observing that \(({\mathbf {F}}_p^{\infty })^{qm}\) is isomorphic to \({\mathbf {F}}_p^{\infty }\) as a vector space), we obtain arbitrarily large sets \(B \subset {\mathbf {F}}_p^{\infty } \times {\mathbf {F}}_p^{\infty }\) such that

$$\begin{aligned} \# \pi _{-1}(B) \geqslant \sup _{r \ne -1} (\# \pi _{r}(B))^{1 + \varepsilon /2}. \end{aligned}$$
(6.2)

Note in particular that \(\pi _{-1}(B)\) becomes arbitrarily large.

For such a B, we construct a finite set \(A \subset {\mathbf {F}}_p^{\infty }\) as follows. If \((x,y) \in B\) and \(x \ne y\), include the entire progression (line) through x and y in A. The points on this line are \(\frac{x + ry}{1 + r}\), for \(r \ne -1\), and y. Therefore

$$\begin{aligned} A \subset \pi _{\infty }(B) \cup \bigcup _{r \ne -1} \frac{1}{1 + r} \cdot \pi _r(B), \end{aligned}$$

and therefore

$$\begin{aligned} \# A \leqslant p \sup _{r \ne -1} \pi _r(B). \end{aligned}$$

On the other hand, A contains a progression of length p (line) and common difference d, for every \(d \in \pi _{-1}(B) {\setminus } \{0\}\). Thus, writing \(N := \pi _{-1}(B) - 1\), we have

$$\begin{aligned} \# A \ll _p N^{\frac{1}{1 + \varepsilon /2}}. \end{aligned}$$
(6.3)

On the other hand we have the following result, whose proof we will supply shortly.

Proposition 6.1

Suppose that \(A \subset {\mathbf {F}}_p^{\infty }\) is a finite set containing a progression of length p(that is, a line) and common difference d, for all d in some set of size N. Then \(\# A \gg _p N^{1 - \frac{\log 2}{\log p} - o(1)}\).

Combining Proposition 6.1 with the construction of A satisfying (6.3) immediately gives the desired upper bound \(\varepsilon = O(\frac{1}{\log p})\), thereby concluding the proof of Theorem 1.8.

It remains to prove Proposition 6.1.

Proof of Proposition 6.1

Set \(A_1 := A\). In its initial stages, the proof of this result goes along rather similar lines to that of Proposition 2.1, only it is rather easier. The use of random projections in a similar context may be found in [7, §3]. Let n be the smallest positive integer for which \(p^n \geqslant N\).

Since \(A_1\) is finite, it is contained in some copy of \({\mathbf {F}}_p^M\). Let \(\pi : {\mathbf {F}}_p^M \rightarrow {\mathbf {F}}_p^n\) be a random linear map, selected by choosing the images of the basis vectors \(e_1,\dots , e_M\) uniformly at random from \({\mathbf {F}}_p^n\). Set \(A_2 := \pi (A_1)\); evidently \(\# A_2 \leqslant \# A_1\). Let \({\mathscr {D}}\) be the set of common differences of progressions (of length p) lying in \(A_1\). Then \(A_2\) contains a progression of length p and common difference \(\pi (d)\), for every \(d \in {\mathscr {D}}\).

Put some arbitrary order \(\prec \) on \({\mathscr {D}}\), and suppose that \(d \prec d'\). Then \(\pi (d) = \pi (d')\) if and only if \(\pi (d - d') = 0\). However, \(\pi (d - d')\) is uniformly distributed in \({\mathbf {F}}_p^n\), and so the probability of this happening is \(p^{-n}\). It follows that the expected number of pairs \((d, d')\) with \(d \prec d'\) and \(\pi (d) = \pi (d')\) is \(p^{-n} \left( {\begin{array}{c}N\\ 2\end{array}}\right) \leqslant \frac{1}{N} \left( {\begin{array}{c}N\\ 2\end{array}}\right) \leqslant N/2\). Pick some map \(\pi \) for which the number of such pairs is at most N. For each \(v \in {\mathbf {F}}_p^n\), write \(f(v) := \# \pi ^{-1}(v)\). Then we have \(\sum _v \left( {\begin{array}{c}f(v)\\ 2\end{array}}\right) \leqslant N/2\), from which we obtain, since \(\sum _v f(v) = N\), that \(\sum _v f(v)^2 \leqslant 2N\). By Cauchy–Schwarz,

$$\begin{aligned} N^2 = \left( \sum _v f(v)\right) ^2 \leqslant \# \{ v : f(v) \ne 0\} \sum _v f(v)^2, \end{aligned}$$

and therefore there are at least N / 2 values of v for which \(f(v) \ne 0\). From the choice of n it is clear that \(p^n \leqslant pN\), and so at least \((\# {\mathbf {F}}_p^n)/2p\) elements of \({\mathbf {F}}_p^n\) lie in the image of \(\pi \), or in other words are common differences of progressions in \(A_2\).

By a random translation argument (see Corollary 6.4), there is a set \(A_3 \subset {\mathbf {F}}_p^n\), \(\# A_3 \ll (n p \log p) \# A_2\), containing a line in every direction. That is, \(A_3\) is a finite field Besicovitch set.

Now we bring in bounds on the size of such sets of a strength which, famously, are available in the finite field setting but not in characteristic zero. By the main result of [6] we have \(\# A_3 \geqslant (p/2)^n = (p^n)^{1 - \frac{\log 2}{\log p}} \geqslant N^{1 - \frac{\log 2}{\log p}}\). The proposition follows. \(\square \)

Remarks

Note that here it was crucial to have an effective lower bound on the size of Kakeya sets for fixed p but with \(n \rightarrow \infty \). For this, the celebrated work of Dvir [5] on the Kakeya problem would not suffice. However (at the cost of weakening the exponents slightly) we could have used the main result of [19], which has a slightly simpler proof than that of [6].

The \(O(\frac{1}{\log p})\) term in Theorem 1.7 is sharp. To see this, pick \(a,b,b'\) independently and uniformly from \({\mathbf {F}}_p\), and define random variables \(\mathsf {X}, \mathsf {Y}\) taking values in \({\mathbf {F}}_p^2\) by

$$\begin{aligned} \mathsf {X}= (a + b, ab), \quad \mathsf {Y}= (a + b', ab'). \end{aligned}$$

Then

$$\begin{aligned} \mathsf {X}- \mathsf {Y}= (b - b', a(b - b')), \end{aligned}$$

which is almost uniformly distributed on \({\mathbf {F}}_p^2\): a short calculation gives

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}- \mathsf {Y}) = 2 \log p + O\left( \frac{\log p}{p}\right) . \end{aligned}$$

By contrast, if \(r \ne -1\) then

$$\begin{aligned} \frac{\mathsf {X}+ r\mathsf {Y}}{1 + r} = \left( a + \frac{b + rb'}{1 + r}, a \cdot \frac{b + rb'}{1 + r} \right) , \end{aligned}$$

and so \(\mathsf {X}+ r\mathsf {Y}\) is supported on a dilate of the set \(V := \{(u + v, uv) : u,v \in {\mathbf {F}}_p\}\), which has cardinality \(\frac{1}{2}p^2 + O(p)\). Therefore

$$\begin{aligned} {\mathbf {H}}(\mathsf {X}+ r\mathsf {Y}) \leqslant 2 \log p - \log 2 + O\left( \frac{\log p}{p}\right) . \end{aligned}$$

Cognoscenti will recognise V as being equivalent to the well-known construction of optimal Kakeya sets in \({\mathbf {F}}_p^2\), due to Mockenhaupt and Tao [15].