Large deviations in the perceptron model and consequences for active learning

Supervised learning consists in presenting a parametric function (often a neural network) with a series of samples (samples) and labels, and adjusting (training) the parameters (network weights) so as to match the network output with the labels as closely as possible. Active learning (AL) is concerned with choosing the most informative samples so that the training requires the least number of labeled samples to reach the same test accuracy. AL is relevant in situations where the potential set of samples is large, but obtaining the labels is expensive (computationally or otherwise). There exist many strategies for AL, see e.g. [1] for a review. In membership-based AL [2–4] the algorithm is allowed to query the label of any sample, most often one it generates itself. In stream-based AL [5] an infinite sequence of samples is presented to the learner which can decide whether or not to query its label. In pool-based AL, which is the object of the present work, the learner can only query samples that belong to a pre-existing, fixed pool of samples. It therefore needs to choose according to some strategy which samples to query so as to have the best possible test accuracy.

Pool-based AL is relevant for many machine learning applications, e.g. because not every possible input vector is of relevance. A beautiful recent application of AL is in computational chemistry [6] where a neural network is trained to predict inter-atomic potentials. In this case the pool of data is large and consists in all possible alloys, but not of arbitrary input vectors, and labelling is extremely expensive, as it demands resource-intensive ab-initio simulations. Consequently, only a limited number of samples can be labeled, i.e. one only possesses a certain budget for the cardinal of the training set. Another setting where a cheap large pool of input data is readily available but labelling is expensive is drug discovery [7], where given a target molecule one aims to find new compounds among the pool able to bind it. Another example would be on text classification [8–10], where labelling a text requires non-negligible human input, while a large pool of texts is readily available on the internet. Establishing efficient pool-based AL procedures in this case implies to select a priori the most informative data samples for labelling.

Main-stream works on AL focus on designing heuristic algorithmic strategies in a variety of settings, and analyzing the performance thereof. It is very rarely known what are the information-theoretic limitations an AL algorithm can face and hence evaluating the distance from optimality is mostly an open question. The main contribution of the present work is to provide a toy model that is at the one hand challenging for AL, and at the same time where the optimal performance of pool-based AL can be computed and heuristic algorithms hence evaluated and bench-marked against the optimal solution. To our knowledge, this is the first work to derive optimal performance results for pool-based AL procedures are computed. More specifically we study the random perceptron model [11]. The available pool of samples is assumed to be i.i.d. vectors following a normal distribution, the teacher generating the labels is taken to be also a perceptron with the vector of teacher-weights having i.i.d. normal components. We compute the large deviation function for how likely it is to find a subset of the samples that leads to a given learning accuracy. Our results are obtained through the non-rigorous yet asymptotically exact (under appropriate probabilistic assumptions on the data) replica method, from theoretical statistical physics [12]. While the presented analysis is based on the so-called replica symmetric (RS) ansatz, we also provide a stability analysis of our results against possible symmetry-breaking effects. Providing a rigorous proof of the obtained results or turning them into rigorous bounds would be a natural, and rather challenging, next step. In the algorithmic part of this work we benchmark several existing algorithms and also propose two new algorithms relying on the approximate-message-passing algorithm for estimation of the label uncertainty for yet unlabeled sample, showing that they closely achieve, in the studied cases, the relevant information-theoretic limitations.

The paper is organized as follows: the problem is defined and related work discussed in section 1.2. In section 2, we propose a measure to quantify the informativeness of given subsets of samples. In section 3, we derive the large deviation function over all possible subset choices and deduce performance boundaries that apply to any pool-based AL strategy. In section 4 we then summarize our results on the large deviations and support them with numerical simulations. In section 5, we then compare these theoretical results with the performance of existing AL algorithms and propose two new ones, based on approximate-message-passing.

A natural modeling framework for analyzing learning processes and generalization properties is the so-called teacher-student (or planted) perceptron model [13], where the input samples are assumed to be random i.i.d. vectors, and the ground truth labels are assumed to be generated by a neural network (denoted as the teacher) belonging to the same hypothesis class as the student-neural-network. In this work we will restrict to single-layer neural networks (without hidden units) for which this setting was defined and studied in [14]. Specifically we collect the input vectors into a matrix $\boldsymbol{F} \in {\mathbb R}^{P \times N}$ where N is the dimension of the input space and P is the number of samples. The teacher generating the labels, called teacher perceptron, is characterized by a teacher-vector of weights $\textbf{x}^0$ and produces the label vector $\boldsymbol{Y}\in\mathbb{R}^{P}$ according to $\textbf{Y} = \mathrm{sign} (\textbf{F}\cdot \textbf{x}^0)$. Learning is then done using a student perceptron and consists in finding a vector x so that for the training set $\textbf{F}$ we have as closely as possible $\textbf{Y} = \mathrm{sign} (\textbf{F}\cdot \textbf{x})$. The relevant notion of error is the test accuracy (generalization error) measuring the agreement between the teacher and the student on a new sample not presented in the training set. Since both teacher and student possess the same architecture, the training process can be rephrased in terms of an inference problem (as discussed for instance in [13]): the student aims to infer the teacher weights, used to generate the labels, from the knowledge of a set of input-output associations. This scenario allows for nice geometrical insights (see for example [15]), as the generalization properties are linked to the distance in weight space between teacher and student functions. Note that, in the case of a noiseless labelling process, the teacher–student scenario guarantees that perfect training is always possible.

Active learning was previously studied in the context of the teacher–student perceptron problem. Best known is the line of work on query by committee (QBC) [4, 16, 17], dealing with the membership based AL setting, i.e. where the samples are picked one by one into the training set and can be absolutely arbitrary N-dimensional vectors. The AL is in that case more a strategy for designing the samples rather than one for selecting them smartly from an predefined set. In the original work [4] the new samples are chosen so that a committee of several student-neural-networks has the maximum possible disagreement on the new sample. The paper shows that in this way one can reach a generalization error that decreases exponentially with the size of the training set, while for a random training set the generalization error can decrease only inversely proportionally to the size of the set [15]. However, in many practical applications the possible set of samples to be picked into the training set is not arbitrarily big, e.g. not every input vector represents an encoding of a molecular structure. We hence argue that the pool-based AL, studied in the present paper, where the samples are selected from a pre-defined set is of larger relevance to many applications.

The theoretical part of this paper is presented for a generalization of the perceptron model, specifically the for the random teacher–student generalized linear models (GLM), see e.g. [18]. The teacher-student GLM setup is an inference problem over a dataset $\{\boldsymbol{F}^\mu,y^\mu\}_{\mu = 1}^{P}$. The labels $y^\mu$ are generated from the data $\boldsymbol{F}^\mu$ through a probability distribution $P_{\mathrm{out}}(y^\mu|\boldsymbol{F}^\mu\cdot \boldsymbol{x^0})$, where $\boldsymbol{x^0}$ is a fixed vector, generated once and for all from a prior measure P_X (·). Given the dataset $\{\boldsymbol{F}^\mu,y^\mu\}_{\mu = 1}^{P}$, the goal of the inference problem is to infer the weight vector $\boldsymbol{x^0}$ that was used to generate the labels. A completely equivalent formulation of the teacher student GLM setting is to consider that the labels $y^\mu$ have been generated by a single-layer 'teacher' neural network with weight vector $\boldsymbol{x^0}$. A 'student' network sharing the exact same architecture is then trained on the resulting dataset $\{\boldsymbol{F}^\mu,y^\mu\}_{\mu = 1}^{P}$. Since a common architecture is shared between the teacher and student network, training is in this case equivalent to the student trying to match its own weight vector with the teacher weights $\boldsymbol{x^0}$. An instance of a GLM is thus specified by a prior measure P_X (·) on the weights $\boldsymbol{x}$, from which the true generative model is assumed to be sampled, and an output channel measure $P_{\mathrm{out}}(\cdot|\cdot)$, defining the generative process for the labels $y^\mu$ given the pre-activations $\boldsymbol{F}^\mu\cdot\boldsymbol{x}$. In the part where results of this work are presented we focus on the prototypical case of the noiseless continuous perceptron, where $P_{X}(x) = e^{-\frac{x^{2}}{2}}/\sqrt{2\pi}$ and $P_{\mathrm{out}}(y|h) = \delta(y-\mathrm{sign}(h))$ where for example µ we have $h^\mu = \textbf{F}^\mu \cdot \textbf{x}$. Moreover, we will consider the setting where the learning model is matched to the generative model and thus the student has perfect knowledge of the correct form of the two above defined measures.

The pool-based AL task can now be more formally stated as follows: given a set of N-dimensional samples $\mathcal{S} = \{\boldsymbol{F}^{\mu}\}$ of cardinality $|\mathcal{S}| = P = \alpha N$, the goal is to select and query the labels of a subset $S\in \mathcal{S}$ of cardinality $|S| = nN$, $0\lt n\leqslant\alpha$, according to some AL criterion. We will refer to n as the budget of the student. The true labels are then obtained through $y^\mu\sim P_{\mathrm{out}}(y^\mu|\boldsymbol{F}^\mu \cdot \boldsymbol{x}^0)$, $\boldsymbol{x}^{0}\sim P_{X}(\boldsymbol{x}^0)$. Henceforth measures with vector arguments are understood to be products over the coordinates of the corresponding scalar measures. For technical reasons, we rely on the strong (but customary) assumption that the samples are i.i.d. Gaussian distributed, $F_{i}^{\mu}\sim\mathcal{N}(0,1)$, ∀i ∈ {1, ..., N},  ∀µ ∈ {1, ..., P}. Note that, while this assumption implies that the full set $\mathcal{S}$ of input data is generally unstructured and uncorrelated, it does not prevent non-trivial correlations to appear in any smaller labeled subset S, selected through an AL procedure.

In pool-based AL settings, it is assumed that the student has a fixed budget n for building its training set, i.e. that only up to nN labels can be queried for training. The AL goal is to select, among the pool $\mathcal{S}$ of available samples, the nN most informative labels, to present to the student so that the latter achieves the best possible generalization performance. While many criteria of informativeness have been considered in the literature, see e.g. [1], in the teacher-student setting there exist a natural measure of informativeness, which we shall define in the next section.

This section details the computation of the free entropy (7) for the generic case of GLM, with arbitrary teacher/student posteriors/priors. The specialization to the particular case of the teacher-student perceptron with no mismatch (Bayes-optimal) will be carried out section 3.2. Our computation borrows from [18, 23] which study the simple measure. Because we study large deviations, our formalism has some semblance with one-step replica breaking (1RSB) equations, a discussion of which can be found for example in [24–26].

3.1.1. Notations and assumptions

We consider a student GLM [23] with N-dimensional weights learning from αN samples $\boldsymbol{F}^{\mu}\in\mathbb{R}^{N}$ stacked in a matrix $\boldsymbol{F}\in\mathcal{M}_{\alpha N,N}(\mathbb{R})$ and the corresponding labels $y^{\mu}$ stacked into $\boldsymbol{Y}\in \mathbb{R}^{\alpha N}$. We assume the student–teacher (or planted) setting [13], where the labels are generated by the ground truth (teacher weights) $\boldsymbol{x}^{0}$ with the channel measure $\overline{P_{\mathrm{out}}}(y^{\mu}|\boldsymbol{F}^{\mu}\cdot\boldsymbol{x^{0}})$. The teacher weight itself is drawn with prior $\overline{P_{X}}(\cdot)$. Given $\boldsymbol{F}$ and $\boldsymbol{Y}$, the student perceptron is trained so that its own weight vector $\boldsymbol{x}$ tries to match the ground truth $\boldsymbol{x}^{0}$. The inference is carried out with the student prior P_X (·) and posterior $P_{\mathrm{out}}(\boldsymbol{Y}|\boldsymbol{F}\cdot\boldsymbol{x})$. Note that the cases where $P_{X}(\cdot)\ne\overline{P_{X}}(\cdot)$ or $P_{\mathrm{out}}(\cdot)\ne\overline{P_{\mathrm{out}}}(\cdot)$ mean that the student ignores the precise Markov chain wherefrom the labels are generated, as discussed in section 1.2, see also [13]. The likelihood that a vector $\boldsymbol{x}$ is the ground truth vector is then $P_{X}(\boldsymbol{x})P_{\mathrm{out}}(\boldsymbol{Y}|\boldsymbol{F}\cdot\boldsymbol{x})$. In this case the Gardner volume reads:

$$\begin{equation} v = \left(\int d\boldsymbol{x}P_{X}(\boldsymbol{x})P_{\mathrm{out}}(\boldsymbol{Y}|\boldsymbol{F}\cdot\boldsymbol{x})\right)^{\frac{1}{N}}. \end{equation} \tag{ 10 }$$

The smaller v, the easier the student inference, see section 1.2. The validity of the Gardner volume as a measure of informativeness is justified for the Bayes-optimal perceptron in section 2.

We consider here pool-based AL, where only a subset S of the pool $\mathcal{S} = \{\boldsymbol{F}^{\mu}\}_{1\leqslant \mu\leqslant\alpha N}$ is used for training. The choice of subset can be conveniently parametrized by the Boolean $\sigma_{\mu}\in\{0,1\}$, where $\sigma_{\mu} = 1$ means sample $\boldsymbol{F}_{\mu}$ is used, while $\sigma_{\mu} = 0$ means $\boldsymbol{F}_{\mu}$ is not selected. For a given budget $0\leqslant n = \frac{1}{N}|S| = \frac{1}{N}\sum\limits_{\mu = 1}^{\alpha N}\sigma_{\mu}\leqslant \alpha$, we intend to find the selection S that minimizes the Gardner volume $v(\sigma_{\mu})$, viz. that allows the best student guess. To do this we shall compute the complexity Σ(n, v), with $e^{N\Sigma(n,v)}$ the number of ways to select nN samples so that the Gardner volume associated with the training of the student is v, as in section 3.

To simplify, the samples are taken to be identically and independently distributed according to a normal distribution $\forall(i,\mu),~F^{\mu}_{i}\overset{d}{ = }\mathcal{N}(0,\frac{1}{\sqrt{N}})$. Moreover all measures over vectors are assumed to be separable, that is factorizable as a product of identical measures over the components. Notation-wise $P_{X}(\boldsymbol{x})$ for example is therefore understood to mean $\prod\limits_{i = 1}^{N}P_{X}(x_{i})$.

3.1.2. Replica trick

The goal is to compute the averaged log partition function (free entropy in statistical physics terms):

$$\begin{equation} \Phi(\beta,\phi) = \mathbb{E}_{\boldsymbol{F},\boldsymbol{Y},\boldsymbol{x^{0}}}\frac{1}{N}\mathrm{ln}\Xi = \mathbb{E}_{\boldsymbol{F},\boldsymbol{Y},\boldsymbol{x^{0}}}\frac{1}{N}\mathrm{ln}\sum\limits_{\sigma_{\mu}}\left[\int d\boldsymbol{x}P_{X}(\boldsymbol{x})\prod\limits_{\mu = 1}^{\alpha N}P_{\mathrm{out}}(y^{\mu}|\boldsymbol{F}^{\mu}\cdot\boldsymbol{x})^{\sigma_{\mu}}\right]^{\beta}e^{\phi \sum\limits_{\mu}\sigma_{\mu}},\end{equation} \tag{ 11 }$$

β can be seen as an inverse temperature and φ as a chemical potential, see section 3. Note that the selection variables $\{\sigma_{\mu}\}$ play in the grand-canonical partition function (11) the role of an annealed disorder in disordered systems terminology [26], and shall be sometimes referred to as such in the following.

The standard way of taking care of the logarithm in equation (11) is the replica trick [3.1, 26, 28],

$$\begin{equation} \Phi(\beta,\phi) = \underset{s\rightarrow0}{\mathrm{lim}}\frac{1}{s}\mathbb{E}_{\boldsymbol{F},\boldsymbol{Y},\boldsymbol{x^{0}}}\Xi^{s}. \end{equation} \tag{ 12 }$$

To compute $\mathbb{E}_{\boldsymbol{F},\boldsymbol{Y},\boldsymbol{x^{0}}}\Xi^{s}$, one needs to further replicate β times to care for the power β involved in the summand in equation (11):

$$\begin{align} \mathbb{E}_{\boldsymbol{F},\boldsymbol{x^{0}}}\Xi^{s}& = \mathbb{E}_{\boldsymbol{F}}\int d\boldsymbol{x}^{0}\overline{P_{X}}(\boldsymbol{x}^{0})\int d\boldsymbol{y}\prod\limits_{\mu = 1}^{\alpha N}\overline{P_{\mathrm{out}}}(y^{\mu}|\boldsymbol{F}^{\mu}\cdot\boldsymbol{x}^{0})\nonumber\\ &\quad \times\sum\limits_{S_{a}^{\mu}}\int \prod\limits_{a = 1}^{s}\prod\limits_{\alpha = 1}^{\beta}(d\boldsymbol{x}^{a\alpha}P_{X}(\boldsymbol{x}^{a\alpha}))\prod\limits_{a\alpha}\prod\limits_{\mu}P_{\mathrm{out}}(y^{\mu}|\boldsymbol{F}^{\mu}\cdot\boldsymbol{x}^{a\alpha})^{\sigma^{a}_{\mu}}e^{\phi \sigma^{a}_{\mu}}\nonumber\\ & = \mathbb{E}_{\boldsymbol{F}}\sum\limits_{S_{a}^{\mu}}\int \prod\limits_{a\alpha}(d\boldsymbol{x}^{a\alpha}P_{X}(\boldsymbol{x}^{a\alpha}))e^{-\phi}\prod\limits_{a\alpha}\prod\limits_{\mu}P_{\mathrm{out}}(y^{\mu}|\boldsymbol{F}^{\mu}\cdot\boldsymbol{x}^{a\alpha})^{\sigma^{a}_{\mu}}e^{\phi \sigma^{a}_{\mu}}. \end{align} \tag{ 13 }$$

In the present problem we thus introduced two replication levels. Each replica is hence characterized by a set of two indices: the first a index runs from 1 to s and specifies the disorder replica, the second α index, running from 1 to β is related to the replication in β. In total there are therefore s × β replicas. The teacher is set as replica 0. Implicitly henceforth aα when summed over will be running over [1, s]×[1, β]∪0. But

$$\begin{align} \mathbb{E}_{\boldsymbol{F}}\prod\limits_{a\alpha}\prod\limits_{\mu}P_{\mathrm{out}}(y^{\mu}|\boldsymbol{F}^{\mu}\cdot\boldsymbol{x}^{a\alpha})^{\sigma^{a}_{\mu}} = & \int \prod\limits_{\mu}\prod\limits_{a\alpha}dh^{\mu}_{a\alpha}(\mathrm{det}2\pi\boldsymbol{Q})^{-\frac{\alpha N}{2}}e^{-\frac{1}{2}\sum\limits_{\mu}\sum\limits{a\alpha,c\gamma}h^{\mu}_{a\alpha}(\boldsymbol{Q}^{-1})^{a\alpha~c\gamma}h^{\mu}_{c\gamma}}\nonumber\\ & \quad \times\int \prod\limits_{a\alpha\ne c\gamma}dq_{a\alpha,c\gamma}\int \prod\limits_{a\alpha\ne c\gamma}d\hat{q}_{a\alpha,c\gamma}e^{\sum\limits_{a\alpha\ne c\gamma}\hat{q}_{a\alpha,c\gamma}(\boldsymbol{x}^{a\alpha}\cdot\boldsymbol{x}^{c\gamma}-Nq_{a\alpha,c\gamma}}\nonumber\\ & \quad \times \prod\limits_{a\alpha}\prod\limits_{\mu}P_{\mathrm{out}}(y^{\mu}|h^{\mu}_{a\alpha})^{\sigma^{a}_{\mu}}, \end{align} \tag{ 14 }$$

where we defined $h^{\mu}_{a\alpha}\equiv\boldsymbol{F}^{\mu}\cdot\boldsymbol{x}^{a\alpha}$ Gaussian because of the central limit theorem and enforced the definition of its covariance matrix $\boldsymbol{Q}$ with integral representations of Dirac deltas. The conjugate matrix is $\hat{\boldsymbol{Q}}$. Matrix elements are noted with small q. Then,

$$\begin{align} \mathbb{E}_{\boldsymbol{F},\boldsymbol{x}^{0}}\Xi^{s} & = \int d\hat{\boldsymbol{Q}}d\boldsymbol{Q}e^{-N\mathrm{Tr}\hat{\boldsymbol{Q}}\boldsymbol{Q}}\left(\int\prod\limits_{a\alpha}d\boldsymbol{x}^{a\alpha}P_{X}(\boldsymbol{x}^{a\alpha})e^{\sum\limits_{a\alpha\ne c\gamma}x^{a\alpha}\hat{q}_{a\alpha,c\gamma}x^{c\gamma}}\right)^{N}\nonumber\\ & \quad \times \left(\sum\limits_{S^{a}}e^{\phi(\sum\limits_{a}S^{a}~-1)}\int dy \prod\limits_{a\alpha}dh_{a\alpha}(\mathrm{det}2\pi\boldsymbol{Q})^{-\frac{1}{2}}e^{-\frac{1}{2}\sum\limits_{a\alpha,c\gamma}h_{a\alpha}(\boldsymbol{Q}^{-1})^{a\alpha~c\gamma}h_{c\gamma}}\prod\limits_{a\alpha}P_{\mathrm{out}}(y^{\mu}|h_{a\alpha})^{S^{a}}\right)^{\alpha N}, \end{align} \tag{ 15 }$$

where we factorized both in i indices (first parenthesis) and in µ indices (second parenthesis). The free entropy defined in (11) then reads:

$$\begin{equation} \Phi(\beta,\phi) = \underset{s\rightarrow 0}{\mathrm{lim}}\frac{1}{s}\underset{\hat{\boldsymbol{Q}},\boldsymbol{Q}}{\mathrm{extr}}\left\{-\mathrm{Tr}\hat{\boldsymbol{Q}}\boldsymbol{Q}+\mathrm{ln}I_{X}(\boldsymbol{\hat{Q}})+\alpha\mathrm{ln}I_{Y}(\boldsymbol{Q})\right\}, \end{equation} \tag{ 16 }$$

with,

$$\begin{align} &I_{X} = \int\prod\limits_{a\alpha}d\boldsymbol{x}^{a\alpha}P_{X}(\boldsymbol{x}^{a\alpha})e^{\sum\limits_{a\alpha\ne c\gamma}x^{a\alpha}\hat{q}_{a\alpha,c\gamma}x^{c\gamma}}, \end{align} \tag{ 17 }$$

$$\begin{align} &I_{Y} = \sum\limits_{S^{a}}e^{\phi(\sum\limits_{a}S^{a}~-1)}\int dy \prod\limits_{a\alpha}dh_{a\alpha}(\mathrm{det}2\pi\boldsymbol{Q})^{-\frac{1}{2}}e^{-\frac{1}{2}\sum\limits_{a\alpha,c\gamma}h_{a\alpha}(\boldsymbol{Q}^{-1})^{a\alpha~c\gamma}h_{c\gamma}}\prod\limits_{a\alpha}P_{\mathrm{out}}(y^{\mu}|h_{a\alpha})^{S^{a}}. \end{align} \tag{ 18 }$$

3.1.3. Replica symmetric (RS) ansatz

The extremization in equation (16) is hard to carry out. As is now standard in the disordered systems literature we can reduce the number of parameters to be extremized over by enforcing the so-called RS ansatz [26] on both replication levels:

$$\begin{align} &q^{0,0} = r^{0},~~\hat{q}^{0,0} = \hat{r}^{0} \end{align} \tag{ 19 }$$

$$\begin{align} &q^{a\alpha,0} = m,~~\hat{q}^{a\alpha,0} = \hat{m} \end{align} \tag{ 20 }$$

$$\begin{align} &q^{a\alpha,a\alpha} = r,~~\hat{q}^{a\alpha,a\alpha} = -\frac{1}{2}\hat{r} \end{align} \tag{ 21 }$$

$$\begin{align} &q^{a\alpha,a\gamma} = Q,~~\hat{q}^{a\alpha,a\gamma} = \hat{Q} \end{align} \tag{ 22 }$$

$$\begin{align} &q^{a\alpha,c\gamma} = q,~~\hat{q}^{a\alpha,c\gamma} = \hat{q}, \end{align} \tag{ 23 }$$

where $q\lt Q$. Physically, the ansatz (19)–(23) means that two replicas seeing the same realization of disorder (i.e. possessing the same first index) have an overlap Q greater than the overlap between students seeing different realizations (an thus possessing different a-index). The $-\frac{1}{2}$ in the definition of $\hat{r}$ (21) is just introduced for latter convenience.

Note finally that while the ansatz (19)–(23) is replica-symmetric for both replications, it gives a set of equations that are formally those of a 1RSB problem [25]. This is also a reason why taking 1RSB ansatz [26] in the present large deviation calculation would be rather involved as it would lead to equations in the usual 2RSB form that are numerically involved to be solved.

We plug the RS ansatz (19)–(23) into the three contributions that make up equation (16). The trace term is:

$$\begin{equation} -\mathrm{Tr}\hat{\boldsymbol{Q}}\boldsymbol{Q} = -\hat{r}^{0}r^{0}-\beta s m\hat{m}+\frac{1}{2}\beta s r\hat{r}-s\frac{\beta(\beta-1)}{2}Q\hat{Q}-\beta^{2}\frac{s(s-1)}{2}q\hat{q}. \end{equation} \tag{ 24 }$$

We can decompose the exponent in (17) according to the ansatz (19)–(23):

$$\begin{align} \sum\limits_{a\alpha\ne c\gamma}x^{a\alpha}\hat{q}_{a\alpha,c\gamma}x^{c\gamma} & = \hat{r}^{0}(x^{0})^{2}+\hat{m}x^{0}\sum\limits_{a\alpha \ne 0}x^{a\alpha}-\frac{\hat{r}+\hat{Q}}{2}\sum\limits_{a\alpha\ne0}(x^{a\alpha})^{2}+\frac{\hat{Q}-\hat{q}}{2}\sum\limits_{a\ne 0}\sum\limits_{\alpha,\gamma}x^{a\alpha}x^{a\gamma}\nonumber\\ &\quad +\frac{\hat{q}}{2}\sum\limits_{a\alpha\ne0,c\gamma\ne 0}x^{a\alpha}x^{c\gamma}. \end{align} \tag{ 25 }$$

In the last but one term index 0 does not intervene. Introducing Hubbard–Stratonovitch fields {λ_a } for the last but one term and Hubbard–Stratonovith field ξ for the last, I_X reads:

$$\begin{equation} I_{X} = \int D\xi \int dx^{0}\overline{P_{X}}(x^{0})e^{\hat{r}^{0}(x^{0})^{2}}\left[\int D\lambda\left(\int dx P_{X}(x)e^{\hat{m}x^{0}x-\frac{\hat{r}+\hat{Q}}{2}x^{2}+\sqrt{\hat{Q}-\hat{q}}\lambda x+\sqrt{\hat{q}}\xi x}\right)^{\beta}\right]^{s}.\end{equation} \tag{ 26 }$$

To carry out the computation for I_Y (equation (18)) we need to explicitly compute the inverse of the Parisi matrix $\boldsymbol{Q}$ involved in equation (18). This is done in the following subsection.

3.1.4. Inverse of the overlap matrix

Name $\boldsymbol{\tilde{Q}}\equiv\boldsymbol{Q}^{-1}$ the inverse of the overlap matrix $\boldsymbol{Q}$. Since $\tilde{\boldsymbol{Q}}$ is clearly of the same form as $\boldsymbol{Q}$, we can parametrize its coefficient in an identical fashion as those of $\boldsymbol{Q}$ $\tilde{r}^{0},\tilde{m},\tilde{r},\tilde{q},\tilde{Q}$. $\boldsymbol{\tilde{Q}}\boldsymbol{Q} = \unicode{x1D7D9}_{\beta s+1}$ means:

$$\begin{align} &r^{0}\tilde{r}^{0}+\beta s m\tilde{m} = 1 \end{align} \tag{ 27 }$$

$$\begin{align} &r^{0}\tilde{m}+m\tilde{r}+(\beta-1)\tilde{Q}m+\beta(s-1)\tilde{q}m = 0, \end{align} \tag{ 28 }$$

$$\begin{align} &\tilde{r}^{0}m+\tilde{m}(r+(\beta-1)Q+\beta(s-1)q) = 0 \end{align} \tag{ 29 }$$

$$\begin{align} &m\tilde{m}+r\tilde{r}+(\beta-1)Q\tilde{Q}+\beta(s-1)q\tilde{q} = 1, \end{align} \tag{ 30 }$$

$$\begin{align} &m\tilde{m}+r\tilde{Q}+\tilde{r}Q+(\beta-2)Q\tilde{Q}+\beta(s-1)q\tilde{q} = 0 \end{align} \tag{ 31 }$$

$$\begin{align} &m\tilde{m}+r\tilde{q}+(\beta-1)Q\tilde{q}+q\tilde{r}+(\beta-1)q\tilde{Q}+\beta(s-2)q\tilde{q} = 0, \end{align} \tag{ 32 }$$

yielding:

$$\begin{align} \tilde{r}^{0} &= \frac{r+(\beta-1)Q+\beta(s-1)q}{r^{0}(r+(\beta-1)Q+\beta(s-1)q)-\beta s m^{2}}, \end{align} \tag{ 33 }$$

$$\begin{align} \tilde{m} & = \frac{-m}{r^{0}(r+(\beta-1)Q+\beta(s-1)q)-\beta s m^{2}} \end{align} \tag{ 34 }$$

$$\begin{align} \tilde{r}& = \frac{\beta m^{2}(q-2Q+r)+\beta((1-s)q^{2}+Q(3Q-2r)+(s-2)q(2Q-r))r^{0}}{(Q-r)(r+(\beta-1)Q-\beta q)(r^{0}(r+(\beta-1)Q+\beta(s-1)q)-\beta s m^{2})}\nonumber\\ &\quad +\frac{\beta^{2}(q-Q)(-m^{2}s+((-1+s)q+Q)r^{0})+(Q-r)(m^{2}+(-2Q+r)r^{0})}{(Q-r)(r+(\beta-1)Q-\beta q)(r^{0}(r+(\beta-1)Q+\beta(s-1)q)-\beta s m^{2})}, \end{align} \tag{ 35 }$$

$$\begin{align} \tilde{Q} & = \frac{(Q-r)(m^{2}-Qr^{0})+\beta(q-Q)(m^{2}s-((s-1)q+Q)r^{0}}{(Q-r)(r+(\beta-1)Q-\beta q)(r^{0}(r+(\beta-1)Q+\beta(s-1)q)-\beta s m^{2})}, \end{align} \tag{ 36 }$$

$$\begin{align} \tilde{q} & = \frac{-m^{2}+qr^{0}}{(r+(\beta-1)Q-\beta q)(r^{0}(r+(\beta-1)Q+\beta(s-1)q)-\beta s m^{2})}. \end{align} \tag{ 37 }$$

To compute the determinant of $\boldsymbol{Q}$, the simplest way is to guess the eigenvectors. For (x, 1, 1, ..., 1)^T we get two eigenvalues $\lambda_{\pm}$ whose product is:

$$\begin{equation} \lambda_{+}\lambda_{-} = r^{0}(r+(\beta-1)Q+(s-1)\beta q)-\beta s m^{2}. \end{equation} \tag{ 38 }$$

Then come s(β − 1) eigenvectors $\boldsymbol{e_{i}}-\boldsymbol{e_{i+1}}, ~i\not\equiv 0[\beta]$ (we are indexing starting from 0), with associated eigenvalues (r − Q). Then for $0\leqslant s\leqslant s-2$,

$$\begin{equation} \sum\limits_{k = s\beta+1}^{(s+1)\beta}\boldsymbol{e_{k}}-\sum\limits_{k = (s+1)\beta+1}^{(s+2)\beta}\boldsymbol{e_{k}}, \end{equation} \tag{ 39 }$$

is an eigenvector with eigenvalue $r+(\beta-1)Q-\beta q$. Then

$$\begin{align} \mathrm{ln~det}\boldsymbol{Q} & = \mathrm{ln}(r^{0}(r+(\beta-1)Q+(s-1)\beta q)-\beta m^{2})+(s-1)\mathrm{ln}(r+(\beta-1)Q-\beta q)\nonumber\\ &\quad +(\beta-1)s\mathrm{ln}(r-Q). \end{align} \tag{ 40 }$$

The same equality holds with tilde quantities in the right-hand side provided the signs are inverted, since $\mathrm{ln~det}\boldsymbol{Q} = -\mathrm{ln~det}\tilde{\boldsymbol{Q}}$. Identifying term by term results straightforwardly in a set of relations between tilde and non-tilde quantities (henceforth referred as determinant relations),

$$\begin{align} &r^{0}(r+(\beta-1)Q+(s-1)\beta q)-\beta m^{2} = [\tilde{r}^{0}(\tilde{r}+(\beta-1)\tilde{Q}+(s-1)\beta \tilde{q})-\beta \tilde{m}^{2}]^{-1} \end{align} \tag{ 41 }$$

$$\begin{align} &r+(\beta-1)Q-\beta q = [\tilde{r}+(\beta-1)\tilde{Q}-\beta \tilde{q}]^{-1} \end{align} \tag{ 42 }$$

$$\begin{align} &r-Q = [\tilde{r}-\tilde{Q}]^{-1}. \end{align} \tag{ 43 }$$

3.1.5. Evaluating the replica symmetric free entropy for GLM

Now decomposing the exponent in I_Y (18)

$$\begin{align} -\frac{1}{2}\sum\limits_{a\alpha,c\gamma}h_{a\alpha}\tilde{q}_{a\alpha,c\gamma}h_{c\gamma} = &-\frac{1}{2}\tilde{r}^{0}(h^{0})^{2}-\tilde{m}h^{0}\sum\limits_{a\alpha\ne0}h_{a\alpha}-\frac{\tilde{r}-\tilde{Q}}{2}\sum\limits_{a\alpha\ne 0}(h_{a\alpha})^{2}\nonumber\\ &+(\tilde{q}-\tilde{Q})\sum\limits_{a}\sum\limits_{\alpha,\gamma}h_{a\alpha}h_{a\gamma}-\tilde{q}\sum\limits_{a\alpha\ne0,c\gamma\ne0}h_{a\alpha}h_{c\gamma}. \end{align} \tag{ 44 }$$

Introducing Hubbard–Stratonovich fields {ζ_a } and η for the last two sums of (44) and factorizing in the index a

$$\begin{align} I_{Y} = &\frac{1}{\sqrt{\mathrm{det}2\pi\boldsymbol{Q}}}\int dy\int D\eta \int dh^{0}\overline{P_{\mathrm{out}}}(y|h^{0})e^{-\frac{1}{2}\tilde{r}^{0}(h^{0})^{2}}\nonumber\\ & \times\left[\sum\limits_{S = 0,1}\int D\zeta e^{\phi S}\left(\int dh P_{\mathrm{out}}(y|h)^{S}e^{-\tilde{m}h^{0}h-\frac{\tilde{r}-\tilde{Q}}{2}h^{2}+\sqrt{\tilde{q}-\tilde{q}}\zeta h+\sqrt{-\tilde{q}}\eta h}\right)^{\beta}\right]^{s}. \end{align} \tag{ 45 }$$

Now that all terms are computed the next step is then to divide by s and take the $s\rightarrow 0$ limit as prescribed by the replica trick (12). First, we need to enforce that all non-vanishing order 0 contribution cancel out, since the free entropy should not be diverging. Then, one needs to actually compute first order terms that will contribute in Φ (16).

At order 0, I_Y  = 1 since

$$\begin{align} \underset{s\rightarrow 0}{\mathrm{lim}}\mathrm{ln~det}(2\pi\boldsymbol{Q})& = \frac{1}{2}\mathrm{ln}(2\pi \tilde{r}^{0})\nonumber\\ & = -\mathrm{ln} \int dy\int D\eta \int dh^{0}\overline{P_{\mathrm{out}}}(y|h^{0})e^{-\frac{1}{2}\tilde{r}^{0}(h^{0})^{2}}. \end{align} \tag{ 46 }$$

The cancellation of order 0 terms imposes:

$$\begin{equation} 0 = \hat{r}^{0}r^{0}+\mathrm{ln}\int dx^{0}\overline{P_{X}}(x^{0})e^{\hat{r}^{0}(x^{0})^{2}}, \end{equation} \tag{ 47 }$$

where the first term comes from the trace term (24). It follows that $\hat{r}^{0} = 0$. Moreover, because of the saddle point equality:

$$\begin{equation} q_{a\alpha,c\gamma} = \frac{\int \prod\limits_{d\delta}(dx^{d\delta}P_{X}(x^{d\delta}))x^{a\alpha}x^{c\gamma}e^{\sum\limits_{d\delta\ne e \epsilon}x^{d\delta}\hat{q}_{d\delta,e\epsilon}x^{e\epsilon}}}{\int \prod\limits_{d\delta}(dx^{d\delta}P_{X}(x^{d\delta}))e^{\sum\limits_{d\delta\ne e \epsilon}x^{d\delta}\hat{q}_{d\delta,e\epsilon}x^{e\epsilon}}}, \end{equation} \tag{ 48 }$$

derived straightforwardly from (16) we also have

$$\begin{equation} r^{0} = \int dx^{0}\overline{P_{X}}(x^{0})(x^{0})^{2}. \end{equation} \tag{ 49 }$$

The order 1 contribution of I_X can be rewritten by carrying out a change of variables $\xi\rightarrow \xi+\hat{q}^{-\frac{1}{2}}\hat{m}x^{0}$ in equation (26), I_X assumes the compact form:

$$\begin{equation} \underset{s\rightarrow 0}{\mathrm{lim}}\frac{1}{s}I_{X} = \int D\xi I_{X}^{0}(\xi)\mathrm{ln}I_{X}^{1}(\xi), \end{equation} \tag{ 50 }$$

where

$$\begin{align} &I_{X}^{0}(\xi) = \int dx^{0}\overline{P_{X}}(x^{0})e^{-\frac{\hat{m}^{2}}{2\hat{q}}(x^{0})^{2}+\frac{\hat{m}}{\sqrt{\hat{q}}}\xi x^{0}} \end{align} \tag{ 51 }$$

$$\begin{align} &I_{X}^{1}(\xi) = \int D\lambda \left[\int dx P_{X}(x)e^{-\frac{\hat{r}+\hat{Q}}{2}x^{2}+(\sqrt{\hat{Q}-\hat{q}}\lambda+\sqrt{\hat{q}}\xi)x}\right]^{\beta}. \end{align} \tag{ 52 }$$

Assessing the order 1contribution from I_Y requires more work. Changing $\eta\rightarrow \eta-\frac{\tilde{m}}{\sqrt{-\tilde{q}}}h^{0}$ in (45) yields:

$$\begin{equation} I_{Y} = \frac{1}{\sqrt{\mathrm{det}2\pi\boldsymbol{Q}}}\int dy\int D\eta g^{0}(y,\eta)(g^{1}(y,\eta))^{s}, \end{equation} \tag{ 53 }$$

with

$$\begin{align} &g^{0}(y,\eta) = \int dh^{0}\overline{P_{\mathrm{out}}}(y|h^{0})e^{-\frac{1}{2}(\tilde{r}^{0}-\frac{\tilde{m}^{2}}{\tilde{q}})(h^{0})^{2}-\frac{\tilde{m}}{\sqrt{-\tilde{q}}}h^{0}\eta} \end{align} \tag{ 54 }$$

$$\begin{align} &g^{1}(y,\eta) = \sum\limits_{S = 0,1}\int D\zeta e^{\phi S}\left(\int dh P_{\mathrm{out}}(y|h)^{S}e^{-\frac{\tilde{r}-\tilde{Q}}{2}h^{2}+\sqrt{\tilde{q}-\tilde{Q}}\zeta h+\sqrt{-\tilde{q}}\eta h}\right)^{\beta}. \end{align} \tag{ 55 }$$

Expanding $\mathrm{ln}I_{Y}$ to $\mathcal{O}(s)$ (subscripts in parentheses standing for order in s) gives

$$\begin{align} \frac{1}{s}\mathrm{ln}I_{Y} = -\frac{1}{2}\mathrm{ln}(\mathrm{det}2\pi\boldsymbol{Q})_{(1)}+\frac{1}{\sqrt{2\pi r^{0}}}\int D\eta\int dy\left[g^{0}_{(1)}(y,\eta)+g^{0}_{(0)}(y,\eta)\mathrm{ln}g^{1}_{(0)}(y,\eta)\right]. \end{align} \tag{ 56 }$$

We used the fact that at order 0 terms canceled, and the identity $\int D\eta \int dy g^{0}(y,\eta) = \sqrt{2\pi r^{0}}+\mathcal{O}(s)$. But

$$\begin{align} \frac{1}{\sqrt{2\pi r^{0}}}\int D\eta\int dy g^{0}_{(1)}(y,\eta)& = \underset{s\rightarrow 0}{\mathrm{lim}}\frac{1}{\sqrt{2\pi r^{0}}}\partial_{s}\left(\int D\eta\int dy g^{0}(y,\eta)\right)\nonumber\\ & = \underset{s\rightarrow 0}{\mathrm{lim}}\frac{1}{\sqrt{2\pi r^{0}}}\partial_{s}\sqrt{\frac{2\pi}{\tilde{r}^{0}}}\nonumber\\ & = \frac{-\beta m^{2}}{2r^{0}(r+(\beta-1)Q-\beta q)}, \end{align} \tag{ 57 }$$

thus

$$\begin{align} \frac{1}{s}\mathrm{ln}I_{Y} = &-\frac{1}{2}\frac{\beta q}{r+(\beta-1)Q-\beta q}-\frac{1}{2}\mathrm{ln}(r+(\beta-1)Q-\beta q)-\frac{1}{2}(\beta-1)\mathrm{ln}(r-Q)\nonumber\\ &+\frac{1}{\sqrt{2\pi r^{0}}}\int dy\int D\eta g^{0}_{(0)}(y,\eta)\mathrm{ln}g^{1}_{(0)}(y,\eta). \end{align} \tag{ 58 }$$

It is actually possible to proceed to Gaussian changes of variables in the last term so as to exactly cancel the first three contributions in (58). To do this:

$$\begin{align} &h\rightarrow \sqrt{r-Q}h+(r-Q)(\sqrt{\tilde{q}-\tilde{Q}}\zeta+\sqrt{-\tilde{q}}\eta) \end{align} \tag{ 59 }$$

$$\begin{align} &\zeta \rightarrow \frac{1}{\sqrt{1-\beta(r-Q)(\tilde{q}-\tilde{Q}}}\zeta -\frac{\beta\sqrt{-\tilde{q}(\tilde{q}-\tilde{Q})}(r-Q)}{1-\beta(r-Q)(\tilde{q}-\tilde{Q}}\eta, \end{align} \tag{ 60 }$$

(we used the determinant relations (41)–(43)) allowing to rewrite the last term in (58) as:

$$\begin{align} & \frac{1}{\sqrt{2\pi r^{0}}}\int dy\int D\eta g^{0}_{(0)}(y,\eta)\mathrm{ln}g^{1}_{(0)}(y,\eta) \nonumber \\ & \quad = \frac{\beta}{2}\mathrm{ln}(r-Q)-\frac{1}{2}\mathrm{ln}(1-\beta(r-Q)(\tilde{q}-\tilde{Q}))\nonumber \\ & \qquad +\int \frac{dy}{\sqrt{2\pi r^{0}}}\int D\eta g^{0}_{(0)}(y,\eta)\mathrm{ln}\left(1+e^{\phi }\int D\zeta \left(\int dh P_{\mathrm{out}}(y|*)\right)^{\beta}\right)\nonumber\\ & \qquad +\frac{1}{2}\left(-\beta(r-Q)\tilde{q}+\frac{\beta^{2}(-\tilde{q}(\tilde{q}-\tilde{Q})(r-Q)^{2}}{1-\beta(r-Q)(\tilde{q}-\tilde{Q}}\right)\int D\eta \eta^{2}\int \frac{dy}{\sqrt{2\pi r^{0}}}g^{0}_{(0)}(y,\eta), \end{align} \tag{ 61 }$$

with

$$\begin{equation} * = \sqrt{r-Q}h+(r-Q)\left(\sqrt{\frac{\tilde{q}-\tilde{Q}}{1-\beta(r-Q)(\tilde{q}-\tilde{Q})}}\zeta +\frac{\sqrt{-\tilde{q}}}{1-\beta(r-Q)(\tilde{q}-\tilde{Q})}\eta\right). \end{equation} \tag{ 62 }$$

It is straightforward to see:

$$\begin{equation} \int D\eta \eta^{2}\int \frac{dy}{\sqrt{2\pi r^{0}}}g^{0}_{(0)}(y,\eta) = \frac{\tilde{r}^{0}\tilde{q}-\tilde{m}^{2}}{\tilde{r}^{0}\tilde{q}}, \end{equation} \tag{ 63 }$$

so the last term in (61) is:

$$\begin{align} & \frac{1}{2}\left(-\beta(r-Q)\tilde{q}+\frac{\beta^{2}(-\tilde{q}(\tilde{q}-\tilde{Q})(r-Q)^{2}}{1-\beta(r-Q)(\tilde{q}-\tilde{Q}}\right)\int D\eta \eta^{2}\int \frac{dy}{\sqrt{2\pi r^{0}}}g^{0}_{(0)}(y,\eta)\nonumber\\ & \quad = -\frac{\beta}{2}\frac{\tilde{r}^{0}\tilde{q}-\tilde{m}^{2}}{\tilde{r}^{0}(\tilde{r}+(\beta-1)\tilde{Q}-\beta\tilde{q})}\nonumber\\ & \quad = \frac{1}{2}\frac{\beta q}{r+(\beta-1)Q-\beta q}. \end{align} \tag{ 64 }$$

Similarly the first terms in (61) are:

$$\begin{align} & \frac{\beta}{2}\mathrm{ln}(r-Q)-\frac{1}{2}\mathrm{ln}(1-\beta(r-Q) (\tilde{q}-\tilde{Q}))\nonumber\\ & \quad = -\frac{\beta-1}{2}\mathrm{ln}(\tilde{r}-\tilde{Q})-\frac{1}{2}\mathrm{ln}(\tilde{r}+(\beta-1)\tilde{Q}-\beta\tilde{q})\nonumber\\ & \quad = \frac{1}{2}\mathrm{ln}(r+(\beta-1)Q-\beta q)+\frac{1}{2}(\beta-1)\mathrm{ln}(r-Q). \end{align} \tag{ 65 }$$

We again used the determinant identity (41)–(43) in the last line. Then tilde quantities in the s = 0 limit can be accordingly be replaced by their expressions (33)–(37):

$$\begin{align} &\underset{s\rightarrow 0}{\mathrm{lim}}\tilde{r}^{0}-\frac{\tilde{m}^{2}}{\tilde{q}} = \frac{q}{qr^{0}-m^{2}} \end{align} \tag{ 66 }$$

$$\begin{align} &\underset{s\rightarrow 0}{\mathrm{lim}}\frac{\tilde{m}}{\sqrt{-\tilde{q}}} = \sqrt{\frac{m^{2}}{r^{0}(qr^{0}-m^{2})}} \end{align} \tag{ 67 }$$

$$\begin{align} &\underset{s\rightarrow 0}{\mathrm{lim}}\sqrt{\frac{\tilde{q}-\tilde{Q}}{1-\beta(r-Q)(\tilde{q}-\tilde{Q})}} = \frac{\sqrt{Q-q}}{r-Q} \end{align} \tag{ 68 }$$

$$\begin{align} &\underset{s\rightarrow 0}{\mathrm{lim}}\frac{\sqrt{-\tilde{q}}}{1-\beta(r-Q)(\tilde{q}-\tilde{Q})} = \sqrt{\frac{qr^{0}-m^{2}}{r^{0}(r-Q)^{2}}}. \end{align} \tag{ 69 }$$

Ultimately some changes of variables can be used to bring $g^{0}_{(0)}$ to a more compact form:

$$\begin{align} &h\rightarrow \sqrt{\frac{qr^{0}-m^{2}}{q}}h^{0}+\sqrt{\frac{m^{2}(qr^{0}-m^{2})}{qr^{0}}},~~\eta \rightarrow \sqrt{\frac{qr^{0}}{qr^{0}-m^{2}}}\eta. \end{align} \tag{ 70 }$$

Finally

$$\begin{equation} \underset{s\rightarrow 0}{\mathrm{lim}}\frac{1}{s}I_{Y} = \int D\eta \int dy I^{0}_{Y}(y,\eta) \mathrm{ln}I^{1}_{Y}(y,\eta), \end{equation} \tag{ 71 }$$

with

$$\begin{align} &I^{0}_{Y}(y,\eta) = \int Dh^{0}\overline{P_{\mathrm{out}}}\left(y|\sqrt{\frac{qr^{0}-m^{2}}{q}}h^{0}+\sqrt{\frac{m^{2}}{q}}\eta\right), \end{align} \tag{ 72 }$$

$$\begin{align} & \left. I^{1}_{Y}(y,\eta) = 1+e^{\phi}\int D\zeta \left(\int dh P_{\mathrm{out}}\left(y|\sqrt{r-Q}h+\sqrt{Q-q}\zeta +\sqrt{q}\eta\right)\right)\right)^{\beta}. \end{align} \tag{ 73 }$$

3.1.5.1. Replica symmetric free entropy for GLM

Putting everything together the replica free entropy (16) reads:

$$\begin{align} \Phi_\mathrm{RS} &= \underset{\hat{m},\hat{r},\hat{q},\hat{Q},r,q,Q}{\mathrm{extr}}\Big\{-\beta m\hat{m}+\frac{\beta}{2}r\hat{r}-\frac{\beta(\beta-1)}{2}Q\hat{Q}+\frac{\beta^{2}}{2}q\hat{q}+\int D\xi I_{X}^{0}(\xi)\mathrm{ln}I_{X}^{1}(\xi)\nonumber\\ & \qquad\qquad\quad\qquad +\alpha \int D\eta \int dy I^{0}_{Y}(y,\eta) \mathrm{ln}I^{1}_{Y}(y,\eta)\Big\} \end{align} \tag{ 74 }$$

$$\begin{align} r^{0} &= \int dx^{0}\overline{P_{X}}(x^{0})(x^{0})^{2} \end{align} \tag{ 75 }$$

$$\begin{align} I_{X}^{0}(\xi) &= \int dx^{0}\overline{P_{X}}(x^{0})e^{-\frac{\hat{m}^{2}}{2\hat{q}}(x^{0})^{2}+\frac{\hat{m}}{\sqrt{\hat{q}}}\xi x^{0}} \end{align} \tag{ 76 }$$

$$\begin{align} I_{X}^{1}(\xi)& = \int D\lambda \left[\int dx P_{X}(x)e^{-\frac{\hat{r}+\hat{Q}}{2}x^{2}+\left(\sqrt{\hat{Q}-\hat{q}}\lambda+\sqrt{\hat{q}}\xi\right)x}\right]^{\beta} \end{align} \tag{ 77 }$$

$$\begin{align} I^{0}_{Y}(y,\eta)& = \int Dh^{0}\overline{P_{\mathrm{out}}}\left(y|\sqrt{\frac{qr^{0}-m^{2}}{q}}h^{0}+\sqrt{\frac{m^{2}}{q}}\eta\right) \end{align} \tag{ 78 }$$

$$\begin{align} I^{1}_{Y}(y,\eta) & = \left. 1+e^{\phi}\int D\zeta \left(\int dh P_{\mathrm{out}}\left(y|\sqrt{r-Q}h+\sqrt{Q-q}\zeta +\sqrt{q}\eta\right)\right)\right)^{\beta}. \end{align} \tag{ 79 }$$

The Bayes-optimal teacher-student setting for the perceptron is defined by the following measures (see section 1.2):

$$\begin{align} &\overline{P_{X}}(x) = P_{X}(x) = \frac{1}{\sqrt{2\pi}}e^{\frac{-1}{2}x^{2}} \end{align} \tag{ 80 }$$

$$\begin{align} &\overline{P_{\mathrm{out}}}(y|h) = P_{\mathrm{out}}(y|h) = \delta(y-\mathrm{sgn}(h)). \end{align} \tag{ 81 }$$

3.2.1. Replica symmetric free entropy for the perceptron

We shall simply plug into the generic GLM (74) expressions the particular priors and posteriors for the perceptron (80) and (81). First,

$$\begin{align} I_{X}^{0}(\xi)& = \int \frac{1}{\sqrt{2\pi}}dx^{0}e^{-\frac{1}{2}(1+\frac{\hat{m}^{2}}{\hat{q}})(x^{0})^{2}+\frac{\hat{m}}{\sqrt{\hat{q}}}\xi x^{0}} \end{align} \tag{ 82 }$$

$$\begin{align} & = \frac{1}{\sqrt{2\pi(1+\frac{\hat{m}^{2}}{\hat{q}})}}e^{\frac{1}{2}\frac{\hat{m}^{2}}{\hat{q}+\hat{m}^{2}}\xi^{2}},\phantom{00} \end{align} \tag{ 83 }$$

while two straightforward Gaussian integrals yield:

$$\begin{align} I_{X}^{1}(\xi)& = \int D\lambda \left[\int \frac{1}{\sqrt{2\pi}} dx P_{X}(x)e^{-\frac{1}{2}(\hat{r}+\hat{Q})x^{2}+(\sqrt{\hat{Q}-\hat{q}}\lambda+\sqrt{\hat{q}}\xi)x}\right]^{\beta}\phantom{0000000} \end{align} \tag{ 84 }$$

$$\begin{align} & = \int d\lambda \frac{1}{(1+\hat{r}+\hat{Q})^{\frac{\beta}{2}}}e^{-\frac{1}{2}(1-\beta\frac{\hat{Q}-\hat{q}}{1+\hat{r}+\hat{Q}})\lambda^{2}+\beta\frac{\sqrt{\hat{q}(\hat{Q}-\hat{q})}}{1+\hat{r}+\hat{Q}}\xi\lambda+\frac{\beta}{2}\frac{\hat{q}}{1+\hat{r}+\hat{Q}}\xi^{2}} \end{align} \tag{ 85 }$$

$$\begin{align} & = (1+\hat{r}+\hat{Q})^{\frac{1-\beta}{2}}(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})^{-\frac{1}{2}}e^{\frac{\beta \hat{q}}{2(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})}\xi^{2}}. \end{align} \tag{ 86 }$$

Thus,

$$\begin{align} \int D\xi I_{X}^{0}(\xi)\mathrm{ln}I_{X}^{1}(\xi) = &-\frac{\beta-1}{2}\mathrm{ln}(1+\hat{r}+\hat{Q})-\frac{1}{2}\mathrm{ln}(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})\nonumber\\ &+\frac{\beta}{2}\frac{\hat{q}+\hat{m}^{2}}{1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q}}. \end{align} \tag{ 87 }$$

Now defining the special function: $H(x)\equiv\frac{1}{\sqrt{2\pi}}\int\limits_{x}^{\infty}Dt$

$$\begin{equation} I^{0}_{Y}(y,\eta) = \int Dh^{0}\delta\left(y-\mathrm{sgn}\left(\sqrt{\frac{q-m^{2}}{q}}h^{0}+\sqrt{\frac{m^{2}}{q}}\eta\right)\right) = H\left(-y\sqrt{\frac{m^{2}}{q-m^{2}}}\eta\right). \end{equation} \tag{ 88 }$$

In writing so we took into account the fact that that y = ±1, which implies also to replace the integral over y in the energetic part by a sum over {±1}. Furthermore,

$$\begin{equation} I^{1}_{Y}(y,\eta) = 1+e^{\phi}\int D\zeta H\left(-\frac{y}{\sqrt{r-Q}}\left(\sqrt{Q-q}\zeta+\sqrt{q}\eta\right)\right)^{\beta}, \end{equation} \tag{ 89 }$$

from which it follows that the energetic term in equation (74) reads:

$$\begin{align} \alpha \int D\eta \sum\limits_{y = \pm1}I^{0}_{Y}(y,\eta) \mathrm{ln}I^{1}_{Y}(y,\eta) = &2\alpha\int D\eta H\left(-\sqrt{\frac{m^{2}}{q-m^{2}}}\eta\right)\nonumber\\ & \times \mathrm{ln}\left[1+e^{\phi}\int D\zeta H\left(-\frac{1}{\sqrt{r-Q}}\left(\sqrt{Q-q}\zeta+\sqrt{q}\eta\right)\right)^{\beta}\right]. \end{align} \tag{ 90 }$$

The y = 1 and y =−1 being equal modulo a double change of variable $\zeta,\eta\rightarrow -\zeta,-\eta$ whence the factor 2. Thus for the perceptron:

$$\begin{align} &\Phi_\mathrm{RS} = \underset{\hat{m},\hat{r},\hat{q},\hat{Q},r,q,Q}{\mathrm{extr}}\Bigg\{\frac{\beta}{2}r\hat{r}-\beta m\hat{m}-\frac{\beta(\beta-1)}{2}Q\hat{Q}+\frac{\beta^{2}}{2}q\hat{q}-\frac{\beta-1}{2}\mathrm{ln}(1+\hat{r}+\hat{Q})\nonumber\\ & \qquad\qquad\qquad\qquad -\frac{1}{2}\mathrm{ln}(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})+\frac{\beta}{2}\frac{\hat{q}+\hat{m}^{2}}{1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q}}\nonumber\\ & \qquad\qquad\qquad\qquad +2\alpha\int D\eta H\left(-\sqrt{\frac{m^{2}}{q-m^{2}}}\eta\right)\mathrm{ln}\left[1+e^{\phi}\int D\zeta H\left(-\frac{1}{\sqrt{r-Q}}(\sqrt{Q-q}\zeta+\sqrt{q}\eta)\right)^{\beta}\right]\Bigg\}. \end{align} \tag{ 91 }$$

We supply numerical evidence for some assumptions made in this work, in particular the replica trick (12) and the use of the Gardner volume as a measure of informativeness (see section 2 in the main text).

First, we sample numerically at random subsets of cardinalities n ∈ {0.3, 0.6, 0.9, 2.7} out of a pool of cardinal given by α = 3, and plot the complexity extracted therefrom in figure 3. The volumes were evaluated using the approximate message passing (AMP) algorithm 2, and simulations were performed at N = 20, with 10⁷ draws, for a fixed teacher. Such a large number of samples and small system size is needed in order to access exponentially rare events. For larger sizes the probability of rare event is exponentially smaller and proportionally more samples would be needed, which is not computationally accessible. Because of the $\mathcal{O}(10^7\times N^2)$ complexity N has been kept small, while AMP is known to be valid only in the $N\uparrow\infty$ limit, hence inducing errors due to finite size. Nevertheless, the agreement with the theoretical curves for Σ(n, v) is quite good.

Zoom In Zoom Out Reset image size

Algorithm 1. Uncertainty sampling.
Select heuristic strategy from table 1
Define batch size k
Initialize $S\subset\mathcal{S} = \{\boldsymbol{F}_{\mu}\}_{1\leqslant\mu\leqslant\alpha N}$ ($|S|\gt0$)
while $|S|\lt nN$ do
Obtain required estimates given S
Obtain model predictions at data-points in Sc
Sort predictions according to sorting criterion
Add first k elements in the sorting permutation to S
end while

Algorithm 2. Single-instance AMP for the perceptron.
Initialize $\boldsymbol{\hat{x}}\leftarrow 0$
Initialize $\boldsymbol{\hat{\Delta}}\leftarrow 1$
Initialize $\boldsymbol{g}_{\mathrm{out}}\leftarrow 0$
while Convergence criterion not satisfied do
$\Gamma_{\mu}^{t}\leftarrow\sum\limits_{i}(F_{i}^{\mu})^{2}\hat{\Delta}_{i}^{t-1}$
$\omega^{t}_{\mu}\leftarrow\sum\limits_{i}F_{i}^{\mu}\hat{x}_{i}^{t-1} -\Gamma_{\mu}^{t}g_{\mathrm{out}}^{t-1}$
$g_{\mathrm{out},\mu}^{t}\leftarrow\frac{y^{\mu}}{\sqrt{2\pi \Gamma_{\mu}^{t}}}\frac{e^{-\frac{(\omega_{\mu}^{t})^{2}}{2\Gamma_{\mu}^{t}}}}{H\left(-\frac{y^{\mu}\omega_{\mu}^{t}}{\sqrt{\Gamma^{t}_{\mu}}}\right)}$
$(\Sigma_{i}^{t})^{-1}\leftarrow-\sum\limits_{\mu}(F_{i}^{\mu})^{2} ( -\frac{\omega^{t}}{V^{t}}g_{\mathrm{out}}^{t}-(g_{\mathrm{out}}^{t})^{2})$
$R_{i}^{t}\leftarrow\hat{x}_{i}^{t-1}+\Sigma_{i}^{t} \sum\limits_{\mu}F_{i}^{\mu} g_{\mathrm{out}}^{t}$
$\hat{x}_{i}^{t}\leftarrow\frac{R_{i}^{t}}{1+\Sigma^{t}_{i}}$
$\hat{\Delta}_{i}^{t}\leftarrow\frac{\Sigma_{i}^{t}}{1+\Sigma^{t}_{i}}$
end while

We finally present a numerical check of the theoretical prediction for the m(v) curves, see figure 1. At large instance size, N = 2000, it is not computationally feasible to obtain sufficient statistics for observing the large deviations of the volume through passive subset sampling, as it was done in the previous experiment. Thus, we resorted to the label-informed AL-AMP AL strategy (see section 5, algorithm 1 and table 1) for biasing the subset selection towards more/less informative subsets. In particular, we constructed each subset by mixing varying ratios of maximally informative samples (selected according to the informed AL-AMP procedure) and minimally informative samples (selected according to the same procedure but with the reversed sorting order). In figure 4, the pool size is α = 10 and the budget is fixed to n = 1.5. For each subset, the AMP algorithm 2 was run to get the estimator $\hat{\boldsymbol{x}}$ and the magnetization $m = \frac{x^0\cdot \hat{x}}{N}$ was deduced therefrom. This incidentally corroborates once more that using the Gardner volume instead of the magnetization to judge for the informativeness of a selection is coherent.

Zoom In Zoom Out Reset image size

Table 1. table summarizing the specifics of the uncertainty sampling strategies considered in this paper.

Uncertainty sampling strategies
Heuristic	Required estimates	Sorting criterion
Agnostic AL-AMP	$\hat{\boldsymbol{x}}_\textrm{AMP}, \hat{\boldsymbol{\Delta}}_\textrm{AMP}$	$\mathrm{argmin}_\mu \left|\mathrm{erf}\left(\frac{\boldsymbol{F}^{^{\prime} \mu}\hat{\boldsymbol{x}}_\textrm{AMP}}{\sqrt{2(\boldsymbol{F}^{^{\prime} \mu})^2\hat{\boldsymbol{\Delta}}_\textrm{AMP}}}\right)\right|$
Informed AL-AMP	$\hat{\boldsymbol{x}}_\textrm{AMP}, \hat{\boldsymbol{\Delta}}_\textrm{AMP}$	$\mathrm{argmax}_\mu \left|y^\mu - \mathrm{erf}\left(\frac{\boldsymbol{F}^{^{\prime} \mu}\hat{\boldsymbol{x}}_\textrm{AMP}}{\sqrt{2(\boldsymbol{F}^{^{\prime} \mu})^2\hat{\boldsymbol{\Delta}}_\textrm{AMP}}}\right)\right|$
Query by committee	$\{\boldsymbol{x}_\textrm{SGD}^k\}_{k = 1}^{K}$	$\mathrm{argmin}_\mu |\sum_{k = 1}^{K} \mathrm{sign}\left(\boldsymbol{F}^{^{\prime} \mu}\cdot \boldsymbol{x}_\textrm{SGD}^k\right)|$
Logistic regression	$\boldsymbol{x_\textrm{log}}$	$\mathrm{argmin}_\mu \left|\boldsymbol{F}^{^{\prime} \mu}\cdot \boldsymbol{x}_\textrm{log}\right|$
Perceptron learning	$\boldsymbol{x_\textrm{perc}}$	$\mathrm{argmin}_\mu \left|\boldsymbol{F}^{^{\prime} \mu}\cdot \boldsymbol{x}_\textrm{perc}\right|$

We remark at this point that the presented replica calculation was obtained in the so-called RA ansatz. In general, it is possible for the RS result not to be asymptotically exact, requiring replica symmetry breaking (RSB) in order to evaluate the correct free entropy Φ(β, φ) [26]. In this model, while RSB is surely not needed close to the maximum of the complexity curves as implied by results in [18], it conversely has to be taken into account away from typical volumes. The volumes for which the RS ansatz (19)–(23) ceases to be valid can be evaluated by considering an infinitesimal perturbation thereof of 1 step RSB form as is detailed in appendix B, yielding the stability condition:

$$\begin{align} &\left|\left(2\beta\frac{\hat{q}+\hat{m}^2}{(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})^3} -\frac{1}{(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})^2}\right)\right|\nonumber\\ &\times \left|-\frac{\partial}{\partial q}\left(\frac{2\alpha e^{2\phi}}{\beta^2 (r-Q)(Q-q)} \int D\rho H\left(-\sqrt{\frac{m^2}{q-m^2}}\rho\right) \left(\frac{\int D\zeta\,\zeta\, H^{\beta}(X)}{1+e^{\phi}\int D\zeta H^{\beta}(X)} \right)^2\right)\right|\lt 1. \end{align} \tag{ 92 }$$

Figure 5 shows in solid lines the regions in v space where the RS assumption holds. At the same time, the presence of RSB usually entails corrections that are very small in magnitude. We hence believe the error bounds here reported under the assumption of replica symmetry to achieve a good degree of accuracy.

Zoom In Zoom Out Reset image size

The setting investigated in this paper provides a unique chance to benchmark the algorithmic performance of any given pool-based AL algorithm against the optimal achievable performance, and to measure how closely are the large deviations results approached. This offers a controlled setup complementary to the usual one of simply comparing different algorithms on standard datasets, where no point of reference exists to evaluate the performances. The aim of the present section is hence to illustrate how the results on Gardner volumes reported above may serve to evaluate existing AL procedures. Before moving to such algorithmic performance we should make a distinction between two possible classes of AL scenarios:

• Label-agnostic settings, where the student has no prior knowledge on the ground truth labels. In other words, the AL selection must be based solely on the knowledge of the input patterns $\{\boldsymbol{x}\}$, and make no use of the true labels {y}. In this case, for binary labels there is a simple lower bound on the Gardner volume reachable with nN samples, $v\geqslant 2^{-n}$, which is obtained by the argument that every new sample can at best divide the current volume by a factor two, see [4]. This strategy is explored in the famous QBC AL strategy, and the classical work [4] argues that the volume halving can be actually achieved when an unlimited set of samples is available. Plotting this volume-halving bound in figure 2 we see that even though there exist subsets of the pool that would lead to smaller Gardner volumes, they cannot be found in a label-agnostic way.
• Label-informed settings, where external knowledge on the true labels is available and can be used for extracting more information during the selection process. In many real-world applications the structure in the input data could be exploited (e.g. through clustering, transfer learning, etc) for making unsupervised guesses of the labels and for bootstrapping an AL strategy. A concrete example where external insight is available is drug discovery [7], where additional information can be inferred from the presence of chemical functional groups (or absence thereof) on the molecules in the data pool. In the present work, we study whether it is possible, with full access to the labels, to devise an efficient method for finding a subset of samples that achieves close to the minimal Gardner volume bound (note that this is still an algorithmically non-trivial problem).

In this section we will investigate both the label-agnostic and label-informed strategies. We will benchmark several well known AL algorithms on the model studied in the present paper as well as design and test a new message passing based AL algorithm. Before doing that let us describe the general strategy.

Many of the commonly used AL criteria rely on some form of label-uncertainty measure. Uncertainty sampling [1, 29] is an AL scheme based on the idea of iteratively selecting and labelling data-points where the prediction of the available trained model is the least confident. In general, the computational complexity associated to this type of scheme is of order $\mathcal{O}(N^{3})$, requiring an extensive number of runs of a training algorithm (which can scale as $\mathcal{O}(N^2)$ at best). Since even training a single model per pattern addition can become expensive in the large N setting, in all our numerical tests we opted for adding to the labeled set batches of k = 20 samples instead of a single sample per iteration. We remark that, despite the k-fold speed-up, the observed performance deterioration is negligible. The structure of this type of algorithm is sketched in algorithm 1.

In general, estimating the Gardner volume on a given training set or the label-uncertainty of a new sample is a computationally hard problem. However, in perceptrons (or more general GLMs) with i.i.d. Gaussian input data $\boldsymbol{F}$, at large system size N one can rely on the estimate provided by a well known algorithm for approximate inference, AMP (historically also referred to as the Thouless–Anderson–Palmer (TAP) equations, see [30]). AMP is a standard iterative procedure used for Bayesian inference on a factor graph associated to a probability measure. We refer the interested reader to the ample literature dedicated to message-passing algorithms ([12, 23, 31] for example) for more detailed discussion thereon. The AMP algorithm [13, 32, 33], yields (at convergence) an estimator of the posterior means, $\hat{\boldsymbol{x}}$, and variances, $\hat{\boldsymbol{\Delta}}$, thus accounting for uncertainty in the inference process including the label of a new sample. The Gardner volume v (corresponding to the so-called Bethe free entropy) can then be expressed as a simple function of the AMP fixed-point messages (see [34] for an example). We provide a pseudo-code of AMP in the case of the perceptron in algorithm 2. An important remark is that when the training set is not sampled randomly from the pool, as in the AL context, correlations can arise and AMP is no longer rigorously guaranteed to converge nor to provide a consistent estimate of the Gardner volume. In the present work, we can only argue that its employment seems to be justified a posteriori by observing the agreement between theoretical predictions and numerical experiments for instance for the generalization error.

We use the AMP algorithm to introduce a new uncertainty sampling procedure relying on the information contained in the AMP messages, denoted as AL-AMP in the following. At each iteration, the single-instance AMP equations are run on the current training subset to yield posterior mean estimate $\hat{\boldsymbol{x}}$ and variance $\boldsymbol{\hat{\Delta}}$. These quantities can then be used to evaluate, for all the unlabeled samples, the output magnetization (i.e. the Bayesian prediction) defined as

$$\begin{equation} m_{\mathrm{out}}^\mu = \mathrm{erf}(\omega^\mu/\sqrt{2\Gamma^\mu})~~~~~~~\forall\mu|\sigma_\mu = 0, \end{equation} \tag{ 93 }$$

where we introduced the output overlaps $\boldsymbol{\omega} = \boldsymbol{F}^{^{\prime}}\cdot\hat{\boldsymbol{x}}$ and variances $\boldsymbol{\Gamma} = (\boldsymbol{F}^{^{\prime}}\odot \boldsymbol{F}^{^{\prime}})\cdot\hat{\boldsymbol{\Delta}}$, where $\odot$ is the component-wise product. The output magnetizations correspond to the weighted output average over all the estimators contained in the current version space, and their magnitude represents the overall confidence in the classification of the still unlabeled samples. This means that AMP provides an extremely efficient way of obtaining the information on uncertainty. The specifics of the algorithm can be found in table 1.

We also explore numerically the label-informed AL regime introduced in the previous section. We consider its limiting case by introducing the informed AL-AMP strategy, which can fully access the true labels $\boldsymbol{Y}$ in order to query the samples $\boldsymbol{F}^{\mu}$ whose output magnetisation $m_{\mathrm{out}}^\mu$ (93) is maximally distant from the correct classification $y^{\mu}$. This selection process can iteratively reduce the Gardner volume of factors larger than 2. Again, the relevant specifics of informed AL-AMP algorithm are detailed in table 1.

One of the widely used uncertainty sampling procedure is the so-called QBC strategy [4, 16]. In QBC, at each time step, a committee of K students is to be sampled from the version space (e.g. via the Gibbs algorithm). The committee is then employed to choose the labels to be queried, by identifying the samples where maximum disagreement in the committee members outputs is observed. The QBC algorithm was introduced as a proxy for doing bisection, i.e. cutting version space into two equal-volume halves. As already mentioned, this constitutes the optimal information gain in an label-agnostic setting [35]. Note that, however, the QBC procedure can achieve volume-halving only in the infinite-size committee limit, $K\uparrow\infty$, with uniform version space sampling and with availability of infinitely many samples. Obviously, running a large number K of ergodic Gibbs sampling procedures quickly becomes computationally unfeasible. Moreover in the pool-based AL the pool of samples is limited. In order to allow comparison with other strategies at finite sizes, we approximated the uniform sampling with a set of greedy optimization procedures (e.g. stochastic gradient descent) from random initialization conditions, checking numerically that this yields a committee of students reasonably spread out in version space. It is possible to ensure a greater coverage of the version space by performing a short Monte-Carlo random walk for each committee member. The effect has been found to be small for computationally reasonable lengths of walk.

We also implemented an alternative uncertainty sampling strategy, relying on a single training procedure (e.g. training with the perceptron algorithm or logistic regression) per iteration: in this case, the uncertainty information is extracted from the magnitude of the pre-activations measured at the unlabeled samples after each training cycle. This strategy implements the intuitive geometric idea of looking for the samples that are most orthogonal to the available updated estimator, which are more likely to halve the version space independently of the value of the true label.

All the tested procedures, with the exception of QBC, display a complexity of approximately $\mathcal{O}(N^3)$, possibly to be corrected by a factor accounting for the number of iterations required at each training step. The adapted QBC algorithm similarly has complexity $\mathcal{O}(KN^3)$, as it involves a committee of K models. Ideally, the original QBC algorithm [4] would require at each step to sample uniformly the current version space, thus implying an exponentially costly Monte Carlo step.

In figure 6, we compare the minimum Gardner volume obtained from the large deviation calculation with the algorithmic performance obtained on synthetic data at finite size, N = 2 × 10³, by the AL-AMP algorithms detailed in algorithm 1 and table 1. The data-pool size is fixed to α = 3. The large deviation analysis yields values for the minimum and maximum achievable Gardner volumes at any budget n. We compare the algorithmic results also with the prediction for the typical case and with the volume-halving curve 2⁻ⁿ . Since in the considered pool-based setting the volume-halving performance cannot be achieved for volumes smaller than the Gardner volume corresponding to the entire pool v(α), the relevant volume-halving bound should be more precisely $\mathrm{max}(2^{-n},v(\alpha))$. Random sampling displays good agreement with the expected typical volumes. Most notably, the label-agnostic AL-AMP algorithm tightly follows the volume-halving bound $\mathrm{max}(2^{-n},v(\alpha))$, thus reaching close to optimal possible performance. Since for large α the behavior of $v(\alpha) = \mathrm{const.}/\alpha$ [15] we conclude that the AL-AMP algorithm will reach close to minimum possible Gardner volumes for a budget $n \sim {\cal O}[\log(\alpha)]$.

Zoom In Zoom Out Reset image size

Theoretical justification of this very good performance is however yet to be established: as a matter of fact, due to the purely sequential nature of the employed AL scheme we are not accounting for possible correlations between subsequent queries. Moreover, in machine learning practice, higher order methods for AL (like expected model change [36], with a $\mathcal{O}(N^4)$ complexity) have been shown to perform potentially better than uncertainty sampling. We conjecture the observed optimal performance of AL-AMP can be traced back to the noiseless nature of the considered learning setting and leave the problem of studying AL in the presence of noise for future work. Note that, even in our pool-based setting, we thus obtain an exponential reduction in the number of samples, similar to the original QBC work [4].

The label-informed AL-AMP also approaches the theoretically minimal volume but not as closely. We remark that an important limit of the AL-AMP algorithm comes from the fact that AMP is not guaranteed to provide good estimators (or converge at all) with correlated data. For example, in the numerical experiments for obtaining the informed AL-AMP curve, we had to resort to mild damping schemes in the message-passing to allow fixed-points being reached. This effect was stronger for the label-informed algorithm than for the label-agnostic one.

In figure 7, we provide a numerical comparison of the performance of the agnostic AL-AMP and the other above mentioned label-agnostic AL algorithms. The finite size experiments were run at N = 2 × 10³, while here we set α = 10. Note that, while the mentioned different AL strategies where employed for selecting the labeled subset, in all cases supervised learning and the related performance estimates were obtained by running AMP. In the plot, we can see that, while AL-AMP is able to extract very close to the maximum amount of information from each query (one bit per pattern, until the volume v(α) is saturated), other heuristics with the same computational complexity are sub-optimal. In particular, in the simplified QBC we observe that increasing the size K of the committee does not yield very noticeable change in its performance, most probably because the committee cannot cover a sufficient portion of the version space if the computational cost is to be kept reasonable. On the other hand, using the information of the magnitude pre-activations allows better performance while being also more time-efficient, since only a single perceptron, rather than a committee thereof, has to be trained at each step. The logistic loss allows a rather good performance, close to that of AL-AMP, while the uncertainty sampling with the perceptron algorithm yields a mitigated performance.

Zoom In Zoom Out Reset image size

We leave a more systematic bench-marking of the many existing strategies for future work, stressing the fact that, while there certainly exist more involved procedures that can yield better performance than the presented heuristics, the absolute performance bounds still apply, agnostic of the implemented AL strategy. Because the reported results in the present work are specific to the GLM setup with Gaussian data, future investigations should also be concerned with extending the AL-AMP procedure to real-word datasets, as opposed to the synthetic data used in the present work, and observe whether the good performance generalizes. It should be stressed that such endeavor would likely require to use a variant of AMP, namely vectorial AMP [37], more robust to mismatches between data and model assumptions.

The canonical way of carrying out the extremization in equation (91) is to take the saddle-point equations (zero-gradient conditions) and to solve them. The saddle point equations associated to Φ(β, φ) (equation (91)) read:

$$\begin{align} m^{t} & = \frac{\hat{m}^{t}}{1+\hat{r}^{t}-(\beta-1)\hat{Q}^{t}+\beta\hat{q}^{t}} \end{align} \tag{ A.1 }$$

$$\begin{align} q^{t} & = \frac{\hat{q}^{t}+(\hat{m}^{t})^{2}}{(1+\hat{r}^{t}-(\beta-1)\hat{Q}^{t}+\beta\hat{q}^{t})^{2}} \end{align} \tag{ A.2 }$$

$$\begin{align} Q^{t} & = \frac{\hat{q}^{t}+(\hat{m}^{t})^{2}}{(1+\hat{r}^{t}-(\beta-1)\hat{Q}^{t}+\beta\hat{q}^{t})^{2}}+\frac{1}{\beta}\frac{1}{1+\hat{r}^{t}-(\beta-1)\hat{Q}^{t}+\beta\hat{q}^{t}}-\frac{1}{\beta}\frac{1}{1+\hat{r}^{t}+\hat{Q}^{t}} \end{align} \tag{ A.3 }$$

$$\begin{align} r^{t} & = \frac{\hat{q}^{t}+(\hat{m}^{t})^{2}}{(1+\hat{r}^{t}-(\beta-1)\hat{Q}^{t}+\beta\hat{q}^{t})^{2}}+\frac{1}{\beta}\frac{1}{1+\hat{r}^{t}-(\beta-1)\hat{Q}^{t}+\beta\hat{q}^{t}}+\frac{\beta-1}{\beta}\frac{1}{1+\hat{r}^{t}+Q^{t}} \end{align} \tag{ A.4 }$$

$$\begin{align} X^{t} & = -\frac{1}{\sqrt{r^{t}-Q^{t}}}(\sqrt{Q^{t}-q^{t}}\zeta+\sqrt{q^{t}}\eta) \end{align} \tag{ A.5 }$$

$$\begin{align} \hat{q}^{t+1} & = 2\alpha \int D\eta H\left(-\sqrt{\frac{(m^{t})^{2}}{q^{t}-(m^{t})^{2}}}\eta\right)\frac{e^{2\phi}}{2\pi(r^{t}-Q^{t})}\left[\frac{\int D\zeta H(X^{t})^{\beta-1}e^{-\frac{1}{2}(X^{t})^{2}}}{1+e^{\phi}\int D\zeta H(X^{t})^{\beta}}\right]^{2} \end{align} \tag{ A.6 }$$

$$\begin{align} \hat{m}^{t+1} & = 2\alpha\int d\eta\frac{1}{(2\pi)^{\frac{3}{2}}\sqrt{r-Q}}\frac{1}{\sqrt{1-\frac{(m^{t})^{2}}{q^{t}}}}e^{-\frac{1}{2}\frac{\eta^{2}}{1-\frac{(m^{t})^{2}}{q^{t}}}}\frac{e^{\phi}\int D\zeta H(X^{t})^{\beta-1}e^{-\frac{1}{2}(X^{t})^{2}}}{1+e^{\phi}\int D\zeta H(X^{t})^{\beta}} \end{align} \tag{ A.7 }$$

$$\begin{align} \hat{Q}^{t+1} & = \frac{2\alpha}{r^{t}-Q^{t}} \int D\eta H\left(-\sqrt{\frac{(m^{t})^{2}}{q^{t}-(m^{t})^{2}}}\eta\right)\frac{\int D\zeta H(X^{t})^{\beta-2}e^{-(X^{t})^{2}}\frac{1}{2\pi}}{1+e^{\phi}\int D\zeta H(X^{t})^{\beta}}, \end{align} \tag{ A.8 }$$

$$\begin{align} \hat{r}^{t+1} & = -\frac{2\alpha}{r^{t}-Q^{t}} \int D\eta H\left(-\sqrt{\frac{(m^{t})^{2}}{q^{t}-(m^{t})^{2}}}\eta\right)\frac{\int D\zeta H(X^{t})^{\beta-1}X^{t}e^{-\frac{1}{2}(X^{t})^{2}}\frac{1}{\sqrt{2\pi}}}{1+e^{\phi}\int D\zeta H(X^{t})^{\beta}}. \end{align} \tag{ A.9 }$$

In practice, equations (A.1)–(A.9) are iterated until convergence. The time indices are derived from an independent computation using AMP [13, 24], not shown here. They indicate in which order the equations ought to be iterated in order to converge. Remark that the update schedule very simply consists in updating in parallel all order parameters m, q, Q, r then all auxiliary (hatted) order parameters $\hat{m},\hat{q},\hat{Q},r$. After convergence the order parameters can be used in equation (91) to evaluate the free entropy Φ and subsequently evaluate the complexity Σ(n, v) by inverting the Legendre transform (9).

We present for illustration in figure A1 the complexity curves for α = 10 for some budgets n, and refer the interested reader to figure 1 for the same plot at pool size α = 3. As the budget n is increased, smaller values of Gardner volumes become accessible, provided sufficiently informative subset are found. A detailed discussion can further be found in section 3.

Zoom In Zoom Out Reset image size

We depart from the RS setting and assume the disorder (selection variables σ^a ) to be 1RSB, with Parisi parameter noted τ [25, 26]. For convenience we shall replace $1\leqslant a \leqslant s$ by a double index $(a^{^{\prime}},a)$, with $1\leqslant a^{^{\prime}}\leqslant \frac{s}{\tau}$ the 1RSB cluster index and $1\leqslant a\leqslant \tau$ indexing the selection variables inside a same block. As a consequence the $\boldsymbol{x}$ variables will carry a triple index $(a^{^{\prime}},a,\alpha)$. We also need to differentiate the former q index into q₁ (for students seeing different disorders pertaining to the same 1RSB cluster) and q₀ (for students seeing disorders from different clusters).

Let us note as before $\boldsymbol{\tilde{Q}} = \boldsymbol{Q}^{-1}$ with elements $\tilde{r}^{0},\tilde{m},\tilde{r},\tilde{Q},\tilde{q_{1}},\tilde{q_{0}}$. Inverting the matrix is tantamount to solving the equations:

$$\begin{align} &r_{0}\tilde{r_{0}}+\beta s m\tilde{m}m = 1, \end{align} \tag{ B.1 }$$

$$\begin{align} &r_{0}\tilde{m}+m\tilde{r}+(beta-1)m\tilde{Q}+\beta(\tau-1) m\tilde{q_{1}}+\beta(s-\tau)m\tilde{q_{0}} = 0, \end{align} \tag{ B.2 }$$

$$\begin{align} &m\tilde{r_{0}}+r\tilde{m}+(\beta-1)Q\tilde{m} +\beta(\tau-1)q_{1}\tilde{m}+\beta(s-\tau)q_{0}\tilde{m} = 0, \end{align} \tag{ B.3 }$$

$$\begin{align} &m\tilde{m}+r\tilde{r}+(\beta-1)Q\tilde{Q}+\beta(\tau-1) q_{1}\tilde{q_{1}}+\beta(s-\tau)q_{0}\tilde{q_{0}} = 1, \end{align} \tag{ B.4 }$$

$$\begin{align} &m\tilde{m}+r\tilde{Q}+Q\tilde{r}+(\beta-2)Q\tilde{Q} +\beta(\tau-1) q_{1}\tilde{q_{1}}+\beta(s-\tau)q_{0}\tilde{q_{0}} = 0, \end{align} \tag{ B.5 }$$

$$\begin{align} &m\tilde{m}+r\tilde{q_{1}}+(\beta-1)Q\tilde{q_{1}}+q_{1}\tilde{r} +(\beta-1)q_{1}\tilde{Q}+\beta(\tau-2)q_{1}\tilde{q_{1}} +\beta(s-\tau)q_{0}\tilde{q_{0}} = 0, \end{align} \tag{ B.6 }$$

$$\begin{align} &m\tilde{m}+r\tilde{q_{0}}+(\beta-1)Q\tilde{ q_{0}}+\beta(\tau-1)q_{1}\tilde{q_{0}}+q_{0}\tilde{r} +(\beta-1)q_{0}\tilde{Q}+\beta(\tau-1)q_{0}\tilde{q_{1}}\nonumber\\ & +\beta(s-2\tau)q_{0}\tilde{q_{1}}+\beta(s-2\tau)q_{0}\tilde{q_{0}} = 0. \end{align} \tag{ B.7 }$$

Solutions are :

$$\begin{align} \tilde{r_{0}} &= \frac{(1-\beta)Q-r+\beta(\tau-s)q_{0 }+\beta(1-\tau)q_{1}}{-rr^{0}+Qr^{0}(1-\beta) +\beta m s^{2}+\beta r^{0}(q_{1}(1-\tau)+q_{0}(\tau-s))}, \end{align} \tag{ B.8 }$$

$$\begin{align} \tilde{m} &= \frac{m}{-rr^{0}+Qr^{0}(1-\beta) +\beta m s^{2}+\beta r^{0}(q_{1}(1-\tau)+q_{0}(\tau-s))}, \end{align} \tag{ B.9 }$$

$$\begin{align} \tilde{r} &= \frac{1}{\beta}\left(\frac{\beta-1}{r-Q} +\frac{1}{(\beta-1)Q-\beta q_{1}+r}\right)\nonumber\\ &\quad+\frac{r_{0}}{ s\beta[(\beta-1)Qr_{0}+rr_{0}+\beta(-ms^{2}+q_{0}(q_{1} (\tau-1)+q_{0}(s-\tau)))] } \nonumber\\ &\quad+\frac{1}{\beta}\frac{Q(1-\beta)-r+\beta(q_{1}(1-s )+q_{0}s)}{s((1-\beta)Q-\beta q_{1}+r)((1-\beta)Q+r+\beta q_{1}(\tau-1)-\beta q_{0}\tau)}, \end{align} \tag{ B.10 }$$

$$\begin{align} &\tilde{Q} = \frac{r_{0}}{-\beta s(-rr^{0}+Qr^{0}(1-\beta) +\beta m s^{2}+\beta r^{0}(q_{1}(1-\tau)+q_{0}(\tau-s)))} \nonumber\\ &\quad+\frac{1}{(\beta-1)Q-\beta q_{1}+r}\left[ \frac{Q-q_{1}}{Q-r}+\frac{Q(1-\beta)-r+\beta(q_{1}(1-s )+q_{0}s)}{\beta s((\beta-1)Q+r+\beta q_{1}(\tau-1)-\beta q_{0}\tau))} \right], \end{align} \tag{ B.11 }$$

$$\begin{align} \tilde{q_{1}} = &\frac{r^{0}}{-\beta s(-rr^{0}+Qr^{0}(1-\beta) +\beta m s^{2}+\beta r^{0}(_{1}(1-\tau)+q_{0}(\tau-s))) }\nonumber\\ &+\frac{Q(1-\beta)-r+\beta(q_{1}(1-s )+q_{0}s)}{\beta s ((\beta-1)Q-\beta q_{1}+r)((\beta-1)Q+r+\beta q_{1}(\tau-1)-\beta q_{0}\tau))}, \end{align} \tag{ B.12 }$$

$$\begin{align} \tilde{q_{0}} = &\frac{-q_{0}r^{0}+m^{2}}{(r^{0}(Q(1-\beta)-r+\beta (q_{1}(1-\tau)+q_{0}(\tau-s))) +\beta m s^{2})}\nonumber\\ &\times \frac{+}{(\beta-1)Q+r+\beta( q_{1}(\tau-1)-q_{0}\tau)}. \end{align} \tag{ B.13 }$$

One can also compute $\mathrm{det}\boldsymbol{Q}$ by finding the eigenvectors. There is a couple of eigenvectors with product

$$\begin{equation} r^{0}(r+(\beta-1)Q+\beta(\tau-1)q_{1}+\beta(s-\tau)q_{0})-\beta s m^{2}, \end{equation} \tag{ B.14 }$$

and $\frac{s}{\tau}$ eigenvectors with eigenvalue:

$$\begin{equation} r+(\beta-1)Q-\beta \tau q_{0}+\beta(\tau-1)q_{1}, \end{equation} \tag{ B.15 }$$

and $\frac{s}{\tau}(\tau-1)$ eigenvectors with eigenvalue:

$$\begin{equation} r+(\beta-1)Q-\beta q_{1}. \end{equation} \tag{ B.16 }$$

Thus,

$$\begin{align} \mathrm{det}\boldsymbol{Q} &= \mathrm{ln}(r^{0} (r+(\beta-1)Q+\beta(\tau-1)q_{1}+\beta(s-\tau)q_{0})-\beta sm^{2})\nonumber\\ &\quad +(\frac{s}{\tau}-1)\mathrm{ln}(r+(\beta-1)Q-\beta\tau q_{0}+\beta (\tau-1)q_{1})+\frac{s}{\tau}(\beta \tau -1)\mathrm{ln} (r+(\beta-1)Q-\beta q_{1}). \end{align} \tag{ B.17 }$$

The trace term $\mathrm{Tr}\boldsymbol{\hat{Q}}\boldsymbol{Q}$ in (16) now reads with the 1RSB ansatz

$$\begin{equation} r_{0}\hat{r}_{0}+\beta s m\hat{m} -\frac{r\hat{r}}{2} \beta s +\frac{\beta(\beta-1)}{2}s Q\hat{Q}+\frac{\beta s} {2}\beta(\tau-1)q_{1}\hat{q}_{1}+\frac{\beta s}{2} \beta (s-\tau)q_{0}\hat{q}_{0}. \end{equation} \tag{ B.18 }$$

For the I_X term we proceed in very similar manner as in the RS case, see section 3.1. The end result is:

$$\begin{align} &I_{X} = \frac{1}{\tau}\int D\eta I^{0}_{X}\mathrm{ln} I^{1}_{X}, \end{align} \tag{ B.19 }$$

$$\begin{align} &I^{0}_{X} = \int d x^{0}\overline{P_{X}}(x^{0}) e^{-\frac{\hat{m}^{2}}{2\hat{q}_{0}}(x^{0})^2+\frac{\hat{m}} {\sqrt{\hat{q_{0}}}}\eta x^{0}}, \end{align} \tag{ B.20 }$$

$$\begin{align} &I^{1}_{X} = \int D\xi\left[ \int D\lambda\left( \int dx P_{X}(x)e^{-\frac{\hat{r}+\hat{Q}}{2}x^2 +(\sqrt{\hat{Q}-\hat{q_{1}}}\lambda +\sqrt{\hat{q_{1}}-\hat{q_{0}}}\xi +\sqrt{\hat{q_{0}}}\eta)} \right)^{\beta} \right]^{\tau}. \end{align} \tag{ B.21 }$$

Note that the results

$$\begin{align} &\hat{r}^{0} = 0 \end{align} \tag{ B.22 }$$

$$\begin{align} &r^{0} = \mathbb{E}_{x^{0}}(x^{0})^2, \end{align} \tag{ B.23 }$$

also carry through to the 1RSB case. The treatment of the I_Y term requires, as in the non-RSB case, more care. The computation is very analogous (though more lengthy) and shall not be further detailed presently. We report only the end result:

$$\begin{align} &I_{Y} = \frac{\alpha}{\tau}\int D\rho I_{Y}^0\mathrm{ln} I_{Y}^1, \end{align} \tag{ B.24 }$$

$$\begin{align} &I^0_{Y} = \int Dh^{0}\overline{P_{\mathrm{out}}}\left(y| \sqrt{\frac{q_{0}r^0-m^2}{q_{0}}}h^0+\sqrt{\frac{m^2}{q_{0}}}\right), \end{align} \tag{ B.25 }$$

$$\begin{align} &I_{Y}^1 = \int D\eta \left[ 1+e^{\phi}\int D\zeta\left( P_{\mathrm{out}}\left(y|\sqrt{r-Q}h+\sqrt{Q-q_{1}}\zeta +\sqrt{q_{1}-q_{0}}+\sqrt{q_{0}}\rho\right) \right)^{\beta} \right]^{\tau}. \end{align} \tag{ B.26 }$$

In summary the 1RSB free entropy reads:

$$\begin{align} \Phi_{\mathrm{1RSB}} &= \underset{r,m,Q,q_{1},q_{0},\hat{r} ,\hat{m},\hat{Q},\hat{q_{1}},\hat{q_{0}}}{\mathrm{extr}} -\beta m\hat{m}+\frac{\beta}{2}r\hat{r}-\frac{\beta(\beta-1)}{2} Q\hat{Q}-\frac{\beta \tau}{2}\beta(\tau-1)q_{1}\hat{q_{1}}\nonumber\\ & \quad +\frac{\beta^2\tau}{2}q_{0}\hat{q_{0}}+\frac{1}{\tau}\int D\eta I^{0}_{X}\mathrm{ln} I^{1}_{X}+\frac{\alpha}{\tau}\int D\rho I_{Y}^0\mathrm{ln} I_{Y}^1, \end{align} \tag{ B.27 }$$

$$\begin{align} I^{0}_{X} &= \int d x^{0}\overline{P_{X}}(x^{0}) e^{-\frac{\hat{m}^{2}}{2\hat{q}_{0}}(x^{0})^2+\frac{\hat{m}} {\sqrt{\hat{q_{0}}}}\eta x^{0}}, \end{align} \tag{ B.28 }$$

$$\begin{align} I^{1}_{X} & = \int D\xi\left[ \int D\lambda\left( \int dx P_{X}(x)e^{-\frac{\hat{r}+\hat{Q}}{2}x^2 +(\sqrt{\hat{Q}-\hat{q_{1}}}\lambda +\sqrt{\hat{q_{1}}-\hat{q_{0}}}\xi +\sqrt{\hat{q_{0}}}\eta)} \right)^{\beta} \right]^{\tau}, \end{align} \tag{ B.29 }$$

$$\begin{align} I^0_{Y} &= \int Dh^{0}\overline{P_{\mathrm{out}}}\left(y| \sqrt{\frac{q_{0}r^0-m^2}{q_{0}}}h^0+\sqrt{\frac{m^2}{q_{0}}}\right), \end{align} \tag{ B.30 }$$

$$\begin{align} I_{Y}^1 &= \int D\eta \left[ 1+e^{\phi}\int D\zeta\left( P_{\mathrm{out}}\left(y|\sqrt{r-Q}h+\sqrt{Q-q_{1}}\zeta +\sqrt{q_{1}-q_{0}}+\sqrt{q_{0}}\rho\right) \right)^{\beta} \right]^{\tau}. \end{align} \tag{ B.31 }$$

We now proceed to specialize the 1RSB free entropy (B.27), derived above for a generic GLM, for the special case of a perceptron. The only term that differs non-trivially from the RS specialization reported in section 3.2 is $I_{X}^1$, for which one further (Gaussian) integration has to be carried out. We give the final formula for the perceptron free entropy:

$$\begin{align} \Phi_{\mathrm{1RSB}} = & -\beta m\hat{m}+\frac{\beta}{2}r\hat{r}-\frac{\beta(\beta-1)}{2} Q\hat{Q}-\frac{\beta }{2}\beta(\tau-1)q_{1}\hat{q_{1}}+\frac{\beta^2\tau}{2}q_{0}\hat{q_{0}} \nonumber\\ &-\frac{\beta-1}{2}\mathrm{ln}(1+\hat{r}+\hat{Q}) +\frac{1}{2}(\frac{1}{\tau}-1)\mathrm{ln}(1+\hat{r}- (\beta-1)\hat{Q}+\beta\hat{q_{1}}) \nonumber\\ &-\frac{1}{2\tau}\mathrm{ln}(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q_{1}} -\beta\tau(\hat{q_{1}}-\hat{q_{0}}))\nonumber\\ &+\frac{\beta}{2}\frac{\hat{q_{0}}+\hat{m}^2}{1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q_{1}} -\beta\tau(\hat{q_{1}}-\hat{q_{0}})}+\frac{2\alpha}{\tau}\int D\rho H\left(-\sqrt{\frac{m^2}{q_{0}-m^2}}\rho\right) \nonumber\\ & \times \mathrm{ln}\left\{\int D\eta \left[ 1+e^{\phi}\int D\zeta H\left(-\frac{\sqrt{Q-q_{1}}\zeta+\sqrt{q_{1}-q_{0}} \eta+\sqrt{q_{0}}\rho}{\sqrt{r-Q}}\right)^{\beta} \right]^{\tau}\right\}. \end{align} \tag{ B.32 }$$

Having derived the free entropy (B.32) in the 1RSB ansatz, we can now study the stability of the RS solution with respect to an infinitesimal 1RSB perturbation. To that end we consider a 1RSB ansatz departing infinitesimally from the RS form:

$$\begin{align} &q_{1}-q_{0} = \epsilon = o(1), \end{align} \tag{ B.33 }$$

$$\begin{align} &\hat{q_{1}}-\hat{q_{0}} = \hat{\epsilon} = o(1). \end{align} \tag{ B.34 }$$

We name $q,\hat{q}$ the common order 0 value for these overlaps. The entropic part (trace term and I_X ) of the free entropy (B.32) then reads to first order in $\epsilon, \hat{\epsilon}$

$$\begin{align} &-\beta m\hat{m}+\frac{\beta}{2}r\hat{r}-\frac{\beta(\beta-1)}{2} Q\hat{Q} -\frac{\beta-1}{2}\mathrm{ln}(1+\hat{r}+\hat{Q}) -\frac{\beta^2(\tau-1)}{2}(q\hat{q}+\epsilon \hat{q}+\hat{\epsilon}q) \nonumber\\ &+\frac{\beta^2\tau}{2}q\hat{q}-\frac{1}{2} \mathrm{ln}(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q}) +\frac{\beta}{2}\frac{\hat{q}+\hat{m}^2}{1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q}}\nonumber\\ &-\frac{\beta}{2}\frac{(\hat{q}+\hat{m}^2)\beta(1-\tau)\hat{\epsilon}}{(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})^2}. \end{align} \tag{ B.35 }$$

Prior to expanding the energetic part I_Y it is convenient to define the shorthand

$$\begin{equation} X = -\frac{1}{\sqrt{r-Q}}(\sqrt{Q-q}\zeta+\sqrt{q}\rho), \end{equation} \tag{ B.36 }$$

as we did in the RS computations, see section 3.2. Expanding the argument of $* = \log\int D\eta \left[ 1+e^{\phi}\int D\zeta H\left(\ldots\right)^{\beta} \right]^{\tau}$

$$\begin{align} *& = \log \left\{\int D\eta [ 1+e^{\phi}\int D\zeta H(X)^{\beta}\right.\nonumber\\ &\quad \left. +\,e^{\phi}\int D\zeta (H^{\beta})^{^{\prime}}(X)\left(\frac{\zeta\epsilon}{2\sqrt{r-Q}\sqrt{Q-q}} -\frac{\sqrt{\epsilon}\eta}{\sqrt{r-Q}}\right)+\frac{1}{2} (H^{\beta})^{^{\prime\prime}}(X)\frac{\epsilon\eta^2}{r-Q} +\ldots]^{\tau}\right\} \end{align} \tag{ B.37 }$$

$$\begin{align} & = \log \Bigg\{\left(1+e^{\phi} \int D\zeta H(X)^{\beta}\right)^\tau \nonumber \\ & \quad \times \int D\eta \Bigg[ 1+ +\epsilon\tau\frac{e^{\phi}\int D\zeta ((H^{\beta})^{^{\prime}}(X)\frac{\zeta}{2\sqrt{r-Q}\sqrt{Q-q}} +\frac{1}{2} (H^{\beta})^{^{\prime\prime}}(X)\frac{\eta^2}{r-Q})}{1+e^{\phi}\int D\zeta H^{\beta}(X)}\nonumber\\ & \quad +\,\epsilon\eta^2\frac{\tau(\tau-1)}{2}\left(\frac{\frac{e^{\phi}}{r-Q}\int D\zeta (H^{\beta})^{^{\prime}}(X)}{1+e^{\phi}\int D\zeta H^{\beta}(X)} \right)^2 \Bigg]\Bigg\} \end{align} \tag{ B.38 }$$

$$\begin{align} & = \tau\log(1+e^{\phi}\int D\zeta H(X)^{\beta})+\log\left\{1+\epsilon\frac{\tau(\tau-1)}{2}\left(\frac{\frac{e^{\phi}}{r-Q}\int D\zeta (H^{\beta})^{^{\prime}}(X)}{1+e^{\phi}\int D\zeta H^{\beta}(X)} \right)^2\right\} \nonumber \\ & = \tau\log(1+e^{\phi}\int D\zeta H(X)^{\beta})+\epsilon\frac{\tau(\tau-1)}{2}\left(\frac{\frac{e^{\phi}}{r-Q}\int D\zeta (H^{\beta})^{^{\prime}}(X)}{1+e^{\phi}\int D\zeta H^{\beta}(X)} \right)^2. \end{align} \tag{ B.39 }$$

In going from the first to the second line we used that the $\mathcal {O}(\epsilon^{\frac{1}{2}})$ term is killed by the integration over η. The first fraction in the second line is found to be vanishing using an integration by parts.

The variables $\epsilon,\hat{\epsilon}$ intervene in the saddle-point Equations. If iterating the SP equations induce $\epsilon,\hat{\epsilon}$ to become large, then the assumption that the RS fixed point is stable ceases to hold. To obtain the dynamical equations for the pair $\epsilon,\hat{\epsilon}$ one has to derive the SP equations associated to zero-gradient conditions in the q and $\hat{q}$ direction. Note that the derivatives of terms not involving or $\hat{\epsilon}$ shall eventually sum to zero, since we assume to be at the RS ($\epsilon = \hat{\epsilon} = 0$) fixed points. The first $\partial_{\hat{q}}\Phi_{\mathrm{1RSB}} = 0$ equation reads

$$\begin{equation} \epsilon = -\left(2\beta\frac{\hat{q}+\hat{m}^2}{(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})^3} -\frac{1}{(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})^2}\right)\hat{\epsilon}, \end{equation} \tag{ B.40 }$$

while the $\partial_{q}\Phi_{\mathrm{1RSB}} = 0$ implies:

$$\begin{align} \hat{\epsilon}& = \left(-\frac{2\alpha}{\beta^2}\frac{\partial}{\partial q} \int D\rho H\left(-\sqrt{\frac{m^2}{q-m^2}}\rho\right) \left(\frac{\frac{e^{\phi}}{r-Q}\int D\zeta (H^{\beta})^{^{\prime}}(X)}{1+e^{\phi}\int D\zeta H^{\beta}(X)} \right)^2\right)\epsilon \end{align} \tag{ B.41 }$$

$$\begin{align} & = -\frac{\partial}{\partial q}\left(\frac{2\alpha e^{2\phi}}{\beta^2 (r-Q)(Q-q)} \int D\rho H\left(-\sqrt{\frac{m^2}{q-m^2}}\rho\right) \left(\frac{\int D\zeta\,\zeta\, H^{\beta}(X)}{1+e^{\phi}\int D\zeta H^{\beta}(X)} \right)^2\right)\epsilon. \end{align} \tag{ B.42 }$$

We choose not to explicit the q derivative as the expression is rather large. The stability condition, expressing the fact that the dynamical system (B.40) and (B.41) converges to $\epsilon = \hat{\epsilon} = 0$, then reads:

$$\begin{align} &\left|\left(2\beta\frac{\hat{q}+\hat{m}^2}{(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})^3} -\frac{1}{(1+\hat{r}-(\beta-1)\hat{Q}+\beta\hat{q})^2}\right)\right|\nonumber\\ & \quad \times\left|-\frac{\partial}{\partial q}\left(\frac{2\alpha e^{2\phi}}{\beta^2 (r-Q)(Q-q)} \int D\rho H\left(-\sqrt{\frac{m^2}{q-m^2}}\rho\right) \left(\frac{\int D\zeta\,\zeta\, H^{\beta}(X)}{1+e^{\phi}\int D\zeta H^{\beta}(X)} \right)^2\right)\right|\lt 1. \end{align} \tag{ B.43 }$$

The stability condition (92) can be evaluated numerically using the values for the order parameters resulting from the convergence of the saddle-point equations (A.1)–(A.9). The results are presented in figure 5. As the budget n increases, the region of validity of the RS assumptions diminishes. This seems to suggest that the selections $\{\sigma_{\mu}\}_{1\leqslant\mu\leqslant\alpha N}$ leading to atypical enough volumes v are grouped into isolated clusters in configuration space, i.e. the landscape is 1RSB, this effect being more pronounced for larger values of budget. That the RS assumption should break down away from typicality is expected. However, in most known settings, this simple ansatz proves to be a very good approximation nonetheless, and taking into account further steps of RS breaking usually yields only minor improvement [12]. We therefore believe that the error bounds here reported have a good degree of accuracy, although they may certainly be improved if further symmetry breaking are taken into account. The precise evaluation of these corrections is left for future investigation.