Replica symmetry breaking in supervised and unsupervised Hebbian networks

We consider a system of N neurons, modelled via Ising spins $\sigma_i \in \{-1, 1 \}$, with $i = {1,\ldots,N}$, where neurons interact via P-node interactions. We assume to have K Rademacher archetypes, defined as N-dimensional vectors $\boldsymbol{\xi}^{\mu}$, with $\mu = 1,\ldots, K$, where each entry is drawn randomly and independently from the distribution

$$\begin{align} \mathbb{P}\left(\xi_i^\mu\right) = \dfrac{1}{2} \left( \delta_{\xi_i^\mu,+1} + \delta_{\xi_i^\mu,-1} \right). \end{align} \tag{ 2.1 }$$

Such archetypes are not provided to the network, instead the network will have to infer them by experiencing solely noisy or corrupted versions of them. In particular, we assume that for each archetype µ, M examples $\boldsymbol{\eta}^{\mu,a}$, with $a = 1, \ldots, M$, are available, which are corrupted versions of the archetype, such that

$$\begin{align} \mathbb{P}\left(\eta_{i}^{\mu,a}|\xi_i^\mu\right) = \frac{1-r}{2} \delta_{\eta_{i}^{\mu,a},-\xi_i^\mu} + \frac{1+r}{2} \delta_{\eta_{i}^{\mu,a},+\xi_i^\mu}, \end{align} \tag{ 2.2 }$$

where $r \in [0,1]$ controls the quality of the dataset, i.e. for r = 1 the example matches perfectly the archetype, while for r = 0 it is purely stochastic. To quantify the information content of the dataset it is useful to introduce the variable

$$\begin{align} \rho = \frac{1-r^2}{Mr^2} \end{align} \tag{ 2.3 }$$

that we shall refer to as the dataset entropy ⁷ . We note that ρ vanishes either when the examples are identical to the archetypes (i.e. r → 1) or when the number of examples is infinite (i.e. $M \to \infty$), or both. If we regard the traditional DAM model as storing patterns and the unsupervised DAM model as learning patterns from stored examples, then ρ can be thought of as the parameter that quantifies the difference between storing and learning (which is larger, the larger ρ).

Definition 1. The cost function (or Hamiltonian) of the 'unsupervised' DAM model is

$$\begin{align} \mathcal{H}_{N}^{\left(P\right)}\left(\boldsymbol{\sigma} \vert \boldsymbol{\eta}\right) = & -\dfrac{1}{\,\mathcal{R}^{P/2}\,M\,N^{P-1}}\displaystyle\sum\limits_{\mu = 1}^{K}\displaystyle\sum\limits_{a = 1}^{M}\left(\displaystyle\sum\limits_{i_1 \lt \cdots \lt i_P}^{N,\cdots, N}\eta_{i_1}^{\mu,a }\cdots\eta_{i_P}^{\mu,a }\sigma_{_{i_1}}\cdots\sigma_{_{i_P}}\right), \end{align} \tag{ 2.4 }$$

where the constant $\mathcal{R}^{P/2}$, with $\mathcal{R} = r^2(1+\rho)$ and ρ defined in (2.3), is included in the definition for mathematical convenience, while the factor $N^{P-1}$ ensures that the Hamiltonian is extensive in the network size N. On the left hand side (LHS), the label (P) denotes the order of the multi-node interactions and η is a short-hand for the collection of all the examples, for all the archetypes $\{\boldsymbol{\eta}^{\mu,a}\}_{\mu = 1,a = 1}^{K,M}$.

Remark 1. For P = 2, i.e. pairwise interactions, the unsupervised DAM model reduces to the Hopfield neural network in the unsupervised setting [33]. In addition, in the absence of dataset corruption, i.e. r = 1, $\mathcal{R} = 1$ and the model reduces to the standard Hopfield model, as analysed in [4, 12].

Definition 2. The partition function associated to the Hamiltonian (2.4) at inverse noise level $\beta\in \mathcal{R}^+$ is defined as

$$\begin{align} \mathcal{Z}^{\left(P\right)}_{N}\left( \boldsymbol{\eta}\right) = \sum_{\boldsymbol{\sigma}}^{2^N} \exp \left( -\beta \mathcal{H}^{\left(P\right)}_{N}\left(\boldsymbol{\sigma} \vert \boldsymbol{\eta}\right)\right) = : \sum_{\boldsymbol{\sigma}}^{2^N} \mathcal{B}^{\left(P\right)}_{N}\left(\boldsymbol{\sigma} \vert \boldsymbol{\eta}\right), \end{align} \tag{ 2.5 }$$

where $\mathcal{B}^{(P)}_{N}(\boldsymbol{\sigma} \vert \boldsymbol{\eta})$ is referred to as the Boltzmann factor. At finite network size N, the free energy of the model $\mathcal{F}^{(P)}_{N}$ is given by

$$\begin{align} \mathcal -\beta \mathcal{F}^{\left(P\right)}_{N} = \frac{1}{N}\mathbb{E}\log \mathcal{Z}^{\left(P\right)}_{N}\left(\boldsymbol{\eta}\right) \end{align} \tag{ 2.6 }$$

where $\mathbb{E}[.]$ denotes the average over the realizations of examples η , regarded as quenched.

We are interested in the behaviour of the system in the limit of large system size $N\to \infty$ and finite network load α, as specified by the following

Definition 3. In the thermodynamic limit $N \to \infty$, the load of the unsupervised DAM model is defined as

$$\begin{align} \lim_{N\to +\infty} \frac{K}{N^{P-1}} = : \alpha \lt \infty \end{align} \tag{ 2.7 }$$

We will mostly focus on the so-called 'saturated' regime, where α > 0. The regime where the system is away from saturation can be inspected by taking the limit α → 0. For convenience, we also introduce the parameter γ, defined by

$$\begin{align} \alpha = \gamma \dfrac{2}{P!}.\end{align} \tag{ 2.8 }$$

The free energy in the thermodynamic limit is denoted as

$$\begin{align} \mathcal F^{\left(P\right)} = \lim_{N \to \infty} \mathcal F^{\left(P\right)}_{N}. \end{align} \tag{ 2.9 }$$

In the following, we focus on the ability of the network to retrieve a single archetype, say ν. Given the invariance of the Hamiltonian w.r.t. permutations of the archetypes, we will set without loss of generality ν = 1. Then, following standard procedures [4], we split the sum over µ in the Hamiltonian into the contribution from µ = 1 (regarded as the signal) and the contributions from the other archetypes µ > 1 (regarded as slow noise). Furthermore, we add in the argument of the exponent of the Boltzmann factor a term $J \sum_i \xi_i^1 \sigma_i$, so that the free energy can serve as a moment generating functional of the so-called Mattis magnetization m₁ (vide infra) by taking derivatives of $\ln\mathcal Z_N^{(P)}$ w.r.t. J. As this term is not part of the original Hamiltonian, J will be set to zero later on.

Starting from (2.5), the resulting partition function is

$$\begin{align} \mathcal{Z}^{\left(P\right)}_{N} \left(\boldsymbol{\eta}\right) = & \lim_{J \rightarrow 0} \sum_{\boldsymbol{\sigma}} \exp \left[ J \sum_{i = 1}^N \xi_i^1 \sigma_i + \dfrac{\beta ^{^{\prime}}}{2\,\mathcal{R}^{P/2} M\,N^{P-1}}\displaystyle\sum\limits_{a = 1}^{M}\left(\sum_{i = 1}^N\eta_{_{i}}^{a,1}\sigma_{_{i}}\right)^{^P} \right. \nonumber \\ &\left.+\dfrac{\beta ^{^{\prime}}P!}{2\mathcal{R}^{P/2} M\,N^{P-1}} \, \displaystyle\sum\limits_{\mu>1}^{K}\displaystyle\sum\limits_{a = 1}^{M} \displaystyle\sum\limits_{\ {i_1 \lt \cdots \lt i_P}}^{N,\cdots,N}\left(\eta^{\mu\,,a}_{i_1}\cdots\eta^{\mu\,,a}_{i_{P}}\:\sigma_{_{i_1}}\cdots\sigma_{_{i_{_{P}}}}\right)\,\right] \end{align} \tag{ 2.10 }$$

where we have set $\beta^{^{\prime}} = 2\beta/ P!$ and for the first pattern (µ = 1) we used $P!\sum_{i_1 \lt\cdots \lt i_P} = \sum_{i_1\neq \cdots\neq i_P}$ and neglected terms which vanish in the thermodynamic limit. Focusing only on the last term in (2.10) we note that the product $\eta_{i_1}^{\mu,a}\ldots \eta_{i_P}^{\mu,a}$ is a random i.i.d. Binomial variable for each $\mu, a$, and the sum over µ converges, by the central limit theorem (CLT), to a Gaussian variable with suitably defined mean and variance, i.e.

$$\begin{align} \dfrac{1}{K}\displaystyle\sum\limits_{\mu>1}^{K}\left(\dfrac{1}{M}\displaystyle\sum\limits_{a = 1}^{M}\eta^{\mu\,,a}_{i_1}\cdots\eta^{\mu\,,a}_{i_{P}}\right)\sim \mu_1+\lambda_{i_1,\ldots,i_P} \frac{\sqrt{\mu_2-\mu_1^2}}{\sqrt{K}},\quad\quad\quad \lambda_{i_1\ldots i_P}\sim {\mathcal N}\left(0,1\right), \end{align} \tag{ 2.11 }$$

where

$$\begin{eqnarray} &&\mu_1 = \mathbb{E}_{\boldsymbol{\xi}}\mathbb{E}_{\left(\boldsymbol{\eta}|\boldsymbol{\xi}\right)}\left[ \dfrac{1}{M}\displaystyle\sum\limits_{a = 1}^{M}\eta^{\mu,a}_{i_1}\cdots\eta^{\mu,a}_{i_{P}} \right] = \mathbb{E}_{\boldsymbol{\xi}} \left[\left(r\xi_{i_1}^\mu\right)\ldots \left(r\xi_{i_P}^\mu\right)\right] = 0 \end{eqnarray} \tag{ 2.12 }$$

and

$$\begin{align} \mu_2& = \mathbb{E}_{\left(\boldsymbol{\xi}\right)}\mathbb{E}_{\left(\boldsymbol{\eta}|\boldsymbol{\xi}\right)}\left[\left(\dfrac{1}{M}\displaystyle\sum\limits_{a = 1}^{M}\eta^{\mu,a}_{i_1}\cdots\eta^{\mu,a}_{i_{P}}\right)^2\right] \nonumber\\ & = \mathbb{E}_{\left(\boldsymbol{\xi}\right)}\mathbb{E}_{\left(\boldsymbol{\eta}|\boldsymbol{\xi}\right)}\left[\dfrac{1}{M^2}\displaystyle\sum\limits_{a = 1}^{M}\left(\eta^{\mu,a}_{i_1}\right)^2\cdots\left(\eta^{\mu,a}_{i_{P}}\right)^2\right]\nonumber\\ &\quad+\mathbb{E}_{\left(\boldsymbol{\xi}\right)}\mathbb{E}_{\left(\boldsymbol{\eta}|\boldsymbol{\xi}\right)}\left[\dfrac{1}{M^2}\displaystyle\sum\limits_{a\neq b}^{M}\left(\eta^{\mu,a}_{i_1}\eta^{\mu,b}_{i_1}\right)\cdots\left(\eta^{\mu,a}_{i_{P}}\eta^{\mu,b}_{i_{P}}\right)\right]\nonumber\\ & = \dfrac{1}{M}+\dfrac{M-1}{M}r^{2P} = r^{2P}\left(1+\dfrac{1-r^{2P}}{M r^{2P}}\right). \end{align} \tag{ 2.13 }$$

This enables us to write

$$\begin{align} \dfrac{1}{K}\displaystyle\sum\limits_{\mu>1}^{K} \left(\dfrac{1}{M}\displaystyle\sum\limits_{a = 1}^{M}\eta^{\mu\,,a}_{i_1}\cdots\eta^{\mu\,,a}_{i_{P}}\right)\sim \dfrac{r^P}{\sqrt{K}}\sqrt{\left(1+\rho_P\right)} \lambda_{i_1\ldots i_P}, \;\;\;\;\; \lambda_{i_1,\ldots, i_P}\sim \mathcal{N}\left(0,1\right) \end{align} \tag{ 2.14 }$$

with

$$\begin{align*} \rho_P = \frac{1-r^{2P}}{Mr^{2P}}.\end{align*}$$

Thus the partition function of the unsupervised DAM model reads as

$$\begin{align} \mathcal{Z}^{\left(P\right)}_{N} \left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}; J\right) & = \lim_{J \rightarrow 0} \sum_{\boldsymbol{\sigma}} \exp \left[ J \sum_{i = 1}^N \xi_i^1 \sigma_i + \dfrac{\beta ^{^{\prime}}}{2\mathcal{R}^{P/2} M\,N^{P-1}}\displaystyle\sum\limits_{a = 1}^{M}\left(\displaystyle\sum\limits_{i}^{N}\eta_{_{i}}^{1,a}\sigma_{_{i}}\right)^{^P} \right. \nonumber \\ &\left.\quad+\frac{\alpha}{\sqrt{K}}\dfrac{\beta ^{^{\prime}}P!\sqrt{\left(1+\rho_P\right)}}{2\left(1+\rho\right)^{P/2}}\left(\displaystyle\sum\limits_{{i_1 \lt\cdots \lt i_P}}^{N,\cdots,N}\lambda_{i_1,\ldots,i_P}\:\sigma_{_{i_1}}\cdots\sigma_{_{i_{_{P}}}}\right)\right]. \end{align} \tag{ 2.15 }$$

To make analytical progress, it is convenient to introduce the order parameters of the model.

Definition 4. The order parameters of the unsupervised DAM model are the Mattis magnetization m₁ of the archetype $\boldsymbol{\xi}^1$, the Mattis magnetizations $n_{a,1}$ of each example $a = 1\ldots M$ of the archetype $\boldsymbol{\xi}^1$ and the two-replica overlaps q_lm , which quantifies the correlations between the variables σ in two copies l and m of the system, sharing the same disorder (i.e. two replicas):

$$\begin{align} m_{1}\left(\boldsymbol{\sigma}\right)&: = \dfrac{1}{N}\displaystyle\sum\limits_{i = 1}^{N}\xi_i^{1}\sigma_i \end{align} \tag{ 2.16 }$$

$$\begin{align} n_{1,a} \left(\boldsymbol{\sigma}\right) &: = \dfrac{r}{\mathcal{R}}\dfrac{1}{N}\displaystyle\sum\limits_{i = 1}^{N}\eta_i^{1\,,a}\sigma_i, \end{align} \tag{ 2.17 }$$

$$\begin{align} q_{lm}\left(\boldsymbol{\sigma}^{\left(l\right)},\boldsymbol{\sigma}^{\left(m\right)}\right)&: = \dfrac{1}{N}\displaystyle\sum\limits_{i = 1}^{N}\sigma_i^{\left(l\right)}\sigma_i^{\left(m\right)}\,. \end{align} \tag{ 2.18 }$$

In the next subsection we calculate the free energy of the system in the thermodynamic limit using Guerra's interpolation technique, assuming one step of replica symmetry breaking (1RSB).

The key idea of Guerra's interpolation method is to define an auxiliary free energy, $\mathcal{F}^{(P)}(t)$, which is function of a parameter $t \in [0,1]$ that interpolates between the free energy of the original DAM model (obtained at t = 1) and the free energy of an exactly solvable one-body model (obtained at t = 0), whose effective fields mimic those of the original model. As the direct calculation of $\mathcal{F}^{(P)}(t = 1)$ is cumbersome, this is obtained by using the fundamental theorem of calculus, namely we first evaluate $\mathcal{F}^{(P)}(t = 0)$ and then we obtain $\mathcal{F}^{(P)}(t = 1)$ as

$$\begin{align} \mathcal{F}^{\left(P\right)}\left(t = 1\right) = \mathcal{F}^{\left(P\right)}\left(t = 0\right) + \int_0^1 \textrm{d}s\,\frac{\textrm{d}\mathcal{F}^{\left(P\right)}}{\textrm{d}t} \vert_{t = s}. \end{align} \tag{ 2.19 }$$

To perform the analysis within the first step of replica symmetry breaking, we make the standard 1RSB assumption [2], stated below:

Assumption 1. In 1RSB assumption, the distribution of the two-replica overlap q₁₂, in the thermodynamic limit, displays two delta-peaks at the values $\bar{q}_1$ and $\bar{q}_2$, and the concentration on these two values is ruled by the parameter $\theta \in [0,1]$, namely

$$\begin{align} \lim_{N \rightarrow + \infty} \mathbb{P}_N\left(q_{12}\right) = \theta \delta \left(q_{12} - \bar{q}_1\right) + \left(1-\theta\right) \delta \left(q_{12} - \bar{q}_2\right), \end{align} \tag{ 2.20 }$$

where

$$\begin{align*} \mathbb{P}_N\left(q_{12}\right) = \sum_{\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}}\sum_{\boldsymbol{\eta}}\mathbb{P}\left(\boldsymbol{\eta}\right)\mathbb{P}_N\left(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}|\boldsymbol{\eta}\right) \delta \left(q_{12}-q_{12}\left(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}\right)\right) \end{align*}$$

and $\mathbb{P}_N(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)}|\boldsymbol{\eta}) = \mathcal{B}_N(\boldsymbol{\sigma}_1|\boldsymbol{\eta})\mathcal{B}_N(\boldsymbol{\sigma}_2|\boldsymbol{\eta})$ as replicas are conditionally independent, given the disorder η .

The Mattis magnetizations m₁ and $n_{1,a}$ are assumed to be self-averaging at their equilibrium values $\bar{m}$ and $\bar{n}$, respectively, namely

$$\begin{align} \lim_{N \rightarrow + \infty} \mathbb{P}_N\left(m_1\right) & = \delta \left(m_1 - \bar{m}\right), \end{align} \tag{ 2.21 }$$

$$\begin{align} \lim_{N \rightarrow + \infty} \mathbb{P}_N\left(n_{1,a}\right) & = \delta \left(n_{1,a} - \bar{n}\right). \end{align} \tag{ 2.22 }$$

where $\mathbb{P}_N(m_1) = \mathbb{E}_{\boldsymbol{\eta}}\sum_{\boldsymbol{\sigma}}\mathcal{B}_N(\boldsymbol{\sigma}|\boldsymbol{\eta}) \delta (m_1-m_1(\boldsymbol{\sigma}))$ and $\mathbb{P}_N(n_{1,a}) = \mathbb{E}_{\boldsymbol{\eta}}\sum_{\boldsymbol{\sigma}}\mathcal{B}_N(\boldsymbol{\sigma}|\boldsymbol{\eta}) \delta (n_{1,a}-n_{1,a}(\boldsymbol{\sigma}))$, with $\mathbb{E}_{\boldsymbol{\eta}}f(\boldsymbol{\eta}) = \sum_{\boldsymbol{\eta}} f(\boldsymbol{\eta}) \mathbb{P}({\boldsymbol{\eta}})$.

Remark 2. We note that for θ = 0, equation (2.20) corresponds to the standard RS ansatz with the distribution of the two-replica overlap being delta-peaked at $\bar{q} = \bar{q}_2$. As θ increases, the 1-RSB Ansatz is gradually introduced. The optimal value of Parisi's parameter θ can in principle be obtained by minimizing the free energy ${\mathcal F}_N^{(P)}$ with respect to θ. However, the explicit form of the extremization condition of the free energy with respect to θ is known to be complicated and hardly used in practice, so that θ is often left as a free parameter (see e.g. [7]). In this work, we will fix θ by following a different procedure, namely by maximizing the region where the network accomplishes pattern retrieval, as done in [45] (see section 2.4 for a more detailed discussion).

Definition 5. Given the interpolating parameter $t \in [0,1]$, the constant $A_1,\ A_2, \ \psi \in \mathbb{R}$ to be fixed later on, and the i.i.d. standard Gaussian variables $Y_i^{(b)} \sim \mathcal{N}(0,1)$ for $i = 1, \ldots, N$ and $b = 1, 2$ (that must be averaged over as explained below), the Guerra's 1RSB interpolating partition function for the unsupervised DAM model is given by

$$\begin{align} \mathcal{Z}^{\left(P\right)}_2\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y}; J, t\right) &: = \sum_{\boldsymbol{\sigma}} \mathcal{B}_{N,2}^{\left(P\right)}\left(\boldsymbol \sigma|\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y}; J, t\right) \end{align} \tag{ 2.23 }$$

$$\begin{align} & = \sum_{\boldsymbol{\sigma}} \exp\left[J \sum_{i = 1}^N \xi_i^1 \sigma_i+\dfrac{t \beta ^{^{\prime}}N}{2M}\left(1+\rho \right)^{P/2}\displaystyle\sum\limits_{a = 1}^{M}n_{1,a}^{^P}\left(\boldsymbol{\sigma}\right) +\psi\left(1-t\right)N\displaystyle\sum\limits_{a = 1}^{M}n_{1,a}\left(\boldsymbol{\sigma}\right)\right. \nonumber\\ &\left.\quad+\sqrt{1-t}\displaystyle\sum\limits_{b = 1}^{2}\left(A_b\displaystyle\sum\limits_{i = 1}^{N}Y_i^{\left(b\right)}\sigma_i\right)+\sqrt{t}\dfrac{\beta ^{^{\prime}}P!\alpha\sqrt{\left(1+\rho_P\right)}}{2\left(1+\rho\right)^{P/2} \sqrt{K}}\displaystyle\sum\limits_{{i_1 \lt\cdots \lt i_P}}^{N,\cdots,N}\lambda_{i_1,\ldots,i_P}\sigma_{_{i_1}}\cdots\sigma_{_{i_{_{P}}}}\right]. \end{align} \tag{ 2.24 }$$

The index 2 on the LHS stands for the number of vectors ${\boldsymbol{Y}}^{(b)}$ that must be averaged over. Their number is equal to $k\!+\!1$ where k is the number of steps of RSB (here k = 1).

The average over the generalised Boltzmann factor associated to the interpolating partition function $\mathcal{Z}^{(P)}_2(\boldsymbol{\eta},\boldsymbol{\eta}, \boldsymbol{Y}; J, t)$, can be defined as

$$\begin{align} \omega_{t} \left(\cdot\right) : = \frac{1}{\mathcal Z^{\left(P\right)}_2\left( \boldsymbol{\eta}^1, \boldsymbol{\lambda}, \boldsymbol{Y};J, t \right)} \, \sum_{\boldsymbol \sigma}~ \cdot ~ \mathcal{B}_{N,2}^{\left(P\right)} \left(\boldsymbol{\sigma} |\boldsymbol{\eta}^1, \boldsymbol{\lambda}, \boldsymbol{Y};J, t \right). \end{align} \tag{ 2.25 }$$

Remark 3. We note that for t = 1, (2.24) recovers the original partition function (2.15), whereas for t = 0 it reduces to the partition function of a system of N non-interacting neurons, described by the Hamiltonian $-\beta^{^{\prime}} \mathcal{H}_{N}^{(1)}(\boldsymbol{\sigma} \vert \boldsymbol{\eta}, {\mathbf{Y}} ; J) = \sum_i h_i \sigma_i$, with local fields $h_i = J\xi_i^1+\psi r\mathcal{R}^{-1}\sum_{a = 1}^M \eta_i^{1,a}+\sum_{b = 1}^2 A_bJ_i^{(b)}$, that is readily evaluated. The parameters of the one-body model ψ, A₁ and A₂ must be chosen in such a way that the t-dependent terms in $d\mathcal{F}^{(P)}/dt$ cancel out in the thermodynamic limit, under the 1RSB assumption.

In what follows, we will average over the fields ${\boldsymbol{Y}}^{(b)}$ for $b = 1,2$ recursively, as explained by Guerra in [43] and in the statements below. To this purpose, we define

$$\begin{align} \mathcal Z_1^{\left(P\right)}\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda},\boldsymbol{Y}^{\left(1\right)}; J, t\right) &: = \left[\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \mathcal Z_2^{\left(P\right)}\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda},\boldsymbol{Y}; J, t\right) ^\theta \right]^{1/\theta} \end{align} \tag{ 2.26 }$$

$$\begin{align} \mathcal Z_0^{\left(P\right)}\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}; J, t\right) &: = \exp \left\{\mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}} \left[ \ln \mathcal Z_1^{\left(P\right)}\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y}^{\left(1\right)}; J, t\right) \right]\right\} \end{align} \tag{ 2.27 }$$

$$\begin{align} \mathcal{Z}^{\left(P\right)}_{N} \left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}; J, t\right) &: = \mathcal Z_0^{\left(P\right)}\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}; J, t\right), \end{align} \tag{ 2.28 }$$

where with $\mathbb{E}_{\boldsymbol{Y}^{(b)}}[.]$ we mean the average over the vectors ${\boldsymbol{Y}}^{(b)}$ for $b = 1, 2$. The interpolating quenched free energy related to the partition function (2.24) is introduced as

$$\begin{align} -\beta^{^{\prime}}\mathcal{F}^{\left(P\right)}_{N}\left(J, t\right) : = \frac{1}{N} \mathbb{E}_0 \left[ \ln \mathcal{Z}^{\left(P\right)}_{N}\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}; J, t\right) \right], \end{align} \tag{ 2.29 }$$

where $\mathbb{E}_0[.]$ denotes the average over the variables $\lambda^{\mu}_{i_1,\ldots,i_P}$'s and $\eta_{i}^{1,a}$'s. In the thermodynamic limit, assuming the limit exists, we write

$$\begin{align} \mathcal{F}^{\left(P\right)}\left(J, t\right) : = \lim_{N \to \infty} \mathcal{F}^{\left(P\right)}_{N}\left(J,t\right). \end{align} \tag{ 2.30 }$$

Remark 4. We note that in the RS case, where $\mathcal Z_2^{(P)}(\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y}; J, t) \equiv \mathcal Z_1^{(P)}(\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y}^{(1)}; J, t)$ and $\boldsymbol{Y}^{(2)} = {0}$, we have

$$\begin{align} -\beta^{^{\prime}} \mathcal{F}^{\left(P\right)}\left(J, t\right) : = \lim_{N \to \infty} \frac{1}{N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}}\left[ \ln \mathcal{Z}^{\left(P\right)}_{1}\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y}^{\left(1\right)}; J, t\right) \right], \end{align} \tag{ 2.31 }$$

hence the average $\mathbb{E}_{\boldsymbol{Y}^{(1)}}$ acts on the same level as $\mathbb{E}_0$ i.e. outside the logarithm. When moving from the RS to the 1RSB scenario, an extra average is introduced, $\mathbb{E}_{\boldsymbol{Y}^{(2)}}$, which acts on a different level than $\mathbb{E}_0$ and $\mathbb{E}_{\boldsymbol{Y}^{(1)}}$ i.e. inside the logarithm. This reflects the presence of hierarchical valleys in the free energy landscape [46] which leads to the well-known multi-scale thermalization in spin glasses. The internal average $\mathbb{E}_{Y^{(2)}}$ is related to thermalization within a valley at the lower level of the (two-level) hierarchy, whereas the external averages account for thermalization across valleys (i.e. at the higher level of the hierarchy). The structure of the hierarchy is captured by the parameter θ, which controls the amplitudes of the valleys at the lower level of the hierarchy and can be related to effective temperatures [47], as discussed in [48] for Parisi's replica approach and in [49] for Guerra's interpolation techniques. Note that the distinction between the two levels of the hierarchy is not made for ψ which accounts for the signal contribution and is kept RS (as usually done for the magnetization in spin-glass models).

Now, following Guerra's prescription [43], given two copies (or replicas) of the system, we define the following averages, corresponding to thermalization within the two different levels of the hierarchy

$$\begin{align} \langle \cdot \rangle_{t,1} &: = \mathbb{E}_0\mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}}\left[\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \mathcal{W}_{2,t} \omega_{t}\left( \cdot\right)\right]^2, \end{align} \tag{ 2.32 }$$

$$\begin{align} \langle \cdot \rangle_{t,2} &: = \mathbb{E}_0\mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}}\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \mathcal{W}_{2,t} \left[\omega_{t}\left(\cdot\right) \right]^2, \end{align} \tag{ 2.33 }$$

where

$$\begin{align} \mathcal{W}_{2,t} : = \dfrac{\mathcal{Z}_{2}^\theta\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y};J,t\right)}{\mathbb{E}_{Y^{\left(2\right)}} \mathcal{Z}^\theta_{2}\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y};J,t\right)}.\end{align} \tag{ 2.34 }$$

At this point, we are able to state our second assumption, that is at each level $a = 1,2$ of the hierarchy, the two-replica overlap $q_{12}(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)})$ self-averages around the value $\bar{q}_a$ of the corresponding peak in the overlap distribution $\mathbb{P}_N(q)$, as given in assumption 1:

Assumption 2. For any $t\in[0;1]$

$$\begin{align} \left\langle \left[q_{12}\left(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}\right) -\bar{q}_a\right]^k\right\rangle_{t,a} \xrightarrow[]{N\to\infty}0\;\;\; \mathrm{for}\; a = 1,2\;\;\mathrm{and}\;k\unicode{x2A7E} 2 \nonumber \end{align}$$

Finally, we provide the explicit expression of the quenched free energy in terms of the control parameters in the next

Proposition 1. In the thermodynamic limit $N\to\infty$, within the 1RSB Assumption and under the assumption 2, the quenched free energy for the unsupervised DAM model, reads as

$$\begin{align} -\beta^{^{\prime}}\mathcal{F}^{\left(P\right)} \left(J = 0\right)& = \dfrac{1}{\theta}\mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)}\mathbb{E}_{Y^{\left(1\right)}}\ln\mathbb{E}_{Y^{\left(2\right)}} \left\{\vphantom{\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P} } \bar{q}_1^{^{P-1}}}}\cosh^\theta\left[\vphantom{\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P} } \left(\bar{q}_2^{P-1} - \bar{q}_1 ^{^{P-1}}\right)}} \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M} \right. \right.\\ & \left.\left.\quad +\ Y^{\left(1\right)} \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P} } \bar{q}_1^{^{P-1}}} + Y^{\left(2\right)} \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P} } \left(\bar{q}_2^{P-1} - \bar{q}_1 ^{^{P-1}}\right)}\right]\right\} \\ & \quad\ -\beta^{^{\prime}} \dfrac{1}{2}\left(P-1\right)\left(1+\rho\right)^{P/2}\bar{n}^P + \dfrac{{\beta^{^{\prime}}}^2}{4}\gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2 }{\left(1+\rho\right)^{P}}\left[1-P\bar{q}_2^{P-1}\right.\nonumber\\ & \quad\left. +\ \left(P-1\right)\bar{q}_2^P\right] -\dfrac{{\beta^{^{\prime}}}^2}{4}\gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\left(P-1\right)\theta\left(\bar{q}_2^P-\bar{q}_1^P\right)+\ln 2. \end{align} \tag{ 2.35 }$$

where

$$\begin{align} \hat{\eta}_{\tiny M}: = \dfrac{1}{rM}\sum_{a = 1}^M\eta^{1,a}, \end{align} \tag{ 2.36 }$$

$\mathbb{E}_{(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1)} = \prod\nolimits_{a = 1}^M \mathbb{E}_{(\eta^{1,a}|\xi^1)}$ and $\mathbb{E}_{\boldsymbol{\xi}^1}$ and $\mathbb{E}_{(\eta^{1,a}|\xi^1)}$ denote the expectations over the Bernoulli distributions (2.1) and (2.2) respectively. In the above, $\bar n$, $\bar{q}_1, \ \bar{q}_2$ fulfill the following self-consistency equations

$$\begin{align} \bar{n} &= \dfrac{1}{1+\rho}\mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)} \hat{\eta}_{\tiny M} \right], \nonumber\\ \bar{q}_1 &= \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)} \right]^2, \nonumber\\ \bar{q}_2 &= \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) \tanh^2 g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)} \right], \end{align} \tag{ 2.37 }$$

with

$$\begin{align} g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) & = \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M}+Y^{\left(1\right)} \beta ^{^{\prime}}\sqrt{\gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \dfrac{P}{2}\bar{q}_1^{^{P-1}}} \nonumber \\ & \quad +Y^{\left(2\right)} \beta ^{^{\prime}}\sqrt{ \gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\dfrac{P}{2} \left(\bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)} \end{align} \tag{ 2.38 }$$

and $\beta^{^{\prime}} = 2\beta/P!$. Furthermore, as $\bar{m} = -\beta^{^{\prime}}\nabla_J \mathcal{F}^{(P)}(J)|_{J = 0}$, we have

$$\begin{align} \bar{m} = \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)}\mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}\xi^1\right]. \end{align} \tag{ 2.39 }$$

In order to prove the aforementioned proposition, we need to show the following

Lemma 1. In the thermodynamic limit, the t derivative of the interpolating quenched free energy (2.29) under the 1RSB assumption and under assumption 2 is given by

$$\begin{align} \dfrac{d \mathcal{F}^{\left(P\right)}\left(J,t\right)}{d t} &= \dfrac{1}{2}\left(P-1\right)\left(1+\rho\right)^{P/2}\bar{n}^P - \dfrac{\beta^{^{\prime}}}{4}\gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\left[1-P\bar{q}_2^{P-1}+\left(P-1\right)\bar{q}_2^P\right] \nonumber\\ &\quad+\dfrac{\beta^{^{\prime}}}{4}\gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\left(P-1\right)\theta\left(\bar{q}_2^P-\bar{q}_1^P\right). \end{align} \tag{ 2.40 }$$

The proof of this lemma is shown in appendix B.1. Now let us prove proposition 1:

Proof. Exploiting the fundamental theorem of calculus, we can relate the free energy of the original model $\mathcal{F}^{(P)}(J,t = 1)$ and the one stemming from the one body terms $\mathcal{F}^{(P)}(J,t = 0)$ via (2.19). Since, the t-derivative (2.40) does not depend on t, all we need is to add the expression above to the free energy $\mathcal{F}^{(P)}(J,t = 0)$, which only contains one body terms. Let us start with the computation of the latter at finite size N:

$$\begin{align} -\beta^{^{\prime}}\mathcal{F}_{N}^{\left(P\right)}\left(J,t = 0\right) &= \dfrac{1}{\theta N} \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_{Y^{\left(1\right)}} \ln \mathbb{E}_{Y^{\left(2\right)}} \sum_{\boldsymbol{\sigma}} \exp \left[-\beta \mathcal{H}_{N}^{\left(1\right)}\left(\boldsymbol{\sigma} \vert \boldsymbol{\eta}^1, {\mathbf{Y}}; J\right)\right] \nonumber \\ &= \ln 2 + \dfrac{1}{\theta} \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_{Y^{\left(1\right)}} \ln \mathbb{E}_{Y^{\left(2\right)}} \cosh^{\theta}\nonumber\\ &\quad\times \left(J\xi^1 + \dfrac{\psi}{\left(1+\rho\right)} \hat\eta_{M} + Y^{\left(1\right)} A_1 + Y^{\left(2\right)}A_2\right), \end{align} \tag{ 2.41 }$$

with the definition of $-\beta \mathcal{H}_{N}^{(1)}(\boldsymbol{\sigma} \vert \boldsymbol{\eta}; J)$ given in remark 3.

Upon setting the constants to the values (B.8) stated in the proof of lemma 1, we have

$$\begin{align} -\beta^{^{\prime}}\mathcal{F}_{N}^{\left(P\right)}\left(J,t = 0\right) &= \ln 2 + \dfrac{1}{\theta} \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_{Y^{\left(1\right)}} \ln \mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta^{^{\prime}}, K, N, \boldsymbol{Y}\right), \end{align} \tag{ 2.42 }$$

where $g(\beta, K,N, \boldsymbol{Y})$ reads as

$$\begin{align} g\left(\beta, K, N, \boldsymbol{Y}\right) &= J \xi^1+\beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M}\nonumber \\ &\quad+Y^{\left(1\right)} \beta ^{^{\prime}}\sqrt{\dfrac{P\sqrt{\left(1+\rho_P\right)}^2 }{2\left(1+\rho\right)^{P}} \dfrac{P!K}{2N^{P-1}} \bar{q}_1^{^{P-1}}}\nonumber \\ &\quad +Y^{\left(2\right)} \beta ^{^{\prime}}\sqrt{ \dfrac{P\sqrt{\left(1+\rho_P\right)}^2 }{2\left(1+\rho\right)^{P}} \dfrac{P!K}{2N^{P-1}}\left(\bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)}.\end{align} \tag{ 2.43 }$$

Recalling that $\lim\limits_{N\to \infty}K/N^{P-1} = 2\gamma/P!$, taking the thermodynamic limit of the above equations and inserting them in the fundamental theorem of calculus (2.19), we reach (2.35). Finally, by maximizing (2.35) w.r.t. to the order parameters $\bar{n}, \bar{q}_1, \bar{q}_2$, we obtain the self-consistency equations (2.37), hence we reach the thesis. □

In this section, we derive the expression of the quenched free energy of the unsupervised DAM model provided in proposition 1 by using the replica method [2, 50], at the first step of RSB [2, 51]. The core of this approach consists in writing the quenched free energy, as defined in (2.6), as

$$\begin{align} -\beta^{^{\prime}} \mathcal{F}_N^{\left(P\right)}\left(J\right) = \dfrac{1}{N} \lim_{n\to 0} \dfrac{\mathbb{E}\mathcal{Z}_N^n\left(J\right)-1}{n}, \end{align} \tag{ 2.44 }$$

with $\beta^{^{\prime}} = 2 \beta/P!$ and where we have used the shorthand $\mathcal{Z}_N(J) = \mathcal{Z}_N^{(P)}(\boldsymbol{\eta};J)$ for the partition function, defined as

$$\begin{align} \mathcal{Z}_N\left(J\right) & = \sum_{\boldsymbol{\sigma}} \exp\left( -\beta^{^{\prime}} \mathcal{H}^{\left(P\right)}_N\left(\boldsymbol{\sigma}| \boldsymbol{\eta}\right) + J \displaystyle\sum\limits_{i = 1}^{N} \xi_i^1\sigma_i\right)\,, \end{align} \tag{ 2.45 }$$

$\mathbb{E}[.]$ denotes, as in (2.6), the average over the quenched disorder η and in accordance with the explanation provided in section 2.1, the inclusion of the last term in the round brackets of (2.45) is intended to make the free energy a moment-generating function for the Mattis magnetization $m_1(\boldsymbol{\sigma})$.

For integer n the function $\mathcal{Z}_N^n$ is the product of a system of n identical replicas of the original system

$$\begin{align} \mathbb{E}\mathcal{Z}_N^n\left(J\right) & = \mathbb{E} \sum_{\boldsymbol{\sigma}^1 \ldots \boldsymbol{\sigma}^n} \exp\left( -\beta^{^{\prime}} \sum_{a = 1}^N \mathcal{H}^{\left(P\right)}_N\left(\boldsymbol{\sigma}^{\left(a\right)}| \boldsymbol{\eta}\right) + J \displaystyle\sum\limits_{a = 1}^{n}\displaystyle\sum\limits_{i = 1}^{N} \xi_i^1\sigma_i^{\left(a\right)}\right) \end{align} \tag{ 2.46 }$$

namely

$$\begin{align} \mathbb{E}\mathcal{Z}_N^n\left(J\right) & = \mathbb{E} \sum_{\boldsymbol{\sigma}^1 \ldots \boldsymbol{\sigma}^n} \exp\left[ \dfrac{\beta^{^{\prime}} P!}{2\mathcal{R}^{P/2}\,M\,N^{P-1}}\displaystyle\sum\limits_{a = 1}^{n}\displaystyle\sum\limits_{\mu = 1}^{K}\displaystyle\sum\limits_{A = 1}^{M}\left(\displaystyle\sum\limits_{i_1 \lt\cdots \lt i_P}^{N,\ldots,N}\eta^{\mu\,,A}_i\cdots\eta^{\mu\,,A}_{i_P}\sigma^{\left(a\right)}_{_{i_1}}\cdots\sigma^{\left(a\right)}_{_{i_P}}\right)\right.\nonumber\\ & \quad\left. +\ J \displaystyle\sum\limits_{a, i}^{n,N}\xi_i^1\sigma_i^{\left(a\right)} \right]\,. \end{align} \tag{ 2.47 }$$

Splitting, as before, the signal term (µ = 1) from the remaining terms µ > 1, which act as a quenched noise on the learning and retrieval of pattern $\boldsymbol{\xi}^1$, we can write

$$\begin{align} \mathbb{E}\mathcal{Z}_N^n\left(J\right) = & \sum_{\boldsymbol{\sigma}^1\ldots \boldsymbol{\sigma}^n} \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^1\right)} \exp\left[ \dfrac{\beta^{^{\prime}} \left(1+\rho\right)^{P/2} }{2 MN} \sum_{a = 1}^n\sum_{A = 1}^M\left( \dfrac{r}{N \mathcal{R}}\sum_i \eta^{1\,,A}_i \sigma_i^{\left(a\right)} \right)^P + J \displaystyle\sum\limits_{a, i}^{n,N}\xi_i^1\sigma_i^{\left(a\right)}\right] \nonumber \nonumber\\ &\!\!\!\!\!\!\quad\cdot \mathbb{E}_{\boldsymbol{\xi}^{\mu>1}}\mathbb{E}_{\left(\boldsymbol{\eta}^{\mu>1}|\boldsymbol{\xi}^{\mu>1}\right)} \exp \left[\dfrac{\beta^{^{\prime}} P!}{2 M\mathcal{R}^{P/2}N^{P-1}} \sum_{\mu>1, a,A}\sum_{i_1 \lt\cdots \lt i_P} \eta^{\mu\,,A}_{i_1}\ldots \eta^{\mu\,,A}_{i_{P}} \sigma_{i_1}^{\left(a\right)}\ldots \sigma_{i_{P}}^{\left(a\right)}\right]. \end{align} \tag{ 2.48 }$$

where $\mathbb{E}_{\boldsymbol{\xi}^\mu}[.] = \prod_i \mathbb{E}_{\xi_i^\mu}[.]$ denotes the average over the pattern $\boldsymbol{\xi}^\mu$, whose entries $\xi_i^\mu$ are drawn from the distribution (2.1) and $\mathbb{E}_{(\boldsymbol{\eta}^\mu|\boldsymbol{\xi}^\mu)} = \prod_{i,a}\mathbb{E}_{(\eta_i^{\mu,a}|\xi_i^\mu)}$ with $\mathbb{E}_{\eta_i^{\mu,A}}$ denoting the expectation over the conditional distribution (2.2). As before, exploiting the CLT to rewrite the last term in (2.48) via (2.14), we obtain

$$\begin{align} \mathbb{E}\mathcal{Z}_N^n\left(J\right) = & \sum_{\boldsymbol{\sigma}^1\ldots \boldsymbol{\sigma}^n} \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^1\right)} \exp\left[ \dfrac{\beta^{^{\prime}} \left(1+\rho\right)^{P/2}N }{2 M} \sum_{a = 1}^n\sum_{A = 1}^M\left( \dfrac{r}{N \mathcal{R}}\sum_i \eta^{1\,,A}_i \sigma_i^{\left(a\right)} \right)^P + J \displaystyle\sum\limits_{a, i}^{n,N}\xi_i^1\sigma_i^{\left(a\right)}\right] \nonumber \\ & \!\!\!\!\!\!\quad \cdot \mathbb{E}_{\boldsymbol{\lambda}} \exp \left[\dfrac{\beta^{^{\prime}} P! \alpha\sqrt{\left(1+\rho_P\right)}}{2 \left(1+\rho\right)^{P/2}\sqrt{K}} \sum_{a}\sum_{i_1 \lt \cdots \lt i_P} \lambda_{i_1,\ldots,i_{P}} \sigma_{i_1}^{\left(a\right)}\ldots \sigma_{i_{P}}^{\left(a\right)}\right]\nonumber \\ = & \sum_{\boldsymbol{\sigma}^1\ldots \boldsymbol{\sigma}^n} \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^1\right)} \mathcal{Z}_\textrm{signal}^n\left(J\right) \mathbb{E}_{\boldsymbol{\lambda}} \mathcal{Z}_\textrm{noise}^n, \end{align} \tag{ 2.49 }$$

Treating the two terms $\mathbb{E}_{\boldsymbol{\xi}^{1}}\mathbb{E}_{(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^{\tiny \mbox{1}})} \mathcal{Z}_\textrm{signal}^n(J)$ and $\mathbb{E}_{\boldsymbol{\lambda}} \mathcal{Z}_\textrm{noise}^n$ separately, we insert in the former the identity

$$\begin{align*} 1 = \prod\limits_{a,A = 1,1}^{n,M}\int \textrm{d} n_{1,A}^{\left(a\right)}\, \delta\left(n_{1,A}^{\left(a\right)} - n_{1,A}^{\left(a\right)}\left(\boldsymbol{\sigma}\right)\right) \end{align*}$$

and use the Fourier representation of the Dirac delta, obtaining

$$\begin{align} \mathbb{E}_{\boldsymbol{\xi}^{1}}\mathbb{E}_{\left(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^{1}\right)} \mathcal{Z}_\textrm{signal}^n\left(J\right) & = \mathbb{E}_{\boldsymbol{\xi}^{1}}\mathbb{E}_{\left(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^{1}\right)} \int \prod_{a,A} \textrm{d} n_{1,A}^{\left(a\right)} \prod_{a,A} \frac{\textrm{d}z_{1,A}^{\left(a\right)}}{2\pi} \exp \left[ \dfrac{\beta^{^{\prime}} N \left(1+\rho\right)^{P/2}}{2M} \sum_{A,a} \left(n_{1,A}^{\left(a\right)}\right)^P \right. \nonumber \\ & \quad \left.+\ i \sum_{a,A} \left(n_{1,A}^{\left(a\right)} - \dfrac{r}{\mathcal{R} N} \sum_{i} \eta_i^{1,A} \sigma_i^{\left(a\right)} \right)z_{1,A}^{\left(a\right)} + J \displaystyle\sum\limits_{a, i}^{n,N}\xi_i^1\sigma_i^{\left(a\right)}\right]. \end{align} \tag{ 2.50 }$$

For the noise term, performing first the expectation over λ

$$\begin{align} \mathbb{E}_{\boldsymbol{\lambda}} \mathcal{Z}_\textrm{noise}^n & = \int \prod_{\boldsymbol{i}} \dfrac{d\lambda_{i_1,\ldots,i_P}}{\sqrt{2\pi}} \exp \left[-\dfrac{\left(\lambda_{i_1,\ldots,i_P}\right)^2}{2}+\dfrac{\beta^{^{\prime}} \alpha\sqrt{\left(1+\rho_P\right)}}{2 \left(1+\rho\right)^{P/2}\sqrt{K}} P!\displaystyle\sum\limits_{a}^{n}\displaystyle\sum\limits_{i_1\lt \ldots\lt i_P}^{N,\ldots,N}\lambda_{i_1,\ldots,i_P}\sigma_{i_1}^{\left(a\right)}\cdots\sigma_{i_P}^{\left(a\right)} \right] \nonumber \\ & = \exp \left[\dfrac{1}{2}\left(\dfrac{\beta^{^{\prime}} \sqrt{\left(1+\rho_P\right)}}{2 \left(1+\rho\right)^{P/2}N^{P-1}}\right)^2 K\left(P!\right)^2 \displaystyle\sum\limits_{i_1\lt \ldots\lt i_P}^{N,\ldots,N}\displaystyle\sum\limits_{a,b}^{n,n}\sigma_{i_1}^{\left(a\right)}\sigma_{i_1}^{\left(b\right)}\cdots\sigma_{i_P}^{\left(a\right)}\sigma_{i_P}^{\left(b\right)} \right] \nonumber \\ & = \exp \left[\dfrac{1}{2}\left(\dfrac{\beta^{^{\prime}} \sqrt{\left(1+\rho_P\right)}}{2 \left(1+\rho\right)^{P/2}N^{P/2-1}}\right)^2 K P! \displaystyle\sum\limits_{a,b}^{n,n}\left(\dfrac{1}{N}\displaystyle\sum\limits_{i}^{N}\sigma_{i_1}^{\left(a\right)}\sigma_{i_1}^{\left(b\right)}\right)^P \:\right] \end{align} \tag{ 2.51 }$$

inserting

$$\begin{align*} 1 = \prod_{a,b}\int \textrm{d}q_{ab}\, \delta\left(q_{ab}-q_{ab}\left(\boldsymbol{\sigma}\right)\right) \end{align*}$$

and using the Fourier representation of the Dirac delta, we have

$$\begin{align} \mathbb{E}_{\boldsymbol{\lambda}} \mathcal{Z}_\textrm{noise}^n & = \int \prod_{a,b} \textrm{d}q_{ab} \int \prod_{a,b} \dfrac{\textrm{d}p_{ab}}{2\pi} \exp \left[\dfrac{1}{2}\left(\dfrac{\beta^{^{\prime}} \sqrt{\left(1+\rho_P\right)}^2}{2 \left(1+\rho\right)^{P/2}N^{P/2-1}} \right)^2 K P!\displaystyle\sum\limits_{a,b}^{n,n}q_{ab}^P \right. \nonumber \\ & \quad \left. +\ i \sum_{a,b} \left( q_{ab} - \dfrac{1}{N}\sum_{i}\sigma_i^{\left(a\right)}\sigma_i^{\left(b\right)} \right)p_{ab} \right]\end{align} \tag{ 2.52 }$$

Inserting equations (2.50) and (2.52) in (2.49), we obtain

$$\begin{align} &\mathbb{E}\mathcal{Z}_N^n\left(J\right) = \int \prod_{a,A} \textrm{d} n_{1,A}^{\left(a\right)} \prod_{a,A} \dfrac{\textrm{d}z_{1,A}^{\left(a\right)}}{2\pi} \int \prod_{a,b} \textrm{d}q_{ab} \int \prod_{a,b} \dfrac{\textrm{d}p_{a,b}}{2\pi} \exp\left( -N A\left[\boldsymbol{Q}, \boldsymbol{P}, \boldsymbol{Z}, \boldsymbol{N}\right] \right) \end{align} \tag{ 2.53 }$$

where

$$\begin{align} A\left[\boldsymbol{Q}, \boldsymbol{P}, \boldsymbol{Z}, \boldsymbol{N}\right]& = - \dfrac{i}{N} \sum_{a,b} p_{ab}q_{ab} -\dfrac{1}{4}\dfrac{\beta^{^{\prime}}\,^2 \sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \gamma \displaystyle\sum\limits_{a,b}^{n,n}q_{ab}^P - \dfrac{\beta^{^{\prime}} \left(1+\rho\right)^{P/2}}{2M} \sum_{A,a} \left(n_{1,A}^{\left(a\right)}\right)^P \nonumber \\ & \quad -\dfrac{i}{N}\sum_{a,A} n_{1,A}^{\left(a\right)} z_{1,A}^{\left(a\right)} -\mathbb{E}_{\boldsymbol{\xi}^{1}}\mathbb{E}_{\left(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^{1}\right)} \log \left[\sum_{\boldsymbol{\sigma}} \exp\left( -\dfrac{i}{N} \sum_{a,b,i} p_{ab} \sigma^{\left(a\right)}\sigma^{\left(b\right)}\right.\right.\nonumber\\&\quad \left.\left. - \dfrac{i r}{\mathcal{R} N} \sum_{a,A} \eta^{1,A} z_{1,A}^{\left(a\right)} \sigma^{\left(a\right)}+ J \displaystyle\sum\limits_{a}^{}\xi^1\sigma^{\left(a\right)}\right)\right]. \end{align} \tag{ 2.54 }$$

and we have used the shorthand notation ${\boldsymbol{Q}} = \{q_{ab}\}_{a,b = 1}^n$ and similarly for P , N and Z . Next, we assume that the limit n → 0 in

$$\begin{align} -\beta^{^{\prime}} \mathcal{F}^{\left(P\right)}\left(\beta\right) = \lim_{N\to\infty} \dfrac{1}{N} \lim_{n\to 0} \dfrac{\mathbb{E}\mathcal{Z}_N^n-1}{n}, \end{align} \tag{ 2.55 }$$

can be taken by analytic continuation and that the two limits $N\to\infty$ and n → 0 can be interchanged (provided they exist), so that the integrals in (2.53) can be performed by steepest descent. This leads to

$$\begin{align} -\beta^{^{\prime}} \mathcal{F}^{\left(P\right)}\left(\beta\right) = - \lim_{n\to 0} \dfrac{1}{n} A\left[\boldsymbol{Q}^\star, \boldsymbol{P}^\star, \boldsymbol{Z}^\star, \boldsymbol{N} ^\star\right] \end{align} \tag{ 2.56 }$$

where $\boldsymbol{Q}^\star, \boldsymbol{P}^\star, \boldsymbol{Z}^\star, \boldsymbol{N} ^\star$, are solution of the saddle point equations

$$\begin{eqnarray} &\dfrac{\partial A}{\partial p_{ab}} = 0&\quad\Rightarrow\quad q_{ab}^* = \langle \sigma^a \sigma^b\rangle_{\mathrm{eff}} \end{eqnarray} \tag{ 2.57 }$$

$$\begin{eqnarray} &\dfrac{\partial A}{\partial z_{1,A}^{\left(a\right)}} = 0&\quad\Rightarrow\quad \left(n_{1,A}^{\left(a\right)}\right)^* = \dfrac{r}{\mathcal{R}}\displaystyle\sum\limits_{A = 1}^{M}\mathbb{E}\langle \eta^{1,A}\sigma^a \rangle_{\mathrm{eff}} \end{eqnarray} \tag{ 2.58 }$$

$$\begin{eqnarray} &\dfrac{\partial A}{\partial q_{ab}} = 0&\quad\Rightarrow\quad p_{ab}^* = \dfrac{i \beta^{^{\prime}}\,^2}{2} \dfrac{P}{2}\gamma \dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)}N \left(q_{ab}^*\right)^{P-1} \end{eqnarray} \tag{ 2.59 }$$

$$\begin{eqnarray} &\dfrac{\partial A}{\partial n_{1,A}^{\left(a\right)}} = 0&\quad\Rightarrow\quad \left(z_{1,A}^{\left(a\right)}\right)^* = i \beta^{^{\prime}} \dfrac{P}{2} \dfrac{\left(1+\rho\right)^{P/2}}{M} N \left[\left(n_{1,A}^{\left(a\right)}\right)^*\right]^{P-1} \end{eqnarray} \tag{ 2.60 }$$

where the average $\langle \ldots \rangle_{\mathrm{eff}}$ is computed over the distribution $p_{\mathrm{eff}}(\boldsymbol{\sigma}) = e^{-\beta^{^{\prime}} H_{\mathrm{eff}}(\boldsymbol{\sigma})}/Z_{\mathrm{eff}}$ where $H_{\mathrm{eff}}(\boldsymbol{\sigma})$ is equal to the terms in the round brackets in the second line of (2.54). Inserting (2.59) and (2.60) in $H_{\mathrm{eff}}(\boldsymbol{\sigma})$, we obtain

$$\begin{align} -\beta^{^{\prime}} H_{\mathrm{eff}}\left(\boldsymbol{\sigma}\right)& = \dfrac{{\beta^{^{\prime}}}^2\gamma}{2}\frac{P\sqrt{\left(1+\rho_P\right)}^2}{2\left(1+\rho\right)^P} \sum_{a,b} \left[q_{ab}^*\right]^{P-1} \sigma^{\left(a\right)}\sigma^{\left(b\right)}\nonumber\\& \quad + \beta^{^{\prime}} \dfrac{P}{2} \left(1+\rho\right)^{P/2-1} \sum_{a}\dfrac{1}{rM}\sum_{A} \eta^{1,A} \left[\left(n_{1,A}^{\left(a\right)}\right)^*\right]^{P-1} \sigma^{\left(a\right)} \end{align} \tag{ 2.61 }$$

and, substituting in (2.56)

$$\begin{align} -\beta^{^{\prime}} \mathcal F^{\left(P\right)}\left(J\right) = &\lim_{n\to 0} \dfrac{1}{n} \left\{\dfrac{{\beta^{^{\prime}}}^2\sqrt{\left(1+\rho_P\right)}^2\gamma}{4\left(1+\rho\right)^P} \sum_{a,b} \left(P-1\right) \left[q_{ab}^*\right]^{P} +\frac{\beta^{^{\prime}} \left(1+\rho\right)^{P/2} \left(P-1\right)}{2M}\sum_{a,A} \left[\left(n_{1,A}^{\left(a\right)}\right)^*\right]^{P}\right.\nonumber \\ & +\left.\mathbb{E}_{\boldsymbol{\xi}^{1}}\mathbb{E}_{\left(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^{1}\right)} \log \left[ \sum_{\boldsymbol{\sigma}} \exp\left(-\beta^{^{\prime}} H_{\mathrm{eff}}\left(\boldsymbol{\sigma}\right)+ J \displaystyle\sum\limits_{a}^{}\xi^1\sigma^{\left(a\right)}\right)\right]\right\}\,. \end{align} \tag{ 2.62 }$$

where $q_{ab}^\star$ and $[n_{1,A}^{(a)}]^*$ must be determined from (2.57) and (2.58). Since $p_{\mathrm{eff}}(\boldsymbol{\sigma})$ depends on Q , N via (2.61), equations (2.57) and (2.58) denote a set of self-consistency equations. Now, in order to proceed, we need to find the form of Q and N in the limit n → 0. We will make the 1RSB ansatz:

$$\begin{align} q_{ab}& = \delta_{ab} + \left(1-\delta_{ab}\right)\left(\theta \bar{q}_1 +\left(1-\theta\right) \bar{q}_2\right), \end{align} \tag{ 2.63 }$$

$$\begin{align} n_{1,A}^{\left(a\right)}& = \bar{n}, \end{align} \tag{ 2.64 }$$

Using (2.63) to evaluate the first term in (2.62) and the first term in (2.61), we get

$$\begin{align} \sum_{ab} q_{ab}^P & = n \left(1+n \theta\bar{q}_1^P -\theta\bar{q}_1^P - \bar{q}_2^P + n \bar{q}_2^P + \theta\bar{q}_2^P - n \theta\bar{q}_2^P \right) \end{align} \tag{ 2.65 }$$

$$\begin{align} \sum_{ab} q_{ab}^{P-1} \sigma^{\left(a\right)}\sigma^{\left(b\right)} & = \frac{1}{2} \left[ 2\left(1-\bar{q}_2^{P-1}\right) n + \left(\bar{q}_2^{P-1} - \bar{q}_1^{P-1}\right) \sum_{k = 1}^{n/\theta}\left( \sum_{a = 1}^\theta \sigma^{\left(\left(k-1\right)\theta+a\right)}\right)^2\right.\nonumber\\ & \quad \left.+\ \bar{q}_1^{P-1} \left( \sum_a \sigma^{\left(a\right)}\right)^2\right]. \end{align} \tag{ 2.66 }$$

Upon inserting in (2.62) we obtain

$$\begin{align} -\beta^{^{\prime}} \mathcal F^{\left(P\right)}\left(J\right) & = \dfrac{{\beta^{^{\prime}}}^2\sqrt{\left(1+\rho_P\right)}^2\gamma}{4\left(1+\rho\right)^P} \left[1 - \left(P-1\right)\theta \left(\bar{q}_2^P-\bar{q}_1^P\right) +\left(P-1\right)\bar{q}_2^P\right] \nonumber \\ &\quad -\frac{\beta^{^{\prime}} \left(1+\rho\right)^{P/2} \left(P-1\right)}{2}\bar{n}^{P} -\frac{\left(\beta^{^{\prime}}\right)^2\sqrt{\left(1+\rho_P\right)}^2\gamma P}{4\left(1+\rho\right)^P} \bar{q}_2^{P-1}\nonumber \\ &\quad +\lim_{n\to 0}\frac{1}{n}\mathbb{E}_{\boldsymbol{\xi}^{1}}\mathbb{E}_{\left(\boldsymbol{\eta}^{1}|\boldsymbol{\xi}^{1}\right)} \log \left[ \sum_{\boldsymbol{\sigma}} \exp\left(-\beta^{^{\prime}} H_{\mathrm{eff}}\left(\boldsymbol{\sigma}\right)+ J \displaystyle\sum\limits_{a}^{}\xi^1\sigma^{\left(a\right)}\right)\right]\, \end{align} \tag{ 2.67 }$$

with

$$\begin{eqnarray} -\beta^{^{\prime}} H_{\mathrm{eff}}\left(\boldsymbol{\sigma}\right)& = &{\beta^{^{\prime}}}^2\gamma\frac{P\sqrt{\left(1+\rho_P\right)}^2}{4\left(1+\rho\right)^P} \left[ \left(\bar{q}_2^{P-1} - \bar{q}_1^{P-1}\right) \sum_{k = 1}^{n/\theta}\left( \sum_{b = 1}^\theta \sigma^{\left(\left(k-1\right)\theta+b\right)}\right)^2+ \bar{q}_1^{P-1} \left( \sum_{a = 1}^n \sigma^{\left(a\right)}\right)^2\right] \nonumber\\ &&+ \beta^{^{\prime}} \dfrac{P}{2}\left(1+\rho\right)^{P/2-1} \hat{\eta}_M \left(\sum_{a = 1}^n\sigma^{\left(a\right)}\right) \bar{n}^{P-1} \end{eqnarray} \tag{ 2.68 }$$

where we have used the definition (2.36).

Next, we apply a Gaussian transformation to the last term in (2.67) to linearize the squared terms in the Hamiltonian (2.68)

$$\begin{align} & \lim_{n\to 0}\dfrac{1}{n}\ln \left[ \sum_{\boldsymbol{\sigma}} \exp\left(-\beta^{^{\prime}} H_{\mathrm{eff}}\left(\boldsymbol{\sigma}\right)+J \displaystyle\sum\limits_{a}^{}\xi^1\sigma^{\left(a\right)}\right)\right] \nonumber \\ & \quad = \lim_{n\to 0}\dfrac{1}{n}\ln \left\{ \int {\mathcal{D}} u\, \int \left[\prod_{k = 1}^{n/\theta}{\mathcal{D}} v_k\right] \sum_{\boldsymbol{\sigma}} \exp \left[\frac{\beta^{^{\prime}} P \left(1+\rho\right)^{P/2-1} }{2} \hat{\eta}_M \bar{n}^{P-1} \,\displaystyle\sum\limits_{a = 1}^{n}\sigma^{\left(a\right)}\right.\right. \nonumber \\ &\quad \quad\left.\left.+\ J \displaystyle\sum\limits_{a = 1}^{n}\xi^1\sigma^{\left(a\right)} \,+\beta^{^{\prime}} u\,\displaystyle\sum\limits_{a = 1}^{n}\sigma^{\left(a\right)}\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\bar{q}_1^{P-1}}\right.\right. \nonumber \\ &\quad \quad\left.\left.+\ \beta^{^{\prime}} \sum_{k = 1}^{n/\theta}v_k\displaystyle\sum\limits_{b = 1}^{\theta}\sigma^{\left(\left(k-1\right)\theta+b\right)}\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\left(\bar{q}_2^{P-1}-\bar{q}_1^{P-1}\right)} \,\right]\right\} \end{align} \tag{ 2.69 }$$

where we have set ${\mathcal{D}} u = \textrm{d}u \,e^{-u^2/2}/\sqrt{2\pi}$ and ${\mathcal{D}} v_k = \textrm{d}v_k \,e^{-v_k^2/2}/\sqrt{2\pi}$ for $k = 1, \ldots, n/\theta$. Now, writing

$$\begin{align*} \displaystyle\sum\limits_{a = 1}^{n}\sigma^{\left(a\right)} = \displaystyle\sum\limits_{k = 1}^{n/\theta}\displaystyle\sum\limits_{b = 1}^{\theta}\sigma^{\left(\left(k-1\right)\theta+b\right)}, \end{align*}$$

the sum over the spin configuration can be done explicitly, finding

$$\begin{align} & = \lim_{n\to 0}\dfrac{1}{n}\ln \left\{ \int {\mathcal{D}} u\, \prod_{k = 1}^{n/\theta}\int {\mathcal{D}} v_k\,\prod_{b = 1}^{\theta}\sum_{\sigma^{\left(\left(k-1\right)\theta+b\right)}} \exp \left( \frac{\beta^{^{\prime}} P \left(1+\rho\right)^{P/2-1} \hat{\eta}_M \bar{n}^{P-1}\sigma^{\left(\left(k-1\right)\theta+b\right)}}{2} \,\right.\right. \nonumber \\ &\quad + J \xi^1\sigma^{\left(\left(k-1\right)\theta+b\right)}+\beta^{^{\prime}} u\,\sigma^{\left(\left(k-1\right)\theta+b\right)}\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\bar{q}_1^{P-1}} \nonumber \\ &\quad \left.\left. +\ \beta^{^{\prime}} v_k\sigma^{\left(\left(k-1\right)\theta+b\right)}\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\left(\bar{q}_2^{P-1}-\bar{q}_1^{P-1}\right)} \,\right)\right\} \nonumber \\ & = \lim_{n\to 0}\dfrac{1}{n}\ln \left\{ \int {\mathcal{D}} u\, \prod_{k = 1}^{n/\theta}\left[\int{\mathcal{D}} v_k 2^\theta \cosh^\theta\left( \frac{\beta^{^{\prime}} P \left(1+\rho\right)^{P/2-1}}{2} \hat{\eta}_M \bar{n}^{P-1} +J \xi^1 \,\right.\right.\right. \nonumber \\ & \quad \left.\left.\left. +\ \beta^{^{\prime}} u\,\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\bar{q}_1^{P-1}}+\beta^{^{\prime}} v_k \sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\left(\bar{q}_2^{P-1}-\bar{q}_1^{P-1}\right)} \,\right)\right]\right\}. \nonumber \end{align}$$

Now applying the n → 0 limit [50] and exploiting the relation $\lim\limits_{n\to 0}\dfrac{1}{n} \ln\mathbb{E}_{Y^{(1)}}[f^n(Y^{(1)})] = \mathbb{E}_{Y^{(1)}}[\ln f(Y^{(1)})]$, we get

$$\begin{align} & = \lim_{n\to 0}\frac{1}{n}\ln \left\{ \mathbb{E}_u\left[\mathbb{E}_v \,2^\theta\cosh^\theta\left(\beta^{^{\prime}} u\,\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\bar{q}_1^{P-1}} \,\right.\right.\right. \nonumber \\ &\left.\left.\left.\quad +\ \beta^{^{\prime}} v\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\left(\bar{q}_2^{P-1}-\bar{q}_1^{P-1}\right)} \,+ \frac{\beta^{^{\prime}} P \left(1+\rho\right)^{P/2-1} }{2} \hat{\eta}_M \bar{n}^{P-1} +J\xi^1\,\right)\right]^{n/\theta}\right\} \nonumber \\ & = \frac{1}{\theta}\mathbb{E}_u\ln \left\{\mathbb{E}_v \,2^\theta\cosh^\theta\left[\beta^{^{\prime}}\left(u\,\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\bar{q}_1^{P-1}} +v\sqrt{\gamma \frac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \frac{P}{2}\left(\bar{q}_2^{P-1}-\bar{q}_1^{P-1}\right)} \right.\right.\right.\nonumber \\ &\,\left.\left.\left.\quad+\ \frac{P \left(1+\rho\right)^{P/2-1}\hat{\eta}_M }{2} \bar{n}^{P-1} +J\xi^1\, \right)\right]\right\}. \nonumber \end{align}$$

Finally, inserting this expression in (2.67) and denoting with $\mathbb{E}_{Y^{(1)}}[.]$ and $\mathbb{E}_{Y^{(2)}}[.]$ the Gaussian averages w.r.t. $Y^{(1)}$ and $Y^{(2)}$ respectively, we reach the following expression for the free energy

$$\begin{align} -\beta^{^{\prime}}\mathcal{F}^{\left(P\right)}\left(J\right) &= \ln 2+ \dfrac{1}{\theta}\mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)}\mathbb{E}_{Y^{\left(1\right)}}\ln\mathbb{E}_{Y^{\left(2\right)}} \left\{\cosh^\theta\left[ J\xi^1+\beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M} \right. \right. \nonumber\\ &\left. \left.\quad+\ Y^{\left(1\right)} \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \bar{q}_1^{^{P-1}}} +Y^{\left(2\right)} \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \left( \bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)}\right]\right\}\nonumber \\ &\quad-\ \frac{{\beta^{^{\prime}}}}{2}\left(P-1\right)\left(1+\rho\right)^{P/2}\bar{n}^P + \dfrac{{\beta^{^{\prime}}}^2}{4}\gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\left[1-P\bar{q}_2^{P-1}+\left(P-1\right)\bar{q}_2^P\right] \nonumber\\ &\quad-\ \frac{{\beta^{^{\prime}}}^2}{4}\gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\left(P-1\right)\theta\left(\bar{q}_2^P-\bar{q}_1^P\right), \end{align}$$

where $\bar{n},\bar{q}_1$ and $\bar{q}_2$ must fulfill the same self-consistency equations (2.37) obtained by using Guerra's interpolation technique, as reported in proposition 1.

Solving numerically the self consistency equations (2.37), we obtain the phase diagram shown in figure 1, in the parameters space (γ, $\tilde{\beta}$), where γ is the storage load defined in (2.8) and $\tilde\beta = \beta^{^{\prime}}/(1+\rho)^{P/2}$ is a scaled inverse temperature, where ρ is the dataset entropy defined in (2.3). Panels show results for P = 4 and three different values of M, as shown in the legend. The result for $M\to \infty$ follows from the analysis carried out in section 2.4.1. In each panel, the grey curve marks the transition from the ergodic phase with $\bar{m} = \bar{q}_1 = \bar{q}_2 = 0$ (above) to the spin glass phase, where either $\bar{q}_1$ or $\bar{q}_2$ becomes non-zero (below). The black curve marks the transition from the retrieval phase with $\bar{m}\neq 0$ (left) to $\bar{m} = 0$ (right). The spin glass solution within the retrieval region is always unstable and it is delimited by the dotted curve (we refer to this as the instability region). In figure 2 we show the results for $P = 6, 8, 10$, as shown in the legend. As P and M grow, the spin glass region shrinks and the retrieval region expands. For $M = +\infty$ the critical storage lines become equal to those of traditional DAM models where archetypes are encoded in the interactions, rather than their noisy examples [45]. This is as expected: when an infinite number of examples is provided, the system reaches the same performance as a neural network where archetypes are stored in the interactions directly. For finite M, however, the network's ability to retrieve the archetypes degrades at values of the storage load which are lower than the storage capacity in traditional DAM models. Both figures 1 and 2 have been obtained for the value of $\theta = \theta_\otimes$ which maximizes the retrieval region, as shown in figure 3(left panel). In the latter, we show the line that separates the retrieval region from the spin-glass or the ergodic region, for different values of θ. For θ = 0 (which corresponds to the RS theory), the line exhibits a re-entrance, as commonly observed in spin-glasses and associative memories [45, 50, 52, 53]. As θ is increased, the retrieval region expands, reaching its maximum at $\theta = \theta_\otimes$, where the re-entrance disappears completely, showing the same qualitative behaviour as in the Hopfield model [12] and traditional DAM [45]. Increasing θ above $\theta_\otimes$ the re-entrance appears again and gets more pronounced as θ approaches 1, where the transition line becomes identical to the RS (and the θ = 0) case, as expected. In figure 3(right panel) we show the dependence of $\theta_\otimes$ on P. As P increases, the value of $\theta_{\otimes}$ decreases. This is as expected from spin-glass theory, as for large P, P-spin models are known to converge to the REM model [54], which is RS. Interestingly, we find that $\theta_\otimes$ takes the same value as in the DAM model considered in [45], for all values of M, hence the optimal 1RSB breaking parameter is not influenced by the use of corrupted examples rather than archetypes.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

2.4.1. Limiting cases: $M\to \infty$ and $\beta\to \infty$.

Finally, we consider two instructive limiting cases. One is the limit $M\to\infty$, where the number of available examples is large. Although idealised, this scenario is becoming less utopian nowadays in a number of applications, and it is instructive as an explicit relation between the archetype magnetization and the mean magnetization of the examples naturally emerges. The second scenario is the zero noise limit $\beta\to\infty$, where the information processing capabilities of the network are expected to be maximal. We now state the following

Corollary 1. In the limit $M\to\infty$, the 1RSB self consistent equations for the order parameters of the unsupervised DAM model are

$$\begin{align} \bar{n}& = \dfrac{\bar{m}}{1+\rho} +\beta ^{^{\prime}} \dfrac{P}{2}\rho\left(1+\rho\right)^{P/2-2}\left[1+\left(\theta-1\right)\bar{q}_2-\theta \bar{q}_1\right]\bar{n}^{^{P-1}},\nonumber \\ \bar{q}_1& = \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right)} \right]^2, \nonumber \\ \bar{q}_2 & = \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right) \tanh^2 g\left(\beta, \gamma, \boldsymbol{Y}\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right)} \right],\\ \bar{m}& = \mathbb{E}_{Y^{\left(1\right)}}\left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right)}\right].\nonumber \end{align} \tag{ 2.70 }$$

where

$$\begin{align} g\left(\beta, \gamma, \boldsymbol{Y}\right) & = \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{^{P-1}}\left(1+\rho\right)^{P/2-1} + Y^{\left(2\right)} \beta^{^{\prime}} \sqrt{ \gamma\dfrac{P}{2} \dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^P}\left(\bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)} \nonumber \\ & \quad +\ \beta ^{^{\prime}}Y^{\left(1\right)}\sqrt{\rho\left(\dfrac{P}{2}\bar{n}^{^{P-1}}\left(1+\rho\right)^{P/2-1}\right)^2+ \gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\,\bar{q}_1^{^{P-1}}\;} \end{align} \tag{ 2.71 }$$

and $\beta^{^{\prime}} = 2\beta /P!$.

Proof. In the limit $M\to \infty$, we can apply the CLT to the sum of the examples defined in (2.36), to write

$$\begin{equation} \hat{\eta}_{\tiny M}\sim \xi^1\left(1+\lambda\sqrt{\rho}\right), \end{equation} \tag{ 2.72 }$$

where λ is a standard Gaussian variable $\lambda\sim \mathcal{N}(0,1)$.

Inserting (2.72) in (2.38) and in the expression for $\bar{n}$ provided in (2.37), we get

$$\begin{align} g\left(\beta, \gamma, \boldsymbol{Y}, \lambda, \xi^1\right) & = \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{P-1}\left(1+\rho\right)^{P/2-1}\xi^1\left(1+\sqrt{\rho} \lambda\right)+Y^{\left(1\right)} \beta ^{^{\prime}}\sqrt{\dfrac{\gamma P \sqrt{\left(1+\rho_P\right)}^2}{2\left(1+\rho\right)^{P}} \bar{q}_1^{^{P-1}}} \nonumber \\ & \quad +\ Y^{\left(2\right)} \beta ^{^{\prime}}\sqrt{ \dfrac{\gamma \sqrt{\left(1+\rho_P\right)}^2 P}{2\left(1+\rho\right)^{P}} \left( \bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)}\,, \end{align} \tag{ 2.73 }$$

and

$$\begin{align*}\bar{n} = \frac{1}{\left(1+\rho\right)} \mathbb{E}_{\xi^1}\mathbb{E}_{\lambda}\mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}}\cosh^{\theta} g\left(\beta,\gamma, \boldsymbol{Y},\lambda, \xi^1\right) \tanh^{\theta} g\left(\beta,\gamma, \boldsymbol{Y},\lambda,\xi^1\right) }{\mathbb{E}_{Y^{\left(2\right)}}\cosh^{\theta} g\left(\beta,\gamma, \boldsymbol{Y},\lambda,\xi^1\right)} \cdot {\xi^1\left(1+\lambda \sqrt{\rho}\right)}\right].\end{align*}$$

Hence, by applying to the standard Gaussian variable λ the Stein's lemma, which states that for a standard Gaussian variable $J\sim N(0, 1)$ and a generic function f(J), for which the two expectations $\mathbb{E}\left( J f(J)\right)$ and $\mathbb{E}\left( \partial_J f(J)\right)$ both exist, one has

$$\begin{equation} \mathbb{E} \left[ J f\left(J\right)\right] = \mathbb{E} \left[ \frac{\partial f\left(J\right)}{\partial J}\right], \end{equation} \tag{ 2.74 }$$

and explicitly averaging over ξ we get the self-consistency equations in (2.70). □

Finally, it has been shown in [38], within a RS analysis, that neglecting the second term in the equation for $\bar{n}$, given in (2.70), has negligible impact on the solutions of the self-consistency equations, in the relevant regime of low noise, where the network works as an associative memory. As the equation for $\bar{n}$ is the same in the RS theory and in the 1RSB theory that we are considering here, the same truncation of $\bar{n}$ can be adopted here

$$\begin{equation} \bar{n} = \dfrac{\bar{m}}{1+\rho}. \end{equation} \tag{ 2.75 }$$

This leads to a simplified expression for g that is

$$\begin{align} g\left(\beta, \gamma, \boldsymbol{Y}\right) & = \tilde\beta\dfrac{P}{2}\bar{m}^{^{P-1}}\left(1+\rho\right)^{P/2-1} + \tilde\beta Y^{\left(2\right)} \sqrt{ \gamma\dfrac{P}{2} \sqrt{\left(1+\rho_P\right)}^2\left(\bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)} \nonumber \\ & \quad + \tilde\beta Y^{\left(1\right)}\sqrt{\rho\left(\dfrac{P}{2}\bar{m}^{^{P-1}}\right)^2+ \gamma \dfrac{P}{2}\sqrt{\left(1+\rho_P\right)}^2\,\bar{q}_1^{^{P-1}}\;}\; \end{align} \tag{ 2.76 }$$

where we re-scaled the noise $\tilde\beta = \beta^{^{\prime}}/(1+\rho)^{P/2}$. Solving numerically the self consistency equations (2.70) where the equation for $\bar{n}$ is replaced with its truncated version (2.75), leads to the phase diagram shown by the lines $M = +\infty$ in figures 1 and 2.

Next, we turn to the analysis of the ground state, i.e. to the limit $\beta \to \infty$, and we state the following

Corollary 2. In the limit $\beta\to\infty$ and for $M\gg 1$, the 1RSB self consistency equations for the order parameters of the unsupervised DAM model are

$$\begin{align} \bar{m} & = 1- 2 \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{1}{1+\exp\left( 2D\left(A_1 + A_2 Y^{\left(1\right)}\right)\right)\mathcal{Q}\left(A\right)}\right] \end{align} \tag{ 2.77 }$$

$$\begin{align} \Delta \bar{q}& = \bar{q}_2-\bar{q}_1 = 4 \mathbb{E}_{Y^{\left(1\right)}} \left\{\dfrac{\exp\left( 2D\left(A_1 + A_2 Y^{\left(1\right)}\right)\right)\mathcal{Q}\left(A\right)}{\left[1+\exp\left( 2D\left(A_1 + A_2 Y^{\left(1\right)}\right)\right)\mathcal{Q}\left(A\right)\right]^2}\right\} \end{align} \tag{ 2.78 }$$

where $D = \tilde\beta \theta$ and

$$\begin{equation} \begin{array}{lll} &\mathcal{Q}\left(A\right) = \dfrac{1+\mathrm{erf}\left(K^{+}\right)}{1+\mathrm{erf}\left(K^{-}\right)}, \ \ \ &K^{\pm} = \dfrac{D A_3^2 \pm \left(A_1 + A_2 Y^{\left(1\right)}\right)}{A_3 \sqrt{2}}, \\ \\ &A_1 = \dfrac{P}{2}\bar{m}^{P-1}, \ \ \ & A_2 = \sqrt{\rho A_1^2+\gamma\sqrt{\left(1+\rho_P\right)}^2\dfrac{P}{2}}, \\ \\ &A_3 = \sqrt{\gamma\sqrt{\left(1+\rho_P\right)}^2\dfrac{P\left(P-1\right)}{2}\Delta\bar{q}}. \end{array} \end{equation} \tag{ 2.79 }$$

We report the proof of this corollary in appendix B.2.

Next we compute, by numerically solving (2.77) and (2.78), the ground-state critical storage capacity γ_C beyond which a black-out scenario emerges, namely $\bar{m}\neq 0$ for $\gamma\lt\gamma_C$ and $\bar{m} = 0$ for $\gamma\gt\gamma_C$. In figure 4 we plot γ_C as a function of the ratio between the number M of experienced examples and the minimum number $M_\otimes$ of examples required by the unsupervised DAM model (with P > 2) to correctly learn and retrieve an archetype, within the RS theory, which has been proved to be

$$\begin{align} M^\textrm{unsup}_{\otimes}\left(r, P, \gamma\right) = \gamma \dfrac{1}{P} \dfrac{1}{r^{2P}} \end{align} \tag{ 2.80 }$$

in [37]. In order to ascertain whether replica symmetry breaking alters such threshold, we plot γ_C within both the 1RSB assumption (blue line) and the RS theory (red line). We see that γ_C becomes non-zero (meaning that retrieval can occur) at the same value of $M = M_\otimes$ for the RS and the 1RSB theory. This shows that the minimum number of examples that a network needs to accomplish retrieval of the archetype is the same within the RS or the 1RSB assumption.

Zoom In Zoom Out Reset image size

Results show that, as in the standard Hebbian storage [44, 53, 55], the phenomenon of replica symmetry breaking induces a mild improvement in terms of the maximal storage.

As in the unsupervised DAM model, we consider a network of N Ising neurons $\sigma_i \in \{-1, +1\}$, $i = 1, \ldots, N$, interacting via P-node interactions. We assume to have K Rademacher archetypes $\boldsymbol{\xi}^\mu \in \{-1, +1\}^N$, with $\mu = 1, \ldots, K$, defined as N-dimensional vectors with entries drawn randomly and independently from the distribution (2.1). In addition, we assume to have M examples $\eta^{\mu,a} \in \{-1, +1\}^N$ for each archetype, which are corrupted version of the archetypes, with entries distributed according to (2.2). We shall refer to this model as the supervised DAM model.

Definition 6. The Hamiltonian of the supervised DAM model is

$$\begin{align} \mathcal H^{\left(P\right)}_{N}\left(\boldsymbol{\sigma} \vert \boldsymbol{\eta}\right) = & -\dfrac{1}{\mathcal{R}^{P/2}N^{P-1}M^P}\displaystyle\sum\limits_{\mu = 1}^{K}\displaystyle\sum\limits_{a_1,\cdots,a_P = 1}^{M,\cdots,M}\left(\displaystyle\sum\limits_{i_1\lt \cdots\lt i_P}^{N,\cdots,N}\eta^{\mu, a_1}_{i_1}\cdots\eta^{\mu, a_P}_{i_P}\sigma_{i_1}\cdots\sigma_{i_P}\right) \end{align} \tag{ 3.1 }$$

where the constant $\mathcal{R}: = r^2 + \frac{1-r^2}{M}$ in the denominator of the r.h.s. is included for mathematical convenience and the factor $N^{P-1}$ ensures the Hamiltonian to be ${\mathcal O}(N)$, as explained previously for the unsupervised model, see equation (2.4).

We highlight the hidden role of a teacher that, before providing the dataset to the network, has grouped examples pertaining to the same archetype together (hence the proliferation of summations in the cost function (3.1) with respect to (2.4)).

As before, we will focus (without loss of generality) on the ability of the network to store and retrieve the first archetype $\boldsymbol \xi^1$, hence the Mattis magnetization $m_1(\boldsymbol{\sigma})$ provided in (2.16) remains a relevant order parameter. However, the set of order parameters for the examples, previously given by $n_{1,a}(\boldsymbol{\sigma})$ with $a = 1\ldots M$ (see (2.17)) have now to be substituted with a single order parameter

$$\begin{equation} n_1\left(\boldsymbol{\sigma}\right) = \dfrac{r}{\mathcal{R}}\dfrac{1}{N M}\displaystyle\sum\limits_{i,a = 1}^{N,M}\eta_i^{1,a}\sigma_i. \end{equation} \tag{ 3.2 }$$

Its probability distribution, in the thermodynamic limit, is assumed self-averaging, namely

$$\begin{equation} \lim_{N \rightarrow + \infty} \mathbb{P}_N\left(n_1|\boldsymbol{\sigma}\right) = \delta \left(n_1\left(\boldsymbol{\sigma}\right) - \bar{n}\right), \end{equation} \tag{ 3.3 }$$

as for the Mattis magnetization, while the distribution for the overlap defined in (2.18) is still assumed bimodal as in assumption 1 (see (2.20)). As for the unsupervised model, we add an extra term in the cost function, namely $J \sum_i \xi_i^1 \sigma_i$, in order to generate the moments of the Mattis magnetization m₁ by taking the derivatives of the quenched free energy w.r.t. J and, as this term is not part of the original Hamiltonian, it will be set to zero at the end of the calculations. Therefore, we write the partition function as

$$\begin{align} \mathcal{Z}^{\left(P\right)}_{N} \left(\boldsymbol{\eta}\right) & = \lim_{J \rightarrow 0} \mathcal{Z}^{\left(P\right)}_{N,K,M,\beta}\left( \boldsymbol{\eta} ;J\right) \nonumber \\ & = \lim_{J \rightarrow 0} \sum_{\boldsymbol{\sigma}} \exp \left\{J \sum_i \xi_i^1 \sigma_i +\beta^{^{\prime}}\dfrac{N}{2}\left(1+\rho\right)^{P/2}n_1^{P}\left(\boldsymbol{\sigma}\right) +\dfrac{\beta^{^{\prime}}P!}{2N^{P-1}\mathcal{R}^{P/2} }\displaystyle\sum\limits_{\mu>1}^{K}\displaystyle\sum\limits_{i_1 \lt \cdots \lt i_{P}}^{N,\cdots,N}\right. \nonumber\\ & \left. \quad\times \left[ \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_1 = 1}^{M}\eta_{i_1}^{\mu,a_1} \right)\ldots \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_P = 1}^{M}\eta_{i_{P}}^{\mu,a_{P}}\right) \right]\sigma_{i_1}\cdots\sigma_{i_{P}} \right\} \end{align} \tag{ 3.4 }$$

where ρ is the dataset entropy defined in (2.3), $\beta^{^{\prime}}: = 2\beta/P!$ and for the first pattern, µ = 1, we have used the relation $P!\sum_{i_1 \lt\cdots \lt i_P} = \sum_{i_1, \cdots, i_P}$ and neglected terms which vanish in the thermodynamic limit. Next, as done for the unsupervised case, we apply the CLT to the variables in the square brackets in (3.4), namely

$$\begin{align} \dfrac{1}{K}\displaystyle\sum\limits_{\mu>1}^{K}\left[ \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_1 = 1}^{M}\eta_{i_1}^{\mu,a_1} \right)\ldots \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_P = 1}^{M}\eta_{i_{P}}^{\mu,a_{P}}\right) \right]\sim \mu_1+\lambda_{i_1,\ldots,i_P} \frac{\sqrt{\mu_2-\mu_1^2}}{\sqrt{K}},\quad \lambda_{i_1\ldots i_P}\sim {\mathcal N}\left(0,1\right), \end{align} \tag{ 3.5 }$$

where

$$\begin{equation} \mu_1 = \mathbb{E}_{\boldsymbol{\xi}}\mathbb{E}_{\left(\boldsymbol{\eta}|\boldsymbol{\xi}\right)}\left[ \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_1 = 1}^{M}\eta_{i_1}^{\mu,a_1} \right)\ldots \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_P = 1}^{M}\eta_{i_{P}}^{\mu,a_{P}}\right) \right] = \mathbb{E}_{\boldsymbol{\xi}}\left[ \left(r\xi_{i_1}^{\mu} \right)\ldots \left(r\xi_{i_{P}}^{\mu}\right) \right] = 0 \end{equation} \tag{ 3.6 }$$

and

$$\begin{align} \mu_2 & = \mathbb{E}_{\boldsymbol{\xi}}\mathbb{E}_{\left(\boldsymbol{\eta}|\boldsymbol{\xi}\right)}\left[ \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_1 = 1}^{M}\eta_{i_1}^{\mu,a_1} \right)^2\ldots \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_P = 1}^{M}\eta_{i_{P}}^{\mu,a_{P}}\right)^2 \right]\nonumber\\ & = r^2\left(1+\dfrac{1-r^2}{Mr^2}\right)\cdots r^2\left(1+\dfrac{1-r^2}{Mr^2}\right) = r^{2P}\left(1+\dfrac{1-r^2}{Mr^2}\right)^P \end{align} \tag{ 3.7 }$$

with

$$\begin{align*} \rho = \dfrac{1-r^2}{Mr^2}. \end{align*}$$

Thus, we can write

$$\begin{align*} \frac{1}{K}\displaystyle\sum\limits_{\mu>1}^{K}\left[ \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_1 = 1}^{M}\eta_{i_1}^{\mu,a_1} \right)\ldots \left(\dfrac{1}{M}\displaystyle\sum\limits_{a_P = 1}^{M}\eta_{i_{P}}^{\mu,a_{P}}\right) \right] = \lambda_{i_1,\ldots,i_P} \sqrt{\dfrac{r^{2P}\left(1+\rho\right)^{P}}{K}} \end{align*}$$

where $\lambda_{i_1,\ldots,i_P}\sim \mathcal{N}(0,1)$.

Inserting all back in (3.4) we get

$$\begin{align} \mathcal{Z}^{\left(P\right)}_{N,K,M,\beta} \left(\boldsymbol{\eta}^1,\boldsymbol{\lambda};J\right) & = \exp \left\{J \sum_i \xi_i^1 \sigma_i +\beta^{^{\prime}}\dfrac{N}{2}\left(1+\rho\right)^{P/2}n_1^{P}\left(\boldsymbol{\sigma}\right)+\dfrac{\beta^{^{\prime}}P!\sqrt{K}}{2N^{P-1} }\displaystyle\sum\limits_{i_1 \lt \cdots \lt i_{P}}^{N,\cdots,N} \lambda_{i_1,\ldots,i_{P}} \sigma_{i_1}\cdots\sigma_{i_{P}} \right\}\,. \end{align} \tag{ 3.8 }$$

In the next subsection we calculate the free energy of the system in the thermodynamic limit using Guerra's interpolation technique, assuming one step of replica symmetry breaking (1RSB).

As for the unsupervised case, the plan is to construct an interpolation between the original model and a simpler one-body model, whose statistical features are as close as possible to the original one, then solve the one body model and finally obtain the solution of the original model via the fundamental theorem of calculus. Thus we define the following interpolating partition function

Definition 7. Given the interpolating parameter $t \in [0,1]$, $A_1,\ A_2, \ \psi \in \mathbb{R}$ constants to be set a posteriori, and the i.i.d. standard Gaussian variables $Y_i^{(b)} \sim \mathcal{N}(0,1)$ for $i = 1, \ldots, N$ and $b = 1,2$, the Guerra's 1-RSB interpolating partition function for the DAM model, trained by a teacher, is given by

$$\begin{align} \mathcal{Z}^{\left(P\right)}_2\left(\boldsymbol{\eta}^1,\boldsymbol{\lambda},\boldsymbol{Y}; J, t\right)& : = \displaystyle\sum\limits_{\lbrace\boldsymbol \sigma\rbrace} \mathcal{B}_{N,2}^{\left(P\right)}\left(\boldsymbol{\sigma} \vert \boldsymbol{\eta}^1,\boldsymbol{\lambda},\boldsymbol{Y}; J, t\right)\nonumber\\ & = \displaystyle\sum\limits_{\lbrace\boldsymbol \sigma\rbrace} \exp\left[J \sum_{i = 1}^N \xi_i^1 \sigma_i+\dfrac{t \beta ^{^{\prime}}N}{2M}\left(1+\rho\right)^{P/2}n_{1}^{^P}\left(\boldsymbol{\sigma}\right) +\psi\left(1-t\right)Nn_{1}\left(\boldsymbol{\sigma}\right)\right.\nonumber\\ & \quad \left.+\ \sqrt{t}\dfrac{\beta ^{^{\prime}}\sqrt{K}}{N^{P-1}}\displaystyle\sum\limits_{{i_1 \lt\cdots \lt i_P}}^{N,\cdots,N}\lambda_{i_1,\ldots,i_P}\sigma_{_{i_1}}\cdots\sigma_{_{i_{_{P}}}}+\sqrt{1-t}\displaystyle\sum\limits_{b = 1}^{2}A_b\displaystyle\sum\limits_{i = 1}^{N}Y_i^{\left(b\right)}\sigma_i\right]. \end{align} \tag{ 3.9 }$$

where $\mathcal{B}_{N,2}^{(P)}(\boldsymbol{\sigma} \vert \boldsymbol{\eta}^1,\boldsymbol{\lambda},\boldsymbol{Y}; J, t)$ is denoted as Boltzmann factor.

As before, the introduction of the interpolating partition function gives rise to a generalized measure, average and interpolating quenched free energy (that we do not repeat here). All these generalizations retrieve the standard definitions when evaluated at t = 1. Following Guerra's method [43], we must average out the fields ${\mathbf{Y}}^{(1)}$ and ${\mathbf{Y}}^{(2)}$ recursively, in the interpolating free energy resulting from the interpolating partition function (3.9), as already done in the previous section for the unsupervised case, see equations (2.26)–(2.28). Proceeding in the same way and omitting obvious details due to the similarity of the proofs in the supervised and unsupervised cases, we state directly the next.

Proposition 2. In the thermodynamic limit $N\to\infty$, within the 1RSB assumption and under the assumption 2, the quenched free energy for the supervised DAM model, reads as

$$\begin{align} -\beta^{^{\prime}}\mathcal{F}^{\left(P\right)} \left(J\right) & = \ln 2+ \dfrac{1}{\theta}\mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)}\mathbb{E}_{Y^{\left(1\right)}} \ln\mathbb{E}_{Y^{\left(2\right)}}\left\{\cosh^\theta\left[ J\xi^1+\beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M} \right. \right.\nonumber\\ &\quad \left. \left. +\ Y^{\left(1\right)} \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2} \bar{q}_1^{^{P-1}}}+Y^{\left(2\right)} \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2} \left(\bar{q}_2^{^{P-1}} -\bar{q}_1 ^{^{P-1}}\right)}\right]\right\} +\dfrac{{\beta^{^{\prime}}}^2}{4}\gamma\left[1-P\bar{q}_2^{P-1}+\left(P-1\right)\bar{q}_2^P\right]\nonumber\\ &\quad -\dfrac{{\beta^{^{\prime}}}}{2}\left(P-1\right)\left(1+\rho\right)^{P/2}\bar{n}^P-\dfrac{{\beta^{^{\prime}}}^2}{4}\gamma\left(P-1\right)\theta\left(\bar{q}_2^P-\bar{q}_1^P\right), \end{align} \tag{ 3.10 }$$

with $\bar n$, $\bar{q}_1, \ \bar{q}_2$ fulfilling the following self-consistency equations

$$\begin{equation} \begin{array}{lll} \bar{n} = \dfrac{1}{1+\rho}\mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}\dfrac{1}{r} \hat{\chi}_{\tiny M} \right], \\ \\ \bar{q}_1 = \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)}\mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)} \right]^2, \\ \\ \bar{q}_2 = \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) \tanh^2 g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)} \right], \end{array} \end{equation} \tag{ 3.11 }$$

where

$$\begin{align} g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) & = \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M}+Y^{\left(1\right)} \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2} \bar{q}_1^{^{P-1}}}+Y^{\left(2\right)} \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2} \left( \bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)} \end{align} \tag{ 3.12 }$$

and $\beta^{^{\prime}} = 2\beta /P!$. Furthermore, as $\bar{m} = -\beta^{^{\prime}} \nabla_J \mathcal{F}^{(P)}(J)|_{J = 0}$, we have

$$\begin{align} \bar{m} = \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)}\mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}, \boldsymbol{\eta}^1\right)}\xi^1\right]. \end{align} \tag{ 3.13 }$$

The proof of the aforementioned proposition is lengthy but the steps to follow are identical to the ones provided for the unsupervised case.

As stated in the previous section, the core of this approach consists in writing the logarithm of the partition function as $\lim\limits_{n\to 0} (\mathbb{E}\mathcal{Z}(J)^n-1)/n$ where

$$\begin{align} \mathbb{E}\mathcal{Z}_{N}^n\left(J\right) & = \mathbb{E} \sum_{\boldsymbol{\sigma}^1 \ldots \boldsymbol{\sigma}^n} \exp\left[\dfrac{\beta^{^{\prime}} P!}{2 \mathcal{R}^{P/2}N^{P-1}M^P}\displaystyle\sum\limits_{a = 1}^{n}\displaystyle\sum\limits_{\mu = 1}^{K}\left(\displaystyle\sum\limits_{i_1 \lt \cdots \lt i_P}^{N,\ldots,N}\displaystyle\sum\limits_{A_1,\ldots,A_P}^{M,\ldots,M}\eta^{\mu, A_1}_{i_1}\cdots\eta^{\mu, A_P}_{i_P}\sigma_{i_1}^{\left(a\right)}\cdots\sigma_{i_P}^{\left(a\right)}\right) \right. \nonumber \\ & \quad \left. +\ J\displaystyle\sum\limits_{a,i}^{n,N}\xi_i^1\sigma_i^{\left(a\right)} \right]. \end{align} \tag{ 3.14 }$$

Proceeding as in the unsupervised case, we compute separately the signal term and the noise term. The former, accounting for the examples pertaining to the first archetype, reads as

$$\begin{align} \mathbb{E}\mathcal{Z}_\textrm{signal}^n\left(J\right) & = \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \sum_{\boldsymbol{\sigma}^1 \ldots \boldsymbol{\sigma}^n} \exp\left[\dfrac{\beta^{^{\prime}} N \left(1+\rho\right)^{P/2}}{2 }\displaystyle\sum\limits_{a = 1}^{n}\left(\dfrac{r}{N M \mathcal{R}}\displaystyle\sum\limits_{i}^{N}\displaystyle\sum\limits_{A}^{M}\eta^{1, A}_{i}\sigma_{i}^{\left(a\right)}\right)^P \right. \nonumber \\ & \quad \left. +\ J\displaystyle\sum\limits_{a = 1}^{n}\displaystyle\sum\limits_{i = 1}^{N}\xi_i^1\sigma_i^{\left(a\right)} \right], \end{align} \tag{ 3.15 }$$

while the latter accounting for the examples pertaining to all the other archetypes but the first one, can be written as

$$\begin{align} \mathbb{E}\mathcal{Z}_\textrm{noise}^n & = \mathbb{E}_{\boldsymbol{\lambda}} \sum_{\boldsymbol{\sigma}^1 \ldots \boldsymbol{\sigma}^n} \exp\left[\dfrac{\beta^{^{\prime}} P!\sqrt{K}}{2 N^{P-1}}\displaystyle\sum\limits_{a = 1}^{n}\left(\displaystyle\sum\limits_{i_1 \lt \cdots \lt i_P}^{N,\ldots,N}\lambda_{i_1,\ldots,i_P}\sigma_{i_1}^{\left(a\right)}\cdots\sigma_{i_P}^{\left(a\right)}\right) \right] \end{align} \tag{ 3.16 }$$

where now $\mathbb{E}_{\boldsymbol{\lambda}}$ is the Gaussian average w.r.t. λ . Performing the integration over λ and inserting the definitions of the order parameters as done for the unsupervised setting, we can write

$$\begin{align} &\mathbb{E}\mathcal{Z}_N^n\left(J\right) = \int \prod_{a} \textrm{d} n_{1}^{\left(a\right)} \prod_{a} \dfrac{\textrm{d}z_{1}^{\left(a\right)}}{2\pi} \int \prod_{a,b} \textrm{d}q_{ab} \int \prod_{a,b} \dfrac{\textrm{d}p_{ab}}{2\pi} \exp\left( -N A\left[\boldsymbol{Q}, \boldsymbol{P},\boldsymbol{N}, \boldsymbol{Z}\right] \right) \end{align} \tag{ 3.17 }$$

where

$$\begin{align} A\left[\boldsymbol{Q},\boldsymbol{P},\boldsymbol{N},\boldsymbol{Z}\right]& = - \dfrac{i}{N} \sum_{a,b} p_{ab}q_{ab} -\dfrac{1}{4}{\beta^{^{\prime}}}^2\gamma \displaystyle\sum\limits_{a,b}^{n,n}q_{ab}^P - \dfrac{\beta^{^{\prime}} \left(1+\rho\right)^{P/2}}{2} \sum_{a} \left(n_{1}^{\left(a\right)}\right)^P-\dfrac{i}{N}\sum_{a} n_{1}^{\left(a\right)} z_{1}^{\left(a\right)} \nonumber \\ & \quad -\mathbb{E} \log \left[ \sum_{\boldsymbol{\sigma}} \exp\left( -\dfrac{i}{N} \sum_{a,b} p_{ab} \sigma^{\left(a\right)}\sigma^{\left(b\right)} - \dfrac{i}{N\left(1+\rho\right)} \sum_{a} \hat\eta_{_M} z_{1}^{\left(a\right)} \sigma^{\left(a\right)}+J\displaystyle\sum\limits_{a}^{}\xi^1\sigma^{\left(a\right)}\right)\right]. \end{align} \tag{ 3.18 }$$

and $\hat{\eta}_{M} = (rM)^{-1}\sum_{A = 1}^M\eta^{1,A}$. Assuming (as usual within the replica method) that the limits $N\to\infty$ and n → 0 can be interchanged, the integrals can be performed by steepest descent. This leads to

$$\begin{align} -\beta^{^{\prime}} \mathcal{F}^{\left(P\right)}\left(J\right) = \lim_{n\to 0} \dfrac{1}{n} A\left[\boldsymbol{Q}^\star, \boldsymbol{P}^\star, \boldsymbol{Z}^\star, \boldsymbol{N} ^\star\right] \end{align} \tag{ 3.19 }$$

where $\boldsymbol{Q}^\star, \boldsymbol{P}^\star, \boldsymbol{Z}^\star, \boldsymbol{N} ^\star$, are solution of the saddle point equations

$$\begin{eqnarray} &\dfrac{\partial A}{\partial p_{ab}} = 0&\quad\Rightarrow\quad q_{ab}^* = \langle \sigma^a \sigma^b\rangle_{\mathrm{eff}} \end{eqnarray} \tag{ 3.20 }$$

$$\begin{eqnarray} &\dfrac{\partial A}{\partial z_{1}^{\left(a\right)}} = 0&\quad\Rightarrow\quad \left(n_{1}^{\left(a\right)}\right)^* = \dfrac{r}{\mathcal{R}}\displaystyle\sum\limits_{A = 1}^{M}\mathbb{E}\langle \eta^{1,A}\sigma^a \rangle_{\mathrm{eff}} \end{eqnarray} \tag{ 3.21 }$$

$$\begin{eqnarray} &\dfrac{\partial A}{\partial q_{ab}} = 0&\quad\Rightarrow\quad p_{ab}^* = i \beta^{^{\prime}}\,^2 \dfrac{P}{4}\gamma N \left(q_{ab}^*\right)^{P-1} \end{eqnarray} \tag{ 3.22 }$$

$$\begin{eqnarray} &\dfrac{\partial A}{\partial n_{1}^{\left(a\right)}} = 0&\quad\Rightarrow\quad \left(z_{1}^{\left(a\right)}\right)^* = i \beta^{^{\prime}} \dfrac{P}{2} \left(1+\rho\right)^{P/2} N \left[\left(n_{1}^{\left(a\right)}\right)^*\right]^{P-1} \end{eqnarray} \tag{ 3.23 }$$

which provides a set of self-consistency equations for Q , N , P , and Z . Using the last two equations to eliminate P and Z from the description, we finally obtain

$$\begin{align} -\beta^{^{\prime}} \mathcal F\left(J\right)& = \lim_{n\to 0} \dfrac{1}{n} \left\{\dfrac{1}{4}{\beta^{^{\prime}}}^2\gamma \left(P-1\right)\displaystyle\sum\limits_{a,b}^{n,n}\left[q_{ab}^*\right]^P + \dfrac{\beta^{^{\prime}} \left(1+\rho\right)^{P/2}}{2}\left(P-1\right) \sum_{a} \left[\left(n_{1}^{\left(a\right)}\right)^*\right]^P \right. \nonumber\\ & \left.\quad -\ \mathbb{E} \log \left[ \sum_{\boldsymbol{\sigma}} \exp\left( \beta^{^{\prime}}\,^2 \dfrac{P}{4}\gamma\sum_{a,b} \left[q_{ab}^*\right]^{P-1} \sigma^{\left(a\right)}\sigma^{\left(b\right)} + \beta^{^{\prime}} \dfrac{P}{2} \left(1+\rho\right)^{P/2-1}\sum_{a} \hat\eta_{_M} \left[\left(n_{1}^{\left(a\right)}\right)^*\right]^{P-1} \sigma^{\left(a\right)}\right.\right.\right. \nonumber \\ &\left. \left. \left.\quad +\ J\displaystyle\sum\limits_{a}^{}\xi^1\sigma^{\left(a\right)}\right)\right]\right\}\,. \end{align} \tag{ 3.24 }$$

To make progress, we need to find the form of Q and N in the limit n → 0. Again, we use the 1RSB ansatz for the two-replica overlap provided in (2.63), while we assume that $n_{1,a}$ is self-averaging, i.e. $n_{1}^{(a)} = \bar{n}$. Proceeding analogously to the unsupervised case, after taking the limit n → 0 we find the same expression for the quenched free energy (see (3.10)) and order parameters (see 3.11) previously obtained by using Guerra's interpolation technique.

Now we state the next corollaries concerning the large datasets $M \to\infty$ limit and the ground state $\beta \to \infty$ limit. We omit their proofs since these are trivial variations of those provided for the unsupervised case (cfr corollaries 1 and 2).

Corollary 3. In the large dataset limit $M\to \infty$, the 1RSB self-consistency equations for the order parameters of the supervised DAM model can be expressed as

$$\begin{align} \bar{n}& = \dfrac{\bar{m}}{1+\rho} +\beta ^{^{\prime}} \dfrac{P}{2}\rho\left(1+\rho\right)^{P/2-2}\left[1+\left(\theta-1\right)\bar{q}_2-\theta \bar{q}_1\right]\bar{n}^{^{P-1}},\nonumber \\ \bar{q}_1& = \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right)} \right]^2, \nonumber \\ \bar{q}_2 & = \mathbb{E}_{Y^{\left(1\right)}} \left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right) \tanh^2 g\left(\beta, \gamma, \boldsymbol{Y}\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right)} \right],\nonumber\\ \bar{m}& = \mathbb{E}_{_1}\left[\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta, \gamma, \boldsymbol{Y}\right) \tanh g\left(\beta, \gamma, \boldsymbol{Y}\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh^\theta g\left(\beta^{^{\prime}}, \gamma, \boldsymbol{Y}\right)}\right].\nonumber \end{align} \tag{ 3.25 }$$

where the expression of g reads as

$$\begin{align} g\left(\beta, \gamma, \boldsymbol{Y}\right)& = \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{^{P-1}}\left(1+\rho\right)^{P/2-1} + Y^{\left(2\right)} \beta^{^{\prime}}\sqrt{ \gamma\dfrac{ P}{2} \left(\bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)} \nonumber \\ &\quad + Y^{\left(1\right)}\beta^{^{\prime}}\sqrt{\rho\left(\dfrac{P}{2}\bar{n}^{^{P-1}}\left(1+\rho\right)^{P/2-1}\right)^2+ \gamma \dfrac{P}{2}\,\bar{q}_1^{^{P-1}}\;}\;.\end{align} \tag{ 3.26 }$$

Moreover, if we use the truncated expression for $\bar{n}$, namely

$$\begin{equation} \bar{n} = \dfrac{\bar{m}}{1+\rho} \end{equation} \tag{ 3.27 }$$

we get the simplified expression of (3.26) that is

$$\begin{align} g\left(\beta, \gamma, \boldsymbol{Y}\right)& = \tilde\beta\dfrac{P}{2}\bar{m}^{^{P-1}} + Y^{\left(2\right)} \tilde\beta \sqrt{ \gamma\dfrac{P}{2}\left(1+\rho\right)^P \left(\bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)} \nonumber \\ &\quad + Y^{\left(1\right)}\tilde\beta\sqrt{\rho\left(\dfrac{P}{2}\bar{m}^{^{P-1}}\right)^2+ \gamma\left(1+\rho\right)^P \dfrac{P}{2}\,\bar{q}_1^{^{P-1}}\;}\;, \end{align} \tag{ 3.28 }$$

where $\tilde\beta = \beta^{^{\prime}}/(1+\rho)^{P/2}$.

Corollary 4. The ground-state (i.e. $\beta \to \infty$) self-consistency equations for the order parameters of the theory in the large dataset limit (i.e. $M\gg 1$) and under the 1RSB assumption are

$$\begin{align} \bar{n} & = \dfrac{\bar{m}}{\left(1+\rho\right)} \end{align} \tag{ 3.29 }$$

$$\begin{align} \bar{m} & = 1- 2 \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{1}{1+\exp\left( 2D\left(A_1 + A_2 Y^{\left(1\right)}\right)\right)\mathcal{Q}\left(A\right)}\right] \end{align} \tag{ 3.30 }$$

$$\begin{align} \Delta \bar{q}& = \bar{q}_2-\bar{q}_1 = 4 \mathbb{E}_{Y^{\left(1\right)}} \left\{\dfrac{\exp\left( 2D\left(A_1 + A_2 Y^{\left(1\right)}\right)\right)\mathcal{Q}\left(A\right)}{\left[1+\exp\left( 2D\left(A_1 + A_2 Y^{\left(1\right)}\right)\right)\mathcal{Q}\left(A\right)\right]^2}\right\} \end{align} \tag{ 3.31 }$$

where $D = \tilde\beta \theta$ and

$$\begin{equation} \begin{array}{lll} &\mathcal{Q}\left(A\right) = \dfrac{1+\mathrm{erf}\left(K^{+}\right)}{1+\mathrm{erf}\left(K^{-}\right)}, \ \ \ &K^{\pm} = \dfrac{D A_3^2 \pm \left(A_1 + A_2 Y^{\left(1\right)}\right)}{A_3 \sqrt{2}}, \\ \\ &A_1 = \dfrac{P}{2}\bar{m}^{P-1}, \ \ \ & A_2 = \sqrt{\rho\left(\dfrac{P \bar{m}^{P-1}}{2}\right)^2+\gamma\left(1+\rho\right)^P\dfrac{P}{2}}, \\ \\ &A_3 = \sqrt{\gamma\left(1+\rho\right)^P\dfrac{P\left(P-1\right)}{2}\Delta\bar{q}}. \end{array} \end{equation} \tag{ 3.32 }$$

By numerically solving the self-consistency equations (3.25), we obtain the phase diagram shown in figure 5, for different values of P and different values of the dataset entropy ρ, as shown in the legend.

Zoom In Zoom Out Reset image size

In [38] it has been proven that the minimum number of examples $M_{\otimes}(r, \gamma)$ required by the supervised DAM model to correctly learn and retrieve an archetype, is given by

$$\begin{align} M_{\otimes}\left(r, \gamma\right) = \left(1+2\gamma\right)\dfrac{1}{r^{\tiny \mbox{2}}} \end{align} \tag{ 3.33 }$$

where γ ≠ 0. In figure 6 we plot the critical storage γ_C in the ground state versus the ratio $M/M_\otimes$. As previously observed for the unsupervised setting, the phenomenon of replica symmetry breaking slightly increases the critical storage and it does not alter the minimum number of examples required for learning.

Zoom In Zoom Out Reset image size

Following the method introduced in [39], we aim to determine the region in the phase diagram where the quenched free energy evaluated within the 1RSB approximation, that from now on we denote for convenience as $\mathcal F^{(P)}_{1RSB}(\bar{n}, \bar{q}_2,\bar{q}_1|\theta)$, is smaller than the free energy evaluated within the RS assumption, $\mathcal F^{(P)}_{RS}(\bar{n}^{^{\prime}}, \bar{q})$, in the limit θ → 1, where the transition from RS to RSB is expected to occur.

We start by recalling the expression for $\mathcal F^{(P)}_{1RSB}(\bar{n}, \bar{q}_2,\bar{q}_1|\theta)$, given in (2.35), with $\bar{n}$, $\bar{q}_1$ and $\bar{q}_2$ determined from the self-consistency equations (2.37), and by providing the expression for the quenched free energy within the RS assumption, as derived in [37]

$$\begin{align} -\beta^{^{\prime}} \mathcal F^{\left(P\right)}_{RS}\left(\bar{n}^{^{\prime}},\bar{q} \right) = & - \dfrac{{\beta^{^{\prime}}}^2 \left(1+\rho\right)^{P/2}}{2}\left(P-1\right) \bar{n}^{^{\prime} P} + \dfrac{{\beta^{^{\prime}}}^2 \gamma \sqrt{\left(1+\rho_P\right)}^2}{4\left(1+\rho\right)^P}\left(1-\bar{q}^P\right)\nonumber \\ &- \dfrac{{\beta^{^{\prime}}}^2 \gamma \sqrt{\left(1+\rho_P\right)}^2}{4\left(1+\rho\right)^P} \bar{q}^{P-1} \left(1-\bar{q}\right)+\ln 2 \nonumber \\ & + \mathbb{E} \ln \cosh \left( \beta^{^{\prime}} \left(1+\rho\right)^{P/2-1} \dfrac{P}{2} \hat{\eta}_{\tiny M} \bar{n}^{^{\prime} P-1} + \beta^{^{\prime}} z \sqrt{\gamma \dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^P} \dfrac{P}{2} \bar{q}^{P-1}}\right) \end{align} \tag{ A.1 }$$

where $\mathbb{E} = \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1)}\mathbb{E}_z$, $\hat{\eta}_{\tiny M}: = \frac{1}{rM}\displaystyle\sum\nolimits_{a = 1}^{M}\eta^{1,a}$, and $\bar{n}^{^{\prime}}$ and $\bar{q}$ fulfill the following self-consistency equations

$$\begin{equation} \begin{array}{lll} \bar{n}^{^{\prime}} = \dfrac{1}{1+\rho}\mathbb{E}\left\{\tanh{\left[\beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{^{\prime} P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M}+z \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^P } \bar{q}^{^{P-1}}}\;\right]}\hat{\eta}_M\right\}\, \end{array} \end{equation} \tag{ A.2 }$$

$$\begin{equation} \begin{array}{lll} \bar{q} = \mathbb{E}\left\{\tanh^{\tiny \mbox{2}}{\left[\beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{^{\prime} P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M}+z \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^P } \bar{q}^{^{P-1}}}\;\right]}\right\}. \end{array} \end{equation} \tag{ A.3 }$$

We note that for θ = 1, $\bar{q}_1 = \bar{q}$, $\bar{n} = \bar{n}^{^{\prime}}$ and the 1RSB expression for the quenched free- energy reduces to the RS one. Now, we expand the 1RSB expression for the quenched free energy $\mathcal F^{(P)}_\textrm{1RSB}(\bar{n},\bar{q}_2,\bar{q}_1|\theta)$, as given in (2.35), around θ = 1, using $\mathcal F^{(P)}_\textrm{1RSB}(\bar{n},\bar{q}_2,\bar{q}_1|\theta)\vert_{\theta = 1} = \mathcal F^{(P)}_\textrm{RS}(\bar{n}^{^{\prime}},\bar{q})$

$$\begin{align} \mathcal{F}_\textrm{1RSB}\left(\bar{n},\bar{q}_2,\bar{q}_1|\theta\right) = \mathcal{F}_\textrm{RS}\left(\bar{n}^{^{\prime}},\bar{q}\right)+ \left(\theta -1\right)\partial_\theta \mathcal{F}_\textrm{1RSB} \left(\bar{n},\bar{q}_2,\bar{q}_1|\theta\right)\vert_{\theta = 1}. \end{align} \tag{ A.4 }$$

Since $\bar{q}_1$ and $\bar{q}_2$ are determined through the self-consistency equations (2.37), they depend on θ as well, so we have to expand (2.37) around θ = 1 too. Following [39] closely, we obtain

$$\begin{eqnarray} \bar{q}_1& = & \bar{q}+\left(\theta-1\right) A\left( \bar{n}^{^{\prime}},\, \bar{q},\tilde q_2\left(\bar{n}^{^{\prime}},\, \bar{q}\right)\right) \end{eqnarray} \tag{ A.5 }$$

$$\begin{eqnarray} \bar{q}_2& = & \tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right)+\left(\theta-1\right) B\left( \bar{n}^{^{\prime}},\, \bar{q},\tilde q_2\left(\bar{n}^{^{\prime}},\, \bar{q}\right)\right) \end{eqnarray} \tag{ A.6 }$$

where $\bar{q}$ is the solution of (A.3), $\tilde q_2(\bar{n}^{^{\prime}},\bar{q})$ is the solution of the following self-consistency equation

$$\begin{align} \bar{q}_2 = \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh {g\left(\bar{n}^{^{\prime}},\bar{q},\tilde q_2\left(\bar{n}^{^{\prime}},\, \bar{q}\right)\right)} \tanh^2 {g\left(\bar{n}^{^{\prime}},\bar{q},\tilde q_2\left(\bar{n}^{^{\prime}},\, \bar{q}\right)\right)}}{\mathbb{E}_{Y^{\left(2\right)}} \cosh {g\left(\bar{n}^{^{\prime}},\bar{q},\tilde q_2\left(\bar{n}^{^{\prime}},\, \bar{q}\right)\right)}}\right] \end{align} \tag{ A.7 }$$

where

$$\begin{align} g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) & = \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{^{\prime} P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M}+Y^{\left(1\right)} \beta ^{^{\prime}}\sqrt{\gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}} \dfrac{P}{2}\bar{q}^{^{P-1}}} \nonumber \\ & \quad +Y^{\left(2\right)} \beta ^{^{\prime}}\sqrt{ \gamma\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\dfrac{P}{2} \left(\left(\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^{^{P-1}} - \bar{q} ^{^{P-1}}\right)}. \end{align} \tag{ A.8 }$$

and the functions $A(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})$ and $B(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})$ are given by

$$\begin{align} & A\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)\nonumber\\ & \quad = 2\mathbb{E}_{Y^{\left(1\right)}}\left\{\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \ln \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) \sinh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) \tanh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)}{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) }\right\} \nonumber \\ & \qquad -2\mathbb{E}_{Y^{\left(1\right)}} \left\{\dfrac{\mathbb{E}_{Y^{\left(2\right)}} \ln \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)}{\left(\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) \right)^2} \right. \nonumber \\ & \left.\qquad \cdot \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \sinh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \tanh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})}{\left(\mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \right)^2 }\right\}\,, \end{align} \tag{ A.9 }$$

$$\begin{align} B(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) & = 2\mathbb{E}_{Y^{(1)}} \left\{\dfrac{\mathbb{E}_{Y^{(2)}} \sinh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \tanh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) }{\left( \mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\right)^2} \right. \nonumber \\ & \quad \left.\cdot \dfrac{\mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \log \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) }{\left( \mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\right)^2}\right\}\nonumber \\ &\quad -2\mathbb{E}_{Y^{(1)}} \left\{\dfrac{\left(\mathbb{E}_{Y^{(2)}} \sinh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\tanh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \right)^2 }{\left( \mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \right)^3}\right. \nonumber \\ & \quad \left. \cdot \dfrac{\mathbb{E}_{Y^{(2)}} \sinh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \log \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) }{\left( \mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \right)^3}\right\}, \end{align} \tag{ A.10 }$$

respectively. Similarly, we can expand $\bar{n}$ as

$$\begin{equation} \bar{n} = \bar{n}^{^{\prime}} + \left(\theta-1\right) C\left( \bar{n}^{^{\prime}},\, \bar{q},\tilde q_2\left(\bar{n}^{^{\prime}},\, \bar{q}\right)\right) \end{equation} \tag{ A.11 }$$

where

$$\begin{align} & C(\bar{n}^{^{\prime}}, \tilde q_2(\bar{n}^{^{\prime}},\bar{q}),\bar{q})\nonumber\\ &\quad = \mathbb{E}_{Y^{(1)}} \left[ \dfrac{\mathbb{E}_{Y^{(2)}} \sinh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\log \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})}{\mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) }\right] \nonumber\\ & \qquad -\mathbb{E}_{Y^{(1)}} \left[ \dfrac{\mathbb{E}_{Y^{(2)}} \sinh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) \log \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})}{\left(\mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\right)^2 }\right]. \end{align} \tag{ A.12 }$$

In the limit θ → 1, equation (A.4) implies that $\mathcal F^{(P)}_\textrm{1RSB}(\bar{n}^{^{\prime}},\bar{q}_2(\bar{q}), \bar{q}|\theta)\lt \mathcal F^{(P)}_\textrm{RS}(\bar{n}^{^{\prime}},\bar{q})$ when we have that $\partial_\theta \mathcal{F}_\textrm{1RSB} (\bar{n}^{^{\prime}},\bar{q}_2(\bar{q}), \bar{q}|\theta)\vert_{\theta = 1}\gt0$. Next, we evaluate

$$\begin{align} K\left(\bar{n}^{^{\prime}}, \tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)& := \partial_\theta \left(-\beta^{^{\prime}} \mathcal F^{\left(P\right)}_{\mathrm{1RSB}}\left(\bar{n}^{^{\prime}},\bar{q}_2\left(\bar{q}\right), \bar{q}|\theta\right) \right)\vert_{\theta = 1} \nonumber\\ & = -\dfrac{{\beta^{^{\prime}}}^2 \gamma \sqrt{\left(1+\rho_P\right)}^2}{4\left(1+\rho\right)^P} \left(P-1\right)\left[\left(\tilde{q}_2\left(\bar{q}\right)\right)^P- \bar{q}^P\right] -\dfrac{{\beta^{^{\prime}}}^2 \gamma \sqrt{\left(1+\rho_P\right)}^2}{4\left(1+\rho\right)^P} P\left[\left(\tilde{q}_2\left(\bar{q}\right)\right)^{P-1} - \bar{q}^{P-1}\right] \nonumber \\ &\quad - \mathbb{E} \ln \cosh \left( \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{^{\prime} P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M}+Y \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^P } \bar{q}^{^{P-1}}}\right) \nonumber \\ &\quad + \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) \log \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}) }{\mathbb{E}_{Y^{(2)}} \cosh g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}) }\right]. \end{align} \tag{ A.13 }$$

In order to determine the sign of the expression above, it is useful to note that $K(\bar{n}^{^{\prime}},\bar{q}, \bar{q}) = 0$, as the last term of (A.8) vanishes so, the last two addends in (A.13) elide each other. This is as expected, as $\bar{q}$ is an extremum of the RS free-energy, which is retrieved for θ = 1. Next, we study $K(\bar{n}^{^{\prime}},x, \bar{q})$ as a function of $x \in [0, \bar{q}]$ and locate its extrema. These are found from

$$\begin{align} \partial_{x} K\left(\bar{n}^{^{\prime}}, x, \bar{q}\right)& = -\dfrac{{\beta^{^{\prime}}}^2 \gamma \sqrt{\left(1+\rho_P\right)}^2 \left(P-1\right) P x^{P-2}}{4\left(1+\rho\right)^P}\nonumber \\ &\quad \cdot\left[x- \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \sinh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) \tanh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) }{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) }\right]\right] = 0 \end{align} \tag{ A.14 }$$

as

$$\begin{align} x = \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \sinh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) \tanh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) }{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) }\right] \equiv \tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right). \end{align} \tag{ A.15 }$$

where the last equality follows from algebraic manipulations of trigonometric functions.

Given that $K(\bar{n}^{^{\prime}},x,\bar{q})$ vanishes for $x = \bar{q}$, if the extremum $x = \tilde q_2(\bar{n}^{^{\prime}},\bar{q})$ is global in the domain considered, we must have that $K(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\gt0$ if $x = \tilde q_2(\bar{n}^{^{\prime}},\bar{q})$ is a maximum and $K(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}),\bar q)\lt0$ if $x = \tilde q_2(\bar{n}^{^{\prime}},\bar{q})$ is a minimum. Evaluating

$$\begin{align} \partial^2_{x} K\left(\bar{n}^{^{\prime}},x, \bar{q}\right)\vert_{x = \tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right)}& = -\dfrac{{\beta^{^{\prime}}}^2 \gamma \sqrt{\left(1+\rho_P\right)}^2 \left(P-1\right) P}{4 \left(1+\rho\right)^P} \left(\tilde{q}_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^{P-2} \nonumber \\ &\quad \cdot\left[ 1-\dfrac{{\beta^{^{\prime}}}^2 \gamma \sqrt{\left(1+\rho_P\right)}^2}{2\left(1+\rho\right)^P}\left(P-1\right) P \left(\tilde{q}_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^{P-2}\right. \nonumber \\ &\quad \left. \times\ \mathbb{E}_{Y^{\left(1\right)}}\left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \textrm{sech}^3 g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) }{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)}\right]\right]. \end{align} \tag{ A.16 }$$

we have that if the expression in the square brackets is positive, namely if the parameter $\gamma {\beta^{^{\prime}}}^2$ satisfies

$$\begin{align} \dfrac{{\beta^{^{\prime}}}^2 \gamma \sqrt{\left(1+\rho_P\right)}^2}{2\left(1+\rho\right)^P}\left(P-1\right) P \left(\tilde{q}_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^{P-2} \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \textrm{sech}^3 g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) }{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)}\right] < 1, \end{align} \tag{ A.17 }$$

$K(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\lt0$ and $\mathcal F^{(P)}_\textrm{1RSB}(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q}|\theta)-\mathcal F^{(P)}_\textrm{RS}(\bar{n}^{^{\prime}},\bar{q}) = -(\theta-1)K(\bar{n}^{^{\prime}},\tilde q_2$ $(\bar{n}^{^{\prime}},\bar{q}), \bar{q})/\beta^{^{\prime}}\lt0$, hence the RS theory is unstable. We plot the line (A.17) in figure 7, for different values of M, in the parameter space $(\gamma, \tilde\beta)$, where $\tilde\beta = \beta^{^{\prime}}/(1+\rho)^{P/2}$, together with the critical lines delimiting the retrieval region (top plots) and the spin-glass region (bottom plots), within the RS and the RSB theory, respectively. We note that the above recovers the expression for the RS instability line found in DAM models [39] in the limits r → 0 or $M \to \infty$, where the values of ρ_P and ρ vanish.

Zoom In Zoom Out Reset image size

In this section we derive the RS instability line for the supervised DAM model. We start by providing the expression for the quenched free-energy in RS assumption as derived in [14] for the standard Dense Hopfield Model.

$$\begin{align} -\beta^{^{\prime}} \mathcal F^{\left(P\right)}_{RS}\left(\bar{n}^{^{\prime}},\bar{q}\right) & = \ln 2 - \dfrac{{\beta^{^{\prime}}}^2}{2}\left(P-1\right) \left(1+\rho\right)^{P/2} \bar{n}^{^{\prime} P} + \dfrac{{\beta^{^{\prime}}}^2 \gamma}{4}\left(1-\bar{q}^P\right)- \dfrac{{\beta^{^{\prime}}}^2 \gamma P }{4} \bar{q}^{P-1} \left(1-\bar{q}\right)\nonumber \\ & \quad + \mathbb{E}_\xi \mathbb{E}_{\left(\eta \vert \xi\right)} \mathbb{E}_z \ln \cosh \left( \beta^{^{\prime}} \dfrac{P}{2} \left(1+\rho\right)^{P/2-1} \hat{\eta}_M \bar{n}^{P-1} + \beta^{^{\prime}} z \sqrt{\gamma \dfrac{P}{2} \bar{q}^{P-1}}\right) \end{align} \tag{ A.18 }$$

with $\beta^{^{\prime}}: = 2\beta/P!$ and where $\bar{q}$ and $\bar{n}$ satisfy the self-consistency equations

$$\begin{align} &\bar{n}^{^{\prime}} = \dfrac{1}{1+\rho}\mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_z \left\{\tanh \left[ \beta^{^{\prime}} \dfrac{P}{2} \left(1+\rho\right)^{P/2-1} \hat{\eta}_M \bar{n}^{^{\prime} P-1} + \beta^{^{\prime}} z \sqrt{\gamma \dfrac{P}{2} \bar{q}^{P-1}}\right]\hat{\eta}_M\right\}, \nonumber \\ &\bar{q} = \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{\left(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1\right)} \mathbb{E}_z \left\{\tanh^2 \left[ \beta^{^{\prime}} \dfrac{P}{2} \left(1+\rho\right)^{P/2-1} \hat{\eta}_M \bar{n}^{^{\prime} P-1} + \beta^{^{\prime}} z \sqrt{\gamma \dfrac{P}{2} \bar{q}^{P-1}}\right]\right\}. \end{align} \tag{ A.19 }$$

We also recall that the quenched free-energy within the 1RSB approximation, $\mathcal F^{(P)}_\textrm{1RSB}(\bar{n},\bar{q}_1, \bar{q}_2 \vert \theta)$, is as given in (3.10), where the order parameters $\bar{q}_2$, $\bar{q}_1$ and $\bar{n}$ satisfy the set of self-consistency equations provided in (3.11). We note that for θ = 1, $\bar{q}_1 = \bar{q}$, $\bar{n} = \bar{n}^{^{\prime}}$ and the 1RSB expression for the quenched free-energy reduces to the RS one. Now, we expand, to the leading order in $\theta-1$, the 1RSB quenched free-energy around its RS expression, as shown in (A.4). Since the self-consistency equations also depend on θ, we need to expand them too. We can write $\bar{q}_1$ as in (A.5), with $A(\bar{n}^{^{\prime}},\bar{q}_2(\bar{n}^{^{\prime}},\bar{q}),\bar{q})$ given in (A.9), and $\bar{q}_2$ as given in (A.6), where $\tilde q_2(\bar{n}^{^{\prime}},\bar{q})$ is the solution of (A.7) and $B(\bar{n}^{^{\prime}},\bar{q}_2(\bar{n}^{^{\prime}},\bar{q}),\bar{q})$ is as given in (A.10) (we recall that now the expression of $g(\bar{n}^{^{\prime}},\bar{q}_2(\bar{n}^{^{\prime}},\bar{q}),\bar{q} )$ is as given in (3.12)). With these expressions in hand, we can now compute the derivative of $\mathcal F^{(P)}_{\mathrm{1RSB}}$ w.r.t. θ when θ = 1, as needed in (A.4)

$$\begin{align} K\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)&: = \partial_\theta \left(-\beta^{^{\prime}} \mathcal F^{\left(P\right)}_{\mathrm{1RSB}}\left(\bar{n}^{^{\prime}},\bar{q}_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}|\theta\right) \right)\vert_{\theta = 1} \nonumber\\ & = -\dfrac{{\beta^{^{\prime}}}^2 \gamma }{4} \left(P-1\right)\left[\left(\tilde{q}_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^P- \bar{q}^P\right] -\dfrac{{\beta^{^{\prime}}}^2 \gamma }{4} P\left[\left(\tilde{q}_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^{P-1} - \bar{q}^{P-1}\right] \nonumber \\ &\quad - \mathbb{E}_\xi \mathbb{E}_{\left(\eta \vert \xi\right)} \mathbb{E}_Y \ln \cosh \left( \beta ^{^{\prime}}\dfrac{P}{2}\bar{n}^{^{\prime} P-1}\left(1+\rho\right)^{P/2-1} \hat{\eta}_{\tiny M}+Y \beta ^{^{\prime}}\sqrt{\gamma \dfrac{P}{2}\bar{q}^{^{P-1}}}\right) \nonumber \\ &\quad + \mathbb{E}_\xi \mathbb{E}_{\left(\eta \vert \xi\right)} \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) \log \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) }{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) }\right]. \end{align} \tag{ A.20 }$$

Again, we have that $K(\bar{n}^{^{\prime}},\bar{q}, \bar{q}) = 0$ (as $\bar{q}$ is the extremum of the RS free-energy). Next, we inspect the sign of $K(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})$. To this purpose, we study $K(\bar{n}^{^{\prime}},x,\bar{q})$ for $x \in [0, \bar{q}]$ and locate its extrema, which are found from

$$\begin{align} \partial_{x} K\left(\bar{n}^{^{\prime}}, x, \bar{q}\right) = & -\dfrac{{\beta^{^{\prime}}}^2 \gamma \left(P-1\right) P x^{P-2}}{4\left(1+\rho\right)^P}\left[x- \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \sinh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) \tanh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) }{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\bar{q}, x\right) }\right]\right] = 0 \end{align} \tag{ A.21 }$$

as

$$\begin{align} x = \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \cosh {g\left(\bar{n}^{^{\prime}},x, \bar{q}\right)} \tanh^2 {g\left(\bar{n}^{^{\prime}},x, \bar{q}\right)}}{\mathbb{E}_{Y^{\left(2\right)}} \cosh {g\left(\bar{n}^{^{\prime}},x, \bar{q}\right)}}\right] \equiv \tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right) \end{align} \tag{ A.22 }$$

where the last equality follows from (A.7). Under the assumption that the extremum $x = \tilde q_2(\bar{n}^{^{\prime}},\bar{q})$ is global in the domain considered, we have that $K(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\gt0$ if $x = \tilde q_2(\bar{n}^{^{\prime}},\bar{q})$ is a maximum and $K(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\lt0$ if it is a minimum. In particular, if

$$\begin{align} \partial^2_{x} K\left(\bar{n}^{^{\prime}},x,\bar{q}\right)\vert_{x = \tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right)} = & -\dfrac{{\beta^{^{\prime}}}^2 \gamma \left(P-1\right) P \left(\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^{P-2}}{4}\nonumber \\ &\cdot\left[ 1-\dfrac{{\beta^{^{\prime}}}^2 \gamma }{2}\left(P-1\right) P \left(\tilde{q}_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^{P-2} \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \textrm{sech}^3 g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) }{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)}\right]\right] \end{align} \tag{ A.23 }$$

is positive, $K(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})\lt0$ and $\mathcal F^{(P)}_\textrm{1RSB}(\bar{n}^{^{\prime}},\bar{q}_2(\bar{n}^{^{\prime}},\bar{q}),\bar{q}|\theta)\lt \mathcal F^{(P)}_\textrm{RS}(\bar{n}^{^{\prime}},\bar{q})$. This happens when the expression in the curly brackets of the equation above is negative, i.e. when the parameter $\gamma {\beta^{^{\prime}}}^2$ satisfies the inequality

$$\begin{equation} \dfrac{{\beta^{^{\prime}}}^2 \gamma }{2}\left(P-1\right) P \left(\tilde{q}_2\left(\bar{n}^{^{\prime}},\bar{q}\right)\right)^{P-2} \mathbb{E}_{Y^{\left(1\right)}} \left[ \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \textrm{sech}^3 g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right) }{\mathbb{E}_{Y^{\left(2\right)}} \cosh g\left(\bar{n}^{^{\prime}},\tilde q_2\left(\bar{n}^{^{\prime}},\bar{q}\right), \bar{q}\right)}\right] >1. \end{equation} \tag{ A.24 }$$

We note that (A.24) is functionally identical to the expression found in [39] for the RS instability line of the standard DAM model, the only difference being encoded in the term $g(\bar{n}^{^{\prime}},\tilde q_2(\bar{n}^{^{\prime}},\bar{q}), \bar{q})$, which is here defined differently, as it reflects the supervised protocol. In particular, in the limit of r → 0 or $M\to \infty$, where ρ and ρ_P vanish, (A.24) retrieves the RS instability line of standard DAM models, as obtained in [39].

Defining the shorthand $\mathcal{Z}_2(J,t) = \mathcal{Z}_2^{(P)}(\boldsymbol{\eta}^1,\boldsymbol{\lambda}, \boldsymbol{Y};J,t)$, we start by computing the derivative of equation (2.29) w.r.t. t

$$\begin{align} -\beta^{^{\prime}}\partial_t \mathcal{F}^{\left(P\right)}_{N}\left(J,t\right) & = \dfrac{1}{N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}}\partial_t\left[ \ln \left[\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \mathcal Z_2^\theta\left(J, t\right) \right]^{1/\theta} \right]\nonumber\\& = \dfrac{1}{N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}}\left[ \dfrac{\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}}\partial_t \mathcal Z_2\left(J, t\right)^{\theta-1}\partial_t \mathcal Z_2\left(J, t\right)}{\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \mathcal Z_2\left(J, t\right)^\theta} \right] \nonumber\\ & = \dfrac{1}{N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}}\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}}\left[\mathcal{W}_{2,t} \dfrac{\partial_t \mathcal Z_2\left(J, t\right)}{\mathcal Z_2\left(J, t\right)} \right] \end{align} \tag{ B.1 }$$

where $\mathbb{E}_0 = \mathbb{E}_{\boldsymbol{\xi}^1}\mathbb{E}_{(\boldsymbol{\eta}^1|\boldsymbol{\xi}^1)}\mathbb{E}_{\boldsymbol{\lambda}}$ and $\mathcal{W}_{2,t}$ is defined in (2.34). Recalling the definition of $\mathcal Z_2(J, t)$ in (2.23) we have:

$$\begin{align} \partial_t \mathcal Z_2(J, t)& = \displaystyle\sum\limits_{\boldsymbol{\sigma}}^{}\mathcal{B}_{N,2}^{(P)}(\boldsymbol{\sigma} ;J, t ) \Bigg[\left(\dfrac{\beta ^{^{\prime}}N}{2M}\left(\dfrac{\mathcal{R}}{r^2} \right)^{P/2}\displaystyle\sum\limits_{a = 1}^{M}n_{1,a}^{^P}(\boldsymbol{\sigma})\right) - \psi N \displaystyle\sum\limits_{a = 1}^{M} n_{1,a}(\boldsymbol{\sigma}) \nonumber\\ & \quad +\dfrac{1}{2}\dfrac{\beta ^{^{\prime}}P!\sqrt{\left(1+\rho_P\right)}\sqrt{K}}{2\sqrt{t} (1+\rho)^{P/2} \,N^{P-1}}\displaystyle\sum\limits_{{i_1 \lt \cdots \lt i_P}}^{N,\cdots,N}\lambda_{i_1,\ldots, i_P}\sigma_{_{i_1}}\cdots\sigma_{_{i_{_{P}}}} -\dfrac{1}{2\sqrt{1-t}}\displaystyle\sum\limits_{b = 1}^{2}\left(A_b\displaystyle\sum\limits_{i = 1}^{N}Y_i^{(b)}\sigma_i\right) \Bigg]. \end{align} \tag{ B.2 }$$

where, in order to lighten the notation we have set $\mathcal{B}^{(P)}_2(\boldsymbol{\sigma}|\boldsymbol{\eta}^1,\boldsymbol{\lambda},\boldsymbol{Y};J,t) = \mathcal{B}_2(\boldsymbol{\sigma};J,t)$. Inserting this expression in (B.1) and using the definition (2.25) for the average of a single replica of the system over the generalised Boltzmann factor $\mathcal{B}_2(\boldsymbol{\sigma};J,t)$, we obtain

$$\begin{align} & -\beta^{^{\prime}}\partial_t \mathcal{F}^{\left(P\right)}_{N}\left(J,t\right)\nonumber\\&\quad = \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}} \mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \left\{\dfrac{\beta ^{^{\prime}}}{2M}\left(\dfrac{\mathcal{R}}{r^2} \right)^{P/2}\mathcal{W}_{2,t} \omega_t \left(\displaystyle\sum\limits_{a = 1}^{M}n_{1,a}^{^P}\left(\boldsymbol{\sigma}\right)\right) - \psi \mathcal{W}_{2,t} \omega_t\left(\displaystyle\sum\limits_{a = 1}^{M} n_{1,a}\left(\boldsymbol{\sigma}\right) \right)\right\} \nonumber \\ & \quad\quad +\dfrac{1}{2}\dfrac{\beta^{^{\prime}} P!\sqrt{\left(1+\rho_P\right)} \sqrt{K}}{2 \sqrt{t} \left(1+\rho\right)^{P/2} N^{P}}\displaystyle\sum\limits_{{i_1 \lt \cdots \lt i_P}}^{N,\cdots,N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}} \mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \left\{\mathcal{W}_{2,t}\dfrac{1}{\mathcal Z_2\left(J,t\right)} \sum_{\boldsymbol{\sigma}} \left(\lambda_{i_1,\ldots,i_P}\sigma_{_{i_1}}\cdots\sigma_{_{i_{_{P}}}} \right)\mathcal{B}_{N,2}^{\left(P\right)}\left(\boldsymbol{\sigma} ;J, t \right) \right\} \nonumber \\ & \quad\quad -\dfrac{1}{2N\sqrt{1-t}}\displaystyle\sum\limits_{b = 1}^{2} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}} \mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \left\{\mathcal{W}_{2,t}\dfrac{1}{\mathcal Z_2\left(J,t\right)} \sum_{\boldsymbol{\sigma}} \left(A_b\displaystyle\sum\limits_{i = 1}^{N}Y_i^{\left(b\right)}\sigma_i\right)\mathcal{B}_{N,2}^{\left(P\right)}\left(\boldsymbol{\sigma} ;J, t \right)\right\}. \end{align} \tag{ B.3 }$$

Next, denoting the combined average over the quenched disorder and the Boltzmann distribution as

$$\begin{equation} \langle \cdot \rangle = \mathbb{E}_0\mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}}\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}}\left[\mathcal{W}_{2,t}\omega_t\left(\cdot\right)\right] \end{equation} \tag{ B.4 }$$

and applying the Stein's lemma (2.74) to the standard Gaussian variables $Y_i^{(b)}$ and $\lambda^{\mu}_{i_1,\ldots,i_P}$, we get

$$\begin{align} -\beta^{^{\prime}}\partial_t \mathcal{F}^{\left(P\right)}_{N} & = \dfrac{\beta ^{^{\prime}}}{2M}\left(\dfrac{\mathcal{R}}{r^2} \right)^{P/2}\displaystyle\sum\limits_{a = 1}^{M} \langle n_{1,a}^P\left(\boldsymbol{\sigma}\right) \rangle_t - \psi \displaystyle\sum\limits_{a = 1}^{M} \langle n_{1,a}\left(\boldsymbol{\sigma}\right) \rangle_t \nonumber \\ & \quad +\ {\dfrac{{\beta^{^{\prime}}}^2 K \sqrt{\left(1+\rho_P\right)}^2}{8 \left(1+\rho\right)^{P} \,N^{2P-1}}}\displaystyle\sum\limits_{i_{_1},\cdots, i_{_{P}}}^{N,\cdots,N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}} \mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \left(\mathcal{W}_{2,t} \omega_t\left(1\right) + \left(\theta-1\right) \mathcal{W}_{2,t} \omega_t^2\left( \sigma_{_{i_1}}\cdots\sigma_{_{i_{_{P}}}}\right)\right) \nonumber \\ & \quad -\ {\dfrac{{\beta^{^{\prime}}}^2P! K\sqrt{\left(1+\rho_P\right)}^2}{8 \left(1+\rho\right)^{P} \,N^{2P-1}}}\displaystyle\sum\limits_{i_{_1}\lt \cdots \lt i_{_{P}}}^{N,\cdots,N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}} \theta \left[\mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \left(\mathcal{W}_{2,t} \omega_t\left( \sigma_{_{i_1}}\cdots\sigma_{_{i_{_{P}}}}\right)\right)\right]^2 \nonumber \\ & \quad -\ \dfrac{\left(A_1\right)^2}{2}\displaystyle\sum\limits_{i = 1}^{N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{\left(1\right)}} \mathbb{E}_{\boldsymbol{Y}^{\left(2\right)}} \left(1 + \left(\theta-1\right) \mathcal{W}_{2,t} \omega_t^2\left(\sigma_i\right)\right)-\dfrac{(A_1)^2}{2}\theta\displaystyle\sum\limits_{i = 1}^{N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{(1)}} \left[ \mathcal{W}_{2,t} \omega_t(\sigma_i)\right]^2 \nonumber \\ & \quad -\ \dfrac{(A_2)^2}{2}\displaystyle\sum\limits_{i = 1}^{N} \mathbb{E}_0 \mathbb{E}_{\boldsymbol{Y}^{(1)}} \mathbb{E}_{\boldsymbol{Y}^{(2)}} \left(1 + (\theta-1) \mathcal{W}_{2,t} \omega_t^2(\sigma_i)\right). \end{align} \tag{ B.5 }$$

Finally, we use the definitions (2.32) and (2.33) and we arrive at the compact expression:

$$\begin{align} -\beta^{^{\prime}}\partial_t \mathcal{F}^{\left(P\right)}_{N} & = \dfrac{\beta ^{^{\prime}}}{2M}\left(\dfrac{\mathcal{R}}{r^2} \right)^{P/2}\displaystyle\sum\limits_{a = 1}^{M} \langle n_{1,a}^P\left(\boldsymbol{\sigma}\right) \rangle_t - \psi \displaystyle\sum\limits_{a = 1}^{M} \langle n_{1,a}\left(\boldsymbol{\sigma}\right) \rangle_t \nonumber \\ & \quad +\ {\dfrac{{\beta^{^{\prime}}}^2\sqrt{\left(1+\rho_P\right)}^2}{4 \left(1+\rho\right)^{P} }} \dfrac{P!K}{2N^{P-1}}\left[1+ \left(\theta-1\right) \langle q_{12}^{P}(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}) \rangle_{t,2} - \theta \langle q_{12}^{P}(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}) \rangle_{t,1}\right]\nonumber \\ & \quad -\ \dfrac{A_1^2}{2} \left[ 1+ \left(\theta-1\right) \langle q_{12}(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}) \rangle_{t,2} - \theta \langle q_{12}\left(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}\right) \rangle_{t,1} \right] \nonumber \\ & \quad -\ \dfrac{A_2^2}{2} \left[ 1+\left(\theta-1\right) \langle q_{12}(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}) \rangle_{t,2} \right]. \end{align} \tag{ B.6 }$$

Next, we take the thermodynamic limit $N\to \infty$, using assumption 2. By manipulating the expression for the moments $\langle q_{12}^P(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)})\rangle_{t,a}$, with $a = 1,2$, using Newton's binomial theorem

$$\begin{align} &\langle q_{12}(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)})^P \rangle_{t,a}\nonumber\\ &\quad = \left\langle \left[(q_{12}(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)})-\bar{q}_a+\bar{q}_a\right]^P \right\rangle_{t,a} = \displaystyle\sum\limits_{k = 0}^{P} \begin{pmatrix}P\\k\end{pmatrix} \langle (q_{12}(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)})-\bar{q}_a)^k \rangle_{t,a} \bar{q}_a^{P-k} \nonumber\\ & \quad = P\,\bar{q}_a^{P-1}\langle q_{12}(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)}) \rangle_{t,a} + \displaystyle\sum\limits_{k = 2}^{P} \begin{pmatrix}P\\k\end{pmatrix} \langle (q_{12}(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)})-\bar{q}_a)^k \rangle_{t,a} \bar{q}_a^{P-k}+(1-P) \bar{q}_a^{P}, \end{align}$$

we obtain, in the thermodynamic limit, under assumption 2,

$$\begin{equation} \begin{array}{lll} \langle q_{12}\left(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}\right)^P \rangle_{t,a} & = & P\,\bar{q}_a^{P-1}\langle q_{12}\left(\boldsymbol{\sigma}^{\left(1\right)},\boldsymbol{\sigma}^{\left(2\right)}\right) \rangle_{t,a} -\left(P-1\right) \bar{q}_a^{P} \end{array} \end{equation} \tag{ B.7 }$$

for $a = 1,2$. Finally, we insert the above in (B.6) and we choose the constants $A_1,\ A_2, \ \psi$ in such a way that the terms dependent on $\langle q_{12}(\boldsymbol{\sigma}^{(1)},\boldsymbol{\sigma}^{(2)}) \rangle_{t,a}$ cancel out,

$$\begin{equation} \begin{array}{lll} &\psi = \beta ^{^{\prime}}\dfrac{P}{2M}\left(1+\rho\right)^{P/2}\bar{n}^{P-1}, \\ \\ &A_1^{\tiny \mbox{2}} = \beta ^{^{\prime}}\dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\dfrac{KP!}{2N^{P-1}}\bar{q}_1^{^{P-1}}, \\ \\ &A_2^{\tiny \mbox{2}} = \beta ^{^{\prime}}\dfrac{P}{2}\dfrac{\sqrt{\left(1+\rho_P\right)}^2}{\left(1+\rho\right)^{P}}\dfrac{KP!}{2N^{P-1}}\left(\bar{q}_2^{^{P-1}}-\bar{q}_1^{^{P-1}}\right). \end{array} \end{equation} \tag{ B.8 }$$

This makes $\textrm{d}\mathcal{F}^{(P)}/\textrm{d}t$ t-independent and leads to the thesis (2.40).

Let us start from the self consistency equations in the $M\to\infty$ limit introduced in corollary 1, we recognize that as $\tilde\beta\to\infty$, we have $\bar{q}_2\to 1$, therefore in order to perform the limit we will introduce the reparametrization

$$\begin{equation} \begin{array}{lll} \bar{q}_2 = 1-\dfrac{\delta\bar{q}_2}{\tilde\beta}&\mathrm{as}&\tilde\beta\to\infty. \end{array} \end{equation} \tag{ B.9 }$$

It is now useful to insert an additional term $\tilde\beta y$ in the expression of $g(\beta, \gamma, \boldsymbol{Y})$ in (2.76), which now reads as

$$\begin{align} g\left(\beta, \gamma, \boldsymbol{Y}\right) & = \tilde\beta \dfrac{P}{2}\bar{m}^{^{P-1}} +\tilde\beta Y^{\left(2\right)} \sqrt{ \gamma\dfrac{P}{2}\sqrt{\left(1+\rho_P\right)}^2 \left(\bar{q}_2^{^{P-1}} - \bar{q}_1 ^{^{P-1}}\right)} \nonumber \\ & \quad +\ \tilde\beta Y^{\left(1\right)}\sqrt{\rho\left(\dfrac{P}{2}\bar{m}^{^{P-1}}\right)^2+\gamma \dfrac{P}{2}\sqrt{\left(1+\rho_P\right)}^2\,\bar{q}_1^{^{P-1}}\;}\; + \tilde\beta y.\end{align} \tag{ B.10 }$$

Using this new parameter y, we can recast the equation for $\bar{q}_2$ as a derivative of the magnetization

$$\begin{equation} \begin{array}{lll} \dfrac{\partial\bar{m}}{\partial y} = \delta\bar{q}_2-D\Delta\bar{q}\Longrightarrow\delta\bar{q}_2 = \dfrac{\partial\bar{m}}{\partial y}+D\Delta\bar{q} \end{array} \end{equation} \tag{ B.11 }$$

where we have used $\Delta\bar{q} = \bar{q}_2-\bar{q}_1$ and, as $\tilde\beta\to\infty$, $\tilde\beta\theta\to D \in\mathbb{R}$. Thus, in the zero temperature limit the last three equations in equation (2.70) become

$$\begin{align} \bar{m} &\to\ \mathbb{E}_{Y^{\left(1\right)}} \left\lbrace \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \left[\mathrm{sign}{\left[g\left(\boldsymbol{Y},\bar{m},\Delta\bar{q}\right)\right]}\:e^{D|g\left(\boldsymbol{Y},\bar{m},\Delta\bar{q}\right)|}\,\right]}{\mathbb{E}_{Y^{\left(2\right)}} \left[e^{D|g\left(\boldsymbol{Y},\bar{m},\Delta\bar{q}\right)|}\right]}\right\rbrace \notag\\[6pt] \Delta\bar{q} &\to\ 1-\mathbb{E}_{Y^{\left(1\right)}} \left\lbrace \dfrac{\mathbb{E}_{Y^{\left(2\right)}} \left[\mathrm{sign}{\left[g\left(\boldsymbol{Y},\bar{m},\Delta\bar{q}\right)\right]}\:e^{D|g\left(\boldsymbol{Y},\bar{m},\Delta\bar{q}\right)|}\,\right]}{\mathbb{E}_{Y^{\left(2\right)}} \left[e^{D|g\left(\boldsymbol{Y},\bar{m},\Delta\bar{q}\right)|}\right]}\right\rbrace^2 \notag\\[6pt] \bar{q}_2 &\to\ 1. \end{align} \tag{ B.12 }$$

Now, if we suppose $\Delta \bar{q}\ll 1$ the (B.10) reduces to

$$\begin{equation} g\left(\boldsymbol{Y},\bar{m},\Delta\bar{q}\right) = C_0+C_1 Y^{\left(1\right)}+C_2 Y^{\left(2\right)}+\mathcal{O}\left(\Delta\bar{q}\right) \end{equation} \tag{ B.13 }$$

where

$$\begin{align} C_0 & = \dfrac{P}{2}\bar{m}^{P-1}\,, C_1 = \sqrt{\rho\left(\dfrac{P}{2}\bar{m}^{P-1}\right)^2+\gamma\sqrt{\left(1+\rho_P\right)}^2\dfrac{P }{2}}\,,\nonumber\\ C_2 & = \sqrt{\gamma\sqrt{\left(1+\rho_P\right)}^2\dfrac{P\left(P-1\right)}{2}\Delta\bar{q}}\,. \end{align} \tag{ B.14 }$$

Performing the integral over $Y^{(2)}$ we reach equations (2.77) and (2.78).