Learning sample-aware threshold for semi-supervised learning

Abstract

Pseudo-labeling methods are popular in semi-supervised learning (SSL). Their performance heavily relies on a proper threshold to generate hard labels for unlabeled data. To this end, most existing studies resort to a manually pre-specified function to adjust the threshold, which, however, requires prior knowledge and suffers from the scalability issue. In this paper, we propose a novel method named Meta-Threshold, which learns a dynamic confidence threshold for each unlabeled instance and does not require extra hyperparameters except a learning rate. Specifically, the instance-level confidence threshold is automatically learned by an extra network in a meta-learning manner. Considering limited labeled data as meta-data, the overall training objective of the classifier network and the meta-net can be formulated as a nested optimization problem that can be solved by a bi-level optimization scheme. Furthermore, by replacing the indicator function existed in the pseudo-labeling with a surrogate function, we theoretically provide the convergence of our training procedure, while discussing the training complexity and proposing a strategy to reduce its time cost. Extensive experiments and analyses demonstrate the effectiveness of our method on both typical and imbalanced SSL tasks.

Appendix A: Theoretical proof of our method

1.1 A.1 Proofs of smoothness

Given a small amount of meta dataset with n samples $\{(\textbf{x}_1^l, \textbf{y}_1^l),...,(\textbf{x}_n^l, \textbf{y}_n^l)\}$ and another unlabeled data $\{\textbf{x}_1,...,\textbf{x}_{(\mu \times n)}\}$ with size of $\mu \times n$. By replacing the indicator function with the approximate function, the meta loss is $L_\textrm{meta}(\textbf{w}^*({\Theta })) = \frac{1}{n} \sum \nolimits _{i=1}^n H(\textbf{y}_i^l, f(\textbf{x}_i^l; \textbf{w}^*({\Theta })))$ and the training loss is

$$\begin{aligned} L_{train}(\textbf{w},\Theta ) = \frac{1}{n\mu } \sum \nolimits _{i=1}^{n\mu } \mathbbm {1}(\max (f(\mathcal {A}^w(\textbf{x}_i); \textbf{w})) > \mathcal {V}_i(\textbf{w}, \Theta )) \cdot H(\hat{\textbf{y}}_i, f(\mathcal {A}^s(\textbf{x}_i); \textbf{w})), \end{aligned}$$

(A1)

where $\mathcal {S}_i(\textbf{w}, \Theta ) = \mathcal {S}( \max (f(\mathcal {A}^w(\textbf{x}_i; \textbf{w}))) - \mathcal {V}_i(\textbf{w}, \Theta ))$.

Firstly, we recall the update equation of the parameters of TGN as follows:

$$\begin{aligned} \Theta ^{(t+1)} = \Theta ^{(t)} - \psi \frac{1}{n} \sum \nolimits _{i=1}^{n} \nabla _{\Theta } H(\textbf{y}_i^l, f(\textbf{x}_i^l; {\hat{{{\textbf{w}}}}}^{(t)}(\Theta ))). \end{aligned}$$

(A2)

To be concise, we formulate $H(\textbf{y}_i^l, f(\textbf{x}_i^l; {\hat{{{\textbf{w}}}}}^{(t)}(\Theta )))$ as $H_i^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ))$. Then, the computation of backpropagation for the above equation can be written as

$$\begin{aligned} \begin{aligned}&\frac{1}{n} \sum \nolimits _{i=1}^{n} \nabla _{\Theta } H_i^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta )) \Big |_{\Theta ^{(t)}} = \frac{1}{n} \sum \nolimits _{i=1}^{n} \frac{\partial H_i^\textrm{meta}(\hat{{{\textbf{w}}}})}{\partial {\hat{{\textbf{w}}}}} \Big |_{\hat{{\textbf{w}}}^{(t)}} \sum \nolimits _{j=1}^{n\mu } \frac{\partial \hat{{{\textbf{w}}}}^{(t)}(\Theta )}{\partial \mathcal {S}_j(\textbf{w}^{(t)}; \Theta )} \, \frac{\partial \mathcal {S}_j(\textbf{w}^{(t)}; \Theta )}{\partial \mathcal {V}_j(\textbf{w}^{(t)}; \Theta )} \, \frac{\partial \mathcal {V}_j(\textbf{w}^{(t)}; \Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}} \\ =&\frac{-\alpha }{n^2\mu } \sum \nolimits _{i=1}^{n} \frac{\partial H_i^\textrm{meta}(\hat{{{\textbf{w}}}})}{\partial {\hat{{\textbf{w}}}}} \Big |_{\hat{{\textbf{w}}}^{(t)}} \sum \nolimits _{j=1}^{n\mu } \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \, \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \Big |_{\textbf{w}^{(t)}} \frac{\partial \mathcal {V}_j(\textbf{w}^{(t)}; \Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}} \\ =&\frac{- \alpha }{n\mu } \sum \nolimits _{j=1}^{n\mu } \bigg {(} \frac{1}{n} \sum \nolimits _{i=1}^{n} \frac{\partial H_i^\textrm{meta}(\hat{{{\textbf{w}}}})}{\partial {\hat{{\textbf{w}}}}} \Big |_{\hat{{\textbf{w}}}^{(t)}}^T \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \, \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \Big |_{\textbf{w}^{(t)}} \bigg {)} \frac{\partial \mathcal {V}_j(\textbf{w}^{(t)}; \Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}}. \end{aligned} \end{aligned}$$

(A3)

Let $G_{ij} = \frac{\partial H_i^\textrm{meta}(\hat{{{\textbf{w}}}})}{\partial {\hat{{\textbf{w}}}}} \big |_{\hat{{\textbf{w}}}^{(t)}}^T \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \, \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \big |_{\textbf{w}^{(t)}}$ and substitute $G_{ij}$ into Eq.  (A3), then

$$\begin{aligned} \Theta ^{(t+1)} = \Theta ^{(t)} + \frac{\alpha \psi }{n\mu } \sum \nolimits _{j=1}^{n\mu } \bigg {(} \frac{1}{n} \sum \nolimits _{i=1}^{n} G_{ij} \bigg {)} \frac{\partial \mathcal {V}_j(\textbf{w}^{(t)}; \Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}}. \end{aligned}$$

(A4)

Proof

The gradient of $\Theta$ w.r.t. meta loss can be formulated as:

$$\begin{aligned} \begin{aligned}&\nabla _{\Theta } H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta )) \Big |_{\Theta ^{(t)}} = -\frac{\alpha }{n\mu } \sum \nolimits _{j=1}^{n\mu } \bigg {(} \frac{\partial H^\textrm{meta}(\hat{{{\textbf{w}}}})}{\partial {\hat{{\textbf{w}}}}} \Big |_{\hat{{\textbf{w}}}^{(t)}}^T \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \, \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \Big |_{\textbf{w}^{(t)}} \bigg {)} \frac{\partial \mathcal {V}_j(\textbf{w}^{(t)}; \Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}}. \end{aligned} \end{aligned}$$

(A5)

Let $\mathcal {V}_j(\Theta ) = \mathcal {V}_j(\textbf{w}^{(t)}; \Theta )$ and introduce $G_{ij}$ which is defined in Eq.  (A4). Taking the gradient of $\Theta$ on both side of Eq.  (A5), we attain

$$\begin{aligned} \begin{aligned} \nabla _{\Theta ^2}^2 H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta )) \Big |_{\Theta ^{(t)}} = -\frac{\alpha }{n\mu } \sum \nolimits _{j=1}^{n\mu } \bigg {[} \frac{\partial }{\partial \Theta } (G_{ij}) \Big |_{\Theta ^{(t)}} \frac{\partial \mathcal {V}_j(\Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}} + (G_{ij}) \frac{\partial ^2 \mathcal {V}_j(\Theta )}{\partial ^2 \Theta } \Big |_{\Theta ^{(t)}} \bigg {]}. \end{aligned} \end{aligned}$$

(A6)

The first term in Eq.  (A6) right hand side can be summarized as

$$\begin{aligned} \begin{aligned}&\left\| \frac{\partial }{\partial \Theta } (G_{ij}) \Big |_{\Theta ^{(t)}} \frac{\partial \mathcal {V}_j(\Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}} \right\| \le \, \delta \left\| \frac{\partial }{\partial \hat{{{\textbf{w}}}}} \bigg {(} \frac{\partial H^\textrm{meta} (\hat{{{\textbf{w}}}})}{\partial \Theta } \Big |_{\Theta ^{(t)}} \bigg {)} \Big |_{\hat{{{\textbf{w}}}}^{(t)}}^T \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \, \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \Big |_{\textbf{w}^{(t)}} \right\| \\ =&\, \delta \left\| \frac{\partial }{\partial \hat{{{\textbf{w}}}}} \bigg {(} \frac{\partial H^\textrm{meta} (\hat{{{\textbf{w}}}})}{\partial \hat{{{\textbf{w}}}}} \Big |_{\hat{{{\textbf{w}}}}^{(t)}} \, \frac{-\alpha }{n\mu } \sum \nolimits _{k=1}^{n\mu } \frac{\partial \ell _{\textbf{x}_k}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \Big |_{\textbf{w}^{(t)}} \frac{\partial \mathcal {V}_k(\Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}} \bigg {)} \Big |_{\hat{{{\textbf{w}}}}^{(t)}}^T \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \, \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \Big |_{\textbf{w}^{(t)}} \right\| \\ =&\, \delta \left\| \bigg {(} \frac{\partial ^2 H^\textrm{meta} (\hat{{{\textbf{w}}}})}{\partial \hat{{{\textbf{w}}}}^2} \Big |_{\hat{{{\textbf{w}}}}^{(t)}} \, \frac{-\alpha }{n\mu } \sum \nolimits _{k=1}^{n\mu } \frac{\partial \ell _{\textbf{x}_k}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \Big |_{\textbf{w}^{(t)}} \frac{\partial \mathcal {V}_k(\Theta )}{\partial \Theta } \Big |_{\Theta ^{(t)}} \bigg {)} \Big |_{\hat{{{\textbf{w}}}}^{(t)}}^T \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \, \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \Big |_{\textbf{w}^{(t)}} \right\| \\ \le&\, \alpha L \delta ^2 \phi ^2 \zeta ^2, \end{aligned} \end{aligned}$$

(A7)

since ${\left\| \frac{\partial H(\hat{{{\textbf{w}}}})}{\partial {\hat{{\textbf{w}}}}} \big |_{\hat{{\textbf{w}}}^{(t)}}^T \right\| \le \rho , \left\| \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \right\| \le \phi , \left\| \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \big |_{\textbf{w}^{(t)}} \right\| \le \zeta , \left\| \frac{\partial ^2 \mathcal {V}_j(\Theta )}{\partial ^2 \Theta } \Big |_{\Theta ^{(t)}} \right\| \le \mathcal {B}}$.

The second term in Eq.  (A6) right hand side can be summarized as

$$\begin{aligned} \begin{aligned} \left\| (G_{ij}) \frac{\partial ^2 \mathcal {V}_j(\Theta )}{\partial ^2 \Theta } \Big |_{\Theta ^{(t)}} \right\| = \left\| \frac{\partial H^\textrm{meta}(\hat{{{\textbf{w}}}})}{\partial {\hat{{\textbf{w}}}}} \big |_{\hat{{\textbf{w}}}^{(t)}}^T \frac{\partial \ell _{\textbf{x}_j}(\mathcal {S}_j(\textbf{w}))}{\partial \mathcal {S}_j(\textbf{w})} \, \frac{\partial \mathcal {S}_j(\textbf{w})}{\partial \textbf{w}} \big |_{\textbf{w}^{(t)}} \frac{\partial ^2 \mathcal {V}_j(\Theta )}{\partial ^2 \Theta } \Big |_{\Theta ^{(t)}} \right\| \le \, \rho \phi \zeta \mathcal {B}. \end{aligned} \end{aligned}$$

(A8)

Combining the results in Eq.  (A7) and Eq.  (A8), we have $\left\| \nabla _{\Theta ^2}^2 H_i^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta )) \Big |_{\Theta ^{(t)}} \right\| \le \phi \zeta (\alpha L \delta ^2 \phi \zeta + \rho \mathcal {B}).$ Define ${{{\hat{L}}}} = \phi \zeta (\alpha L \delta ^2 \phi \zeta + \rho \mathcal {B})$, based on the Lagrange mean value theorem, we have:

$$\begin{aligned} \left\| \nabla {L_\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta _1))} - \nabla {L_\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta _2))} \right\| \le {{{\hat{L}}}} \left\| \Theta _1 - \Theta _2 \right\| , \, \text {for all} \, \Theta _1, \Theta _2, \end{aligned}$$

(A9)

where $\nabla {L_\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta _1))} = \nabla _\Theta {L_\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ))}\Big |_{\Theta _1}$. $\square$

1.2 A.2 Proofs of convergence

Proof

The update of parameters $\Theta$ in t-th iteration can be written as $\Theta ^{(t+1)} = \Theta ^{(t)} - \psi \frac{1}{n} \sum \nolimits _{i=1}^n \nabla _\Theta H_i^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta )) \Big |_{\Theta ^{(t)}}.$ Training with a mini-batch of meat-data $\textrm{B}_t$ that is uniformly drawn from the data set, we rewrite the equation above as:

$$\begin{aligned} \Theta ^{(t+1)} = \Theta ^{(t)} - \psi _t \Big [ \sum \nolimits _{i=1}^n \nabla _\Theta H_i^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta )) + \varepsilon ^{(t)} \Big ], \end{aligned}$$

(A10)

where $\varepsilon ^{(t)} = \nabla _\Theta H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta )) \Big |_{\textrm{B}_t} - \nabla _\Theta H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ))$. Note that the expectation of $\varepsilon ^{(t)}$ obeys $\mathbbm {E}[\varepsilon ^{(t)}]=0$ and its variance is finite. Consider that

$$\begin{aligned} \begin{aligned}&H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t+1)}(\Theta ^{(t+1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \\ =&\, \underbrace{H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t+1)}(\Theta ^{(t+1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t +1 )}))}_\textrm{term 1} + \underbrace{H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t+1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)}))}_\textrm{term 2}. \end{aligned} \end{aligned}$$

(A11)

For $\textrm{term 1}$, by Lipschitz smoothness of the meta loss function for $\Theta$, we have

$$\begin{aligned} \begin{aligned}&H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t+1)}(\Theta ^{(t+1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t+1)})) \\ \le&\, \left\langle \nabla H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t+1)})), \hat{{{\textbf{w}}}}^{(t+1)}(\Theta ^{(t+1)}) - \hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t+1)}) \right\rangle + \frac{L}{2} \left\| \hat{{{\textbf{w}}}}^{(t+1)}(\Theta ^{(t+1)}) - \hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t+1)}) \right\| _2^2. \end{aligned} \end{aligned}$$

According to Eq. (6) (8) (A1), then we have

$$\begin{aligned} \left\| H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t+1)}(\Theta ^{(t+1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t +1 )})) \right\| \le \alpha _t \rho ^2 + \frac{1}{2} L \alpha _t \rho ^2 = \alpha \rho ^2 (1 + \frac{\alpha _t L}{2}) \end{aligned}$$

(A12)

since ${ \left\| \frac{\partial H_j(\textbf{w})}{\partial \textbf{w}} \big |_{\hat{{{\textbf{w}}}}^{(t)}}\right\| \le \rho , \left\| \frac{\partial H_i^\textrm{meta}(\textbf{w})}{\partial \hat{{{\textbf{w}}}}} \big |_{\hat{{{\textbf{w}}}}^{(t)}}^T \right\| \le \rho }$.

For $\mathrm term 2$, considering Lipschitz continuity of $\nabla H_\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ))$ demonstrated in Lemma 1, we can obtain the following:

$$\begin{aligned} \begin{aligned}&H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t+1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \\ \le&\, \left\langle H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t+1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})), \Theta ^{(t+1)} - \Theta ^{(t)} \right\rangle + \frac{L}{2} \left\| \Theta ^{(t+1)} - \Theta ^{(t)} \right\| _2^2 \\ =&-(\psi _t - \frac{L \psi _t^2}{2}) \left\| \nabla H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \right\| _2^2 + \frac{L \psi _t^2}{2} \left\| \varepsilon ^{(t)} \right\| _2^2 - (\psi _t - L\psi _t^2) \left\langle H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})), \varepsilon ^{(t)} \right\rangle . \end{aligned} \end{aligned}$$

(A13)

Summing up the Eq.  (A12)  (A13), the Eq.  (A11) can be summarized as

$$\begin{aligned} \begin{aligned}&H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t+1)}(\Theta ^{(t+1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \\ \le&\, \alpha \rho ^2 (1 + \frac{\alpha _t L}{2}) - (\psi _t - \frac{L \psi _t^2}{2}) \left\| \nabla H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \right\| _2^2 + \frac{L \psi _t^2}{2} \left\| \varepsilon ^{(t)} \right\| _2^2 - (\psi _t - L\psi _t^2) \left\langle H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})), \varepsilon ^{(t)} \right\rangle . \end{aligned} \end{aligned}$$

Rearranging the terms, we can obtain

$$\begin{aligned} \begin{aligned}&(\psi _t - \frac{L \psi _t^2}{2}) \left\| \nabla H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \right\| _2^2 \\ \le&\, \alpha \rho ^2 (1 + \frac{\alpha _t L}{2}) - (\psi _t - \frac{L \psi _t^2}{2}) \left\| \nabla H^\textrm{meta} \big ( \hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)}) \big ) \right\| _2^2 + \frac{L \psi _t^2}{2} \left\| \varepsilon ^{(t)} \right\| _2^2 \\&- (\psi _t - L\psi _t^2) \left\langle H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})), \varepsilon ^{(t)} \right\rangle . \end{aligned} \end{aligned}$$

Summing up the above inequalities and rearranging the terms, we can obtain

$$\begin{aligned} \begin{aligned}&\sum \nolimits _{t=1}^T (\psi _t - \frac{L \psi _t^2}{2}) \left\| \nabla H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \right\| _2^2\\ \le&\, H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)})) - H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) + \\&\quad \quad \quad \sum \nolimits _{t=1}^T \alpha \rho ^2 (1 + \frac{\alpha _t L}{2}) - \sum \nolimits _{t=1}^T (\psi _t - L\psi _t^2) \left\langle H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})), \varepsilon ^{(t)} \right\rangle \, + \, \frac{L}{2} \sum \nolimits _{t=1}^T \left\| \varepsilon ^{(t)} \right\| _2^2 \\ \le&\, H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)})) + \sum \nolimits _{t=1}^T \alpha \rho ^2 (1 + \frac{\alpha _t L}{2})\\&- \sum \nolimits _{t=1}^T (\psi _t - L\psi _t^2) \left\langle H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})), \varepsilon ^{(t)} \right\rangle + \frac{L}{2} \sum \nolimits _{t=1}^T \left\| \varepsilon ^{(t)} \right\| _2^2. \end{aligned} \end{aligned}$$

(A14)

We take the expectations w.r.t. $\varepsilon ^{(N)}$ on both size of Eq.  (A14), then we have:

$$\begin{aligned} \begin{aligned}&\sum \nolimits _{t=1}^T (\psi _t - \frac{L \psi _t^2}{2}) \mathop {\mathbbm {E}}\nolimits _{\varepsilon ^{(N)}} \left\| \nabla H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \right\| _2^2 \le H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)}))\\&+ \sum \nolimits _{t=1}^T \alpha \rho ^2 (1 + \frac{\alpha _t L}{2}) + \frac{L\sigma ^2}{2} \sum \nolimits _{t=1}^T \psi _t^2, \end{aligned} \end{aligned}$$

since ${\mathop {\mathbbm {E}}\nolimits _{\varepsilon ^{(N)}} \left\langle H^\textrm{meta}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})), \varepsilon ^{(t)} \right\rangle =0}$ and $\mathbbm {\left\| \varepsilon ^{(t)} \right\| _2^2} \le \sigma ^2$, where $\sigma ^2$ represents the variance of $\varepsilon ^{(t)}$. Eventually, we deduce that

$$\begin{aligned} \mathop {\min }\nolimits _{t}&\, \mathbbm {E} \Big [ \left\| \nabla H^{\textrm{meta}}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \right\| _2^2 \Big ] \le \frac{\sum \nolimits _{t=1}^T (\psi _t - \frac{L \psi _t^2}{2}) \mathop {\mathbbm {E}}\nolimits _{\varepsilon ^{(N)}} \left\| \nabla H^{\textrm{meta}}(\hat{{{\textbf{w}}}}^{(t)}(\Theta ^{(t)})) \right\| _2^2 }{\sum \nolimits _{t=1}^T (\psi _t - \frac{L \psi _t^2}{2})} \\ \le&\, \frac{1}{\sum \nolimits _{t=1}^T (2\psi _t - L\psi _t^2)} \Big [ 2H^{\textrm{meta}}(\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)})) + \sum \nolimits _{t=1}^T \alpha \rho ^2 (2 + \alpha _t L) + L\sigma ^2 \sum \nolimits _{t=1}^T \psi _t^2 \Big ] \\ \le&\, \frac{1}{\sum \nolimits _{t=1}^T \psi _t} \Big [ 2H^{\textrm{meta}} (\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)})) + \sum \nolimits _{t=1}^T \alpha \rho ^2 (2 + \alpha _t L) + L\sigma ^2 \sum \nolimits _{t=1}^T \psi _t^2 \Big ] \\ \le&\, \frac{1}{T \psi _t} \Big [ 2H^{\textrm{meta}}(\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)})) + \alpha _1 \rho ^2 T (2 + L) + L\sigma ^2 \sum \nolimits _{t=1}^T \psi _t^2 \Big ] \\ \le&\, \frac{2H^{\textrm{meta}}(\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)}))}{T} \, \frac{1}{\psi _t} + \frac{2 \alpha _1 \rho ^2 (2 + L)}{\psi _t} + L\sigma ^2 \psi _t \\ =&\, \frac{H^{\textrm{meta}}(\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)}))}{T} \max \{L, \frac{\sigma \sqrt{T}}{\textrm{c}}\} + \min \{1, \frac{k}{T}\}\max \{L, \frac{\sigma \sqrt{T}}{\textrm{c}}\}\rho ^2(2+L) + L\sigma ^2 \min \{\frac{1}{L}, \frac{\textrm{c}}{\sigma \sqrt{T}}\} \\ \le&\, \frac{\sigma H^{\textrm{meta}}(\hat{{{\textbf{w}}}}^{(1)}(\Theta ^{(1)}))}{{\textrm{c}} \sqrt{T}} + \frac{k\sigma \rho ^2(2+L)}{{\textrm{c}} \sqrt{T}} + \frac{L\sigma {\textrm{c}}}{\sqrt{T}} = \mathcal {O}(\frac{1}{\sqrt{T}}). \end{aligned}$$

Therefore, we can conclude that under some mild conditions, our algorithm can always achieve $\min _{0 \le t \le T} \mathbbm {E} \Big [ \left\| \nabla H^\textrm{meta}(\Theta ^{(t)}) \right\| _2^2 \Big ] \le \mathcal {O}(\frac{1}{\sqrt{T}})$ in T steps. $\square$