Factor-$\sqrt{2}$ Acceleration of Accelerated Gradient Methods

Abstract

The optimized gradient method (OGM) provides a factor-$\sqrt{2}$ speedup upon Nesterov’s celebrated accelerated gradient method in the convex (but non-strongly convex) setup. However, this improved acceleration mechanism has not been well understood; prior analyses of OGM relied on a computer-assisted proof methodology, so the proofs were opaque for humans despite being verifiable and correct. In this work, we present a new analysis of OGM based on a Lyapunov function and linear coupling. These analyses are developed and presented without the assistance of computers and are understandable by humans. Furthermore, we generalize OGM’s acceleration mechanism and obtain a factor-$\sqrt{2}$ speedup in other setups: acceleration with a simpler rational stepsize, the strongly convex setup, and the mirror descent setup.

Appendices

Appendix A: Method Reference

For reference, we restate all aforementioned methods. In all methods, we assume that f is L-smooth function, $\{ \theta _k \}_{k=0}^\infty$ and $\{ \varphi _k \}_{k=0}^\infty$ are the sequences of positive scalars, and $x_0=y_0=z_0$.

OGM One form of OGM is

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}\nabla {f(x_{k})}\\ x_{k+1}&= y_{k+1} + \frac{\theta _{k}-1}{\theta _{k+1}}(y_{k+1}- y_k) + \frac{\theta _k}{\theta _{k+1}}(y_{k+1} - x_{k}) \end{aligned}$$

and an equivalent form with z-iterates is

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}\nabla {f(x_{k})}\\ z_{k+1}&= z_{k} - \frac{2\theta _k}{L}\nabla {f(x_{k})}\\ x_{k+1}&= \left( 1-\frac{1}{\theta _{k+1}}\right) y_{k+1} + \frac{1}{\theta _{k+1}}z_{k+1} \end{aligned}$$

for $k=0,1,\dots$. The last-step modification on the secondary sequence can be written as

$$\begin{aligned} \tilde{x}_{k+1}&= y_{k+1} + \frac{\theta _{k}-1}{\varphi _{k+1}}(y_{k+1}- y_k) + \frac{\theta _k}{\varphi _{k+1}}(y_{k+1} - x_{k})\\&= \left( 1-\frac{1}{\varphi _{k+1}}\right) y_{k+1} + \frac{1}{\varphi _{k+1}}z_{k+1} \end{aligned}$$

where $k=0,1,\dots$.

OGM-simple OGM-simple is a simpler variant of OGM with $\theta _k = \frac{k+2}{2}$ and $\varphi _k = \frac{k+1 +\frac{1}{\sqrt{2}}}{\sqrt{2}}$. One form of OGM-simple is

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}\nabla {f(x_{k})}\\ x_{k+1}&= y_{k+1} + \frac{k}{k+3}(y_{k+1}- y_k) + \frac{k+2}{k+3}(y_{k+1} - x_{k}) \end{aligned}$$

and an equivalent form with z-iterates is

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}\nabla {f(x_{k})}\\ z_{k+1}&= z_{k} - \frac{k+2}{L}\nabla {f(x_{k})}\\ x_{k+1}&= \left( 1-\frac{2}{{k+3}} \right) y_{k+1} + \frac{2}{k+3}z_{k+1} \end{aligned}$$

for $k=0,1,\dots$. The last-step modification on secondary sequence is written as

$$\begin{aligned} \tilde{x}_{k+1}&= y_{k+1} + \frac{k}{\sqrt{2}(k+2) + 1}(y_{k+1} - y_{k}) + \frac{k+2}{\sqrt{2}(k+2) + 1}(y_{k+1} - x_k) \end{aligned}$$

where $k=0,1,\dots$.

SC-OGM Here, we assume that f is a $\mu$-strongly convex function, condition number of f is $\kappa = L/\mu$, and $\gamma = \frac{\sqrt{8\kappa +1}+3}{2\kappa -2}$. SC-OGM is written as

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}\nabla {f(x_{k})}\\ x_{k+1}&= y_{k+1} + \frac{1}{2\gamma + 1}(y_{k+1}-y_k) + \frac{1}{2\gamma + 1}(y_{k+1} - x_{k}) \end{aligned}$$

for $k=0,1,\dots$.

LC-OGM LC-OGM (Linear Coupling OGM) is defined as

$$\begin{aligned} y_{k+1}&= x_k-L^{-1}Q^{-1}\nabla f(x_k)\\ z_{k+1}&= \mathop {{{\,\mathrm{arg\,min}\,}}}\limits _{y\in {\mathbb {R}}^n}\left\{ V_{z_k}(y)+\langle \alpha _{k+1}\nabla f(x_k),y-x_k\rangle \right\} \\ x_{k+1}&= (1-\tau _{k+1}) y_{k+1} + \tau _{k+1} z_{k+1} \end{aligned}$$

for $k=0,1,\dots$, where $V_z(y)$ is a Bregman divergence, $\{\alpha _k\}_{k=1}^\infty$ and $\{\tau _k\}_{k=1}^\infty$ are nonnegative sequences defined as $\alpha _1 = \frac{2}{L}$, $0 \le \alpha _{k+1}^2L -2\alpha _{k+1} \le \alpha _k^2L$, $\tau _k = \frac{2}{\alpha _{k+1}L}$, and Q is a positive definite matrix defining $\left\Vert x\right\Vert ^2 = x^T Q x$.

For last step modification, we define positive sequences $\{\tilde{\alpha }_k\}_{k=1}^\infty$ and $\{\tilde{\tau }_k\}_{k=1}^\infty$ as $\alpha _1 = \frac{1}{L}$, $0 \le \tilde{\alpha }_{k+1}^2L - \tilde{\alpha _{k+1}} \le \frac{1}{2}\alpha _k^2L$, and $\tilde{\tau }_k = \frac{1}{\tilde{\alpha }_{k+1}L}$, and also define

$$\begin{aligned} \tilde{x}_k = (1-\tilde{\tau }_k)y_k + \tilde{\tau }_k z_k \end{aligned}$$

for $k=1,2,\dots$.

Unification of AGM and OGM Using LC-OGM, we can unify AGM and OGM as

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}\nabla {f(x_{k})} \\ z_{k+1}&= z_{k} - \frac{2t\theta _k}{L}\nabla {f(x_{k})}\\ x_{k+1}&= \left( 1-\frac{1}{\theta _{k+1}}\right) y_{k+1} + \frac{1}{\theta _{k+1}}z_{k+1}. \end{aligned}$$

for $k=0,1,\dots$. This is equivalent to

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}\nabla {f(x_{k})} \\ x_{k+1}&= y_{k+1} + \frac{\theta _{k}-1}{\theta _{k+1}}(y_{k+1}- y_k)+(2t-1)\frac{\theta _k}{\theta _{k+1}}(y_{k+1} - x_{k}). \end{aligned}$$

LC-SC-OGM LC-SC-OGM (Linear coupling strongly convex OGM) is

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}Q^{-1}\nabla {f(x_{k})}\\ z_{k+1}&= \frac{1}{1+\gamma }\left( z_k + \gamma x_{k} -\frac{\gamma }{\mu }Q^{-1}\nabla f(x_{k})\right) \\ x_{k+1}&= \tau z_{k+1} + (1-\tau ) y_{k+1}, \end{aligned}$$

for $k=0,1,\dots$, where Q is a positive definite matrix.

Appendix B: Co-coercivity Inequality in General Norm

Lemma 7

Let f be a closed convex proper function. Then,

$$\begin{aligned} 0 \le f(x) + f^* (u) - \langle x, u \rangle \end{aligned}$$

and

$$\begin{aligned}{} & {} \inf _{x} \{f(x) + f^*(u) - \langle x, u \rangle \} = 0\\{} & {} \inf _{u} \{f(x) + f^*(u) - \langle x, u \rangle \} = 0. \end{aligned}$$

Proof

By the definition of the conjugate function,

$$\begin{aligned} -f^*(u) = \inf _x \left\{ f(x) - \langle x, u \rangle \right\} \end{aligned}$$

and

$$\begin{aligned} \inf _{x} \{f(x) + f^*(u) - \langle x, u \rangle \} = 0. \end{aligned}$$

Therefore,

$$\begin{aligned} 0 \le f(x) + f^* (u) - \langle x, u \rangle \quad \forall x. \end{aligned}$$

The statement with u follows from the same argument and the fact that $f^{**} = f$. $\square$

Lemma 8

Consider a norm $\Vert \cdot \Vert$ and its dual norm $\Vert \cdot \Vert _*$. Then,

$$\begin{aligned} 0 \le \frac{1}{2}\left\Vert x\right\Vert ^2 + \frac{1}{2} \left\Vert u\right\Vert _*^2 - \langle x, u \rangle \end{aligned}$$

and

$$\begin{aligned}{} & {} \inf _{x \in {\mathbb {R}}^n} \left\{ \frac{1}{2}\left\Vert x\right\Vert ^2 + \frac{1}{2} \left\Vert u\right\Vert _*^2 - \langle x, u \rangle \right\} =0\\{} & {} \inf _{u \in {\mathbb {R}}^n} \left\{ \frac{1}{2}\left\Vert x\right\Vert ^2 + \frac{1}{2} \left\Vert u\right\Vert _*^2 - \langle x, u \rangle \right\} = 0. \end{aligned}$$

Proof

This follows from Lemma 7 with $f(x) = \frac{1}{2}\left\Vert x\right\Vert ^2$ and $\left( \frac{1}{2}\left\Vert \cdot \right\Vert ^2\right) ^* = \frac{1}{2}\left\Vert \cdot \right\Vert _*^2$. $\square$

Lemma 9

Let

$$\begin{aligned} \texttt{Grad}(x) = \mathop {{{\,\mathrm{arg\,min}\,}}}\limits _{y \in {\mathbb {R}}^n}\left\{ \frac{L}{2}\left\Vert y-x\right\Vert ^2 + \langle \nabla f(x), y-x \rangle \right\} . \end{aligned}$$

Then,

$$\begin{aligned} \langle \nabla f(x), \texttt{Grad}(x) - x \rangle + \frac{L}{2}\left\Vert \texttt{Grad}(x) - x \right\Vert ^2 = -\frac{1}{2L}\left\Vert \nabla f(x)\right\Vert _*^2. \end{aligned}$$

Proof

Let $z= L (\texttt{Grad}(x) - x)$. By the definition of $\texttt{Grad}(x)$ and Lemma 8, we have

$$\begin{aligned} \frac{1}{2L}\left\Vert \nabla f(x)\right\Vert _*^2 +&\frac{L}{2} \left\Vert \texttt{Grad}(x) - x\right\Vert ^2 + \langle \nabla f(x) , \texttt{Grad}(x) - x \rangle \\&= \inf _{z \in {\mathbb {R}}^n} \frac{1}{2L}\left\Vert \nabla f(x)\right\Vert _*^2 + \frac{1}{2L} \left\Vert z\right\Vert ^2 + \frac{1}{L} \langle \nabla f(x), z \rangle \\&= 0. \end{aligned}$$

$\square$

Lemma 10

Let $f: {\mathbb {R}}^n \rightarrow {\mathbb {R}}$ be a differentiable convex function such that

$$\begin{aligned} \left\Vert \nabla f(x) - \nabla f(y)\right\Vert _* \le L \left\Vert x-y\right\Vert \end{aligned}$$

for all $x,y\in {\mathbb {R}}^n$. Then

$$\begin{aligned} f(y) \le f(x) + \langle \nabla f(x), y-x \rangle + \frac{L}{2}\left\Vert y-x\right\Vert ^2. \end{aligned}$$

Proof

Since a differentiable convex function is continuously differentiable [45, Theorem 25.5],

$$\begin{aligned} f(y) - f(x)&= \int _0^1 \langle \nabla f(x + t(y-x)) , y-x \rangle dt\\&= \int _0^1 \langle \nabla f(x + t(y-x)) - \nabla f(x) , y-x \rangle dt + \langle \nabla f(x), y-x \rangle \\&\le \int _0^1 \left\Vert \nabla f(x + t(y-x)) - \nabla f(x)\right\Vert _* \left\Vert y-x\right\Vert dt + \langle \nabla f(x), y-x \rangle \\&\le \int _0^1 t L \left\Vert y-x\right\Vert ^2 dt + \langle \nabla f(x), y-x \rangle = \frac{L}{2} \left\Vert y-x\right\Vert ^2 + \langle \nabla f(x) , y-x \rangle . \end{aligned}$$

$\square$

Lemma 11

(Co-coercivity inequality with general norm) Let $f: {\mathbb {R}}^n \rightarrow {\mathbb {R}}$ be a differentiable convex function such that

$$\begin{aligned} \left\Vert \nabla f(x) - \nabla f(y)\right\Vert _* \le L \left\Vert x-y\right\Vert \end{aligned}$$

for all $x,y\in {\mathbb {R}}^n$. Then

$$\begin{aligned} f(y) \ge f(x) + \langle \nabla f(x), y-x \rangle + \frac{1}{2L}\left\Vert \nabla f(x)- \nabla f(y)\right\Vert _*^2. \end{aligned}$$

Proof

Set $\phi (y) = f(y) - \langle \nabla f(x), y-x\rangle$. Then $x \in {{\,\mathrm{arg\,min}\,}}\phi$. So by Lemma 9,

$$\begin{aligned} \phi (x)&\le \phi (\texttt{Grad}(y))\\&\le \phi (y) + \langle \nabla \phi (y), \texttt{Grad}(y) -y \rangle + \frac{L}{2}\left\Vert \texttt{Grad}(y) - y\right\Vert ^2\\&= \phi (y) - \frac{1}{2L}\left\Vert \nabla \phi (y)\right\Vert _*^2. \end{aligned}$$

Substituting f back in $\phi$ yields the co-coercivity inequality. $\square$

Appendix C: Telescoping Sum Argument

Suppose we established the inequality

$$\begin{aligned} a_i F_i + b_i G_i \le c_i F_{i-1} + d_i G_{i-1} - E_i \end{aligned}$$

for $i=1,2,\dots$, where $E_i, F_i, G_i$ are nonnegative quantities and $a_i$, $b_i$, $c_i$, and $d_i$ are nonnegative scalars. Assume $c_i \le a_{i-1}$ and $d_i \le b_{i-1}$. By summing the inequalities for $i=1, 2, \dots , k$, we obtain

$$\begin{aligned} a_k F_k&\le - b_k G_k - \sum _{i=2}^{k} (a_{i-1} - c_i) F_{i-1} - \sum _{i=2}^k (b_{i-1} - d_i) G_{i-1} - \sum _{i=2}^k E_i + c_1 F_{0} + d_1 G_{0}\\&\le c_1 F_{0} + d_1 G_{0}. \end{aligned}$$

However, note that the

$$\begin{aligned} - b_k G_k - \sum _{i=2}^{k} (a_{i-1} - c_i) F_{i-1} - \sum _{i=2}^k (b_{i-1} - d_i) G_{i-1} - \sum _{i=1}^k E_i \end{aligned}$$

terms are wasted in the analysis. If one has the freedom to do so, it may be good to choose parameters so that

$$\begin{aligned} a_{i-1} = c_i,\, b_{i-1} = d_i \end{aligned}$$

and $E_i = 0$ for $i=1,2,\dots$. Not having wasted terms may be an indication that the analysis is tight.

Appendix D: SC-OGM via Linear Coupling

In this section, we analyze SC-OGM through the linear coupling analysis. We consider the linear coupling form

$$\begin{aligned} y_{k+1}&= x_{k} - \frac{1}{L}Q^{-1}\nabla {f(x_{k})}\\ z_{k+1}&= \frac{1}{1+\gamma }\left( z_k + \gamma x_{k} -\frac{\gamma }{\mu }Q^{-1}\nabla f(x_{k})\right) \\ x_{k+1}&= \tau z_{k+1} + (1-\tau ) y_{k+1}, \end{aligned}$$

where $\tau$ is a coupling coefficient to be determined. As an aside, we can view $z_{k+1}$ as a mirror descent update of the form

$$\begin{aligned} z_{k+1} =\mathop {{{\,\mathrm{arg\,min}\,}}}\limits _{z} \left\{ \frac{1}{2}\left\Vert z-z_k\right\Vert ^2 + \frac{\gamma }{2}\left\Vert z-x_k\right\Vert ^2 +\frac{\gamma }{\mu } \langle \nabla f(x_k), z \rangle \right\} , \end{aligned}$$

which is similar to what was considered in [6].

Lemma 12

Assume (A1), (A2) and (A3). Then,

$$\begin{aligned}&\frac{\gamma }{\mu }\langle \nabla f(x_{k}) , z_{k+1} - x_\star \rangle - \frac{\gamma }{2}\left\Vert x_k - x_\star \right\Vert ^2\\&\quad \le -\frac{\gamma ^2}{2(1+\gamma )\mu ^2}\left\Vert \nabla f(x_k)\right\Vert _*^2+ \frac{1}{2} \left\Vert z_k - x_\star \right\Vert ^2 - \frac{1+\gamma }{2} \left\Vert z_{k+1} - x_\star \right\Vert ^2 \end{aligned}$$

for $k = 0,1, \dots$.

Proof

This proof follows steps similar to that of [6, Lemma 5.4].

From the definition of $z_{k+1}$, we say

$$\begin{aligned} 0=&\left\langle \frac{\partial }{\partial z}\left\{ \frac{1}{2}\left\Vert z-z_k\right\Vert ^2 + \frac{\gamma }{2}\left\Vert z-x_k\right\Vert ^2 +\frac{\gamma }{\mu } \langle \nabla f(x_k), z \rangle \right\} \bigg |_{z_{k+1}}, z_{k+1} - x_\star \right\rangle \\ =&\langle Q(z_{k+1} - z_k), z_{k+1} - x_\star \rangle + \frac{\gamma }{\mu } \langle \nabla f(x_{k}), z_{k+1} - x_\star \rangle + \gamma \langle Q(z_{k+1} - x_k),z_{k+1} - x_\star \rangle \end{aligned}$$

By three point equation,

$$\begin{aligned}&\frac{\gamma }{\mu }\langle \nabla f(x_{k}) ,z_{k+1} - x_\star \rangle + \gamma \left( \frac{1}{2} \left\Vert x_k - z_{k+1}\right\Vert ^2 - \frac{1}{2} \left\Vert x_k - x_\star \right\Vert ^2 \right) \\&\quad = -\frac{1}{2} \left\Vert z_{k}- z_{k+1}\right\Vert ^2 + \frac{1}{2} \left\Vert z_k - x_\star \right\Vert ^2 - \frac{1+\gamma }{2} \left\Vert z_{k+1} - x_\star \right\Vert ^2. \end{aligned}$$

Plugging the definition of $z_{k+1}$,

$$\begin{aligned}&\frac{\gamma }{2} \left\Vert x_k - z_{k+1}\right\Vert ^2 + \frac{1}{2}\left\Vert z_k - z_{k+1}\right\Vert ^2 \\&\quad = \frac{\gamma }{2} \left\Vert \frac{1}{1+\gamma }(x_k - z_k) + \frac{\gamma }{(1+\gamma )\mu } Q^{-1}\nabla f(x_k)\right\Vert ^2 \\&\qquad + \frac{1}{2}\left\Vert -\frac{\gamma }{1+\gamma }(x_k - z_k) + \frac{\gamma }{(1+\gamma )\mu }Q^{-1}\nabla f(x_k)\right\Vert ^2\\&\quad \ge \frac{\gamma ^2}{2(1+\gamma )\mu ^2}\left\Vert \nabla f(x_k)\right\Vert _*^2. \end{aligned}$$

Combining results above, we get

$$\begin{aligned}&\frac{\gamma }{\mu }\langle \nabla f(x_{k}) , z_{k+1} - x_\star \rangle - \frac{\gamma }{2}\left\Vert x_k - x_\star \right\Vert ^2\\&\quad \le -\frac{\gamma ^2}{2(1+\gamma )\mu ^2}\left\Vert \nabla f(x_k)\right\Vert _*^2+ \frac{1}{2} \left\Vert z_k - x_\star \right\Vert ^2 - \frac{1+\gamma }{2} \left\Vert z_{k+1} - x_\star \right\Vert ^2 . \end{aligned}$$

$\square$

Lemma 13

(Coupling lemma in SC-OGM) Assume (A1), (A2) and (A3). Then

$$\begin{aligned}&(1+\gamma )\biggl (f(x_k) - \frac{1}{2L} \left\Vert \nabla f(x_k)\right\Vert _*^2 + \frac{\mu }{2}\left\Vert z_k - x_\star \right\Vert ^2 \biggr ) \\&\quad \le \left( f(x_{k-1}) - \frac{1}{2L}\left\Vert \nabla f(x_{k-1})\right\Vert _*^2 + \frac{\mu }{2}\left\Vert z_{k-1} - x_\star \right\Vert ^2 \right) \end{aligned}$$

holds for $k = 1, 2, \dots$

Proof

We have

$$\begin{aligned}&\gamma \left( f(x_{k}) - f(x_\star ) \right) \\&\quad \le \gamma \langle \nabla f(x_{k}), x_{k}- x_\star \rangle - \frac{\mu \gamma }{2}\left\Vert x_k - x_\star \right\Vert ^2 \\&\quad = \gamma \langle \nabla f(x_{k}), x_{k}- z_{k}\rangle + \gamma \langle \nabla f(x_{k}), z_{k}- x_\star \rangle - \frac{\mu \gamma }{2}\left\Vert x_k - x_\star \right\Vert ^2\\&\quad = \frac{1-\tau }{\tau }\gamma \langle \nabla f(x_k) ,y_k - x_k \rangle + \gamma \langle \nabla f(x_{k}), z_{k}- x_\star \rangle - \frac{\mu \gamma }{2}\left\Vert x_k - x_\star \right\Vert ^2\\&\quad = \frac{1-\tau }{\tau }\gamma \langle \nabla f(x_k) , x_{k-1} - x_{k} - \frac{1}{L}Q^{-1}\nabla f(x_{k-1}) \rangle \\&\qquad + \gamma \langle \nabla f(x_{k}), z_{k}- x_\star \rangle - \frac{\mu \gamma }{2}\left\Vert x_k - x_\star \right\Vert ^2 \\&\quad \le \left( \frac{1-\tau }{\tau }\gamma - 1 \right) \langle \nabla f(x_k) , x_{k-1} - x_{k} - \frac{1}{L}Q^{-1}\nabla f(x_{k-1}) \rangle \\&\qquad +\left( f(x_{k-1}) - f(x_k) - \frac{1}{2L}\left\Vert \nabla f(x_{k-1})\right\Vert _*^2 - \frac{1}{2L}\left\Vert \nabla f(x_k)\right\Vert _*^2\right) \\&\qquad +\gamma \langle \nabla f(x_{k}), z_{k}-z_{k+1} \rangle + \gamma \langle \nabla f(x_{k}), z_{k+1}- x_\star \rangle - \frac{\mu \gamma }{2}\left\Vert x_k - x_\star \right\Vert ^2\\&\quad \le \left( \frac{1-\tau }{\tau }\gamma - 1 \right) \langle \nabla f(x_k) , y_{k} - x_{k}\rangle \\&\qquad +\left( f(x_{k-1}) - f(x_k) - \frac{1}{2L}\left\Vert \nabla f(x_{k-1})\right\Vert _*^2 - \frac{1}{2L}\left\Vert \nabla f(x_k)\right\Vert _*^2\right) \\&\qquad +\gamma \langle \nabla f(x_{k}), z_{k}-z_{k+1} \rangle -\frac{\gamma ^2}{2(1+\gamma )\mu }\left\Vert \nabla f(x_k)\right\Vert _*^2\\&\qquad +\frac{\mu }{2} \left\Vert z_k - x_\star \right\Vert ^2 - \frac{(1+\gamma )\mu }{2} \left\Vert z_{k+1} - x_\star \right\Vert ^2, \end{aligned}$$

where the last inequality is an application of Lemma 12. Note that

$$\begin{aligned} z_k - z_{k+1}&= z_k - \frac{1}{1+\gamma }\left( z_k + \gamma x_k -\frac{\gamma }{\mu }Q^{-1}\nabla f(x_k) \right) \\&= \frac{\gamma }{1+\gamma }(z_k-x_k) + \frac{\gamma }{(1+\gamma )\mu } Q^{-1} \nabla f(x_k) \\&= \frac{\gamma }{1+\gamma }\frac{1-\tau }{\tau }(x_k - y_k) + \frac{\gamma }{(1+\gamma )\mu } Q^{-1} \nabla f(x_k). \end{aligned}$$

To eliminate the $\langle \nabla f(x_k), \cdot \rangle$ term, we choose $\tau$ to satisfy

$$\begin{aligned} \frac{1-\tau }{\tau }\gamma - 1 = \frac{\gamma }{1+\gamma }\frac{1-\tau }{\tau }. \end{aligned}$$

(9)

Plugging this in, the inequality above is

$$\begin{aligned}&\gamma \left( f(x_{k}) - f(x_\star ) \right) \\&\quad \le \left( f(x_{k-1}) - f(x_k) - \frac{1}{2L}\left\Vert \nabla f(x_{k-1})\right\Vert _*^2 - \frac{1}{2L}\left\Vert \nabla f(x_k)\right\Vert _*^2\right) \\&\quad +\frac{\gamma ^2}{2(1+\gamma )\mu }\left\Vert \nabla f(x_k)\right\Vert _*^2+ \frac{\mu }{2} \left\Vert z_k - x_\star \right\Vert ^2 - \frac{(1+\gamma )\mu }{2} \left\Vert z_{k+1} - x_\star \right\Vert ^2. \end{aligned}$$

In order to make the telescoping form such as

$$\begin{aligned}&M_{k}\biggl (f(x_{k}) - B_{k}\left\Vert \nabla f(x_{k})\right\Vert _*^2 + C_k \left\Vert z_{k+1} - x_\star \right\Vert ^2 \biggr ) \\&\quad \le N_{k-1}\left( f(x_{k-1}) - B_{k-1} \left\Vert \nabla f(x_{k-1})\right\Vert _*^2+ C_{k-1} \left\Vert z_{k} - x_\star \right\Vert ^2 \right) , \end{aligned}$$

we chose $B_k = \frac{1}{2L}$ and $C_k = \frac{\mu }{2}$, which leads to the choice of $\gamma$ satisfying

$$\begin{aligned} \frac{2+\gamma }{2L} = \frac{\gamma ^2}{2(1+\gamma )\mu }. \end{aligned}$$

(10)

We get the desired result by plugging (9) and (10) in the above inequality. $\square$

Appendix E: Asymptotic Characterization of $\theta _k$

Theorem 7

Let the positive sequence $\{\theta _k\}_{k=0}^\infty$ satisfy $\theta _0 = 1$ and $\theta _{k+1}^2 - \theta _{k+1} - \theta _{k}^2=0$ for $k = 0,1, \dots$. Then,

$$\begin{aligned} \theta _k = \frac{k+\zeta +1}{2} + \frac{\log k}{4} + o(1). \end{aligned}$$

Proof

Let $\theta _k = \frac{k+2}{2} + c_k\log k$. The proof consists of the following 3 steps:

• 1. If $c_k < \frac{1}{4}$, then $c_{k+1} < \frac{1}{4}$.

• 2. $c_k \rightarrow \frac{1}{4}$ as $k \rightarrow \infty$.

• 3. If $\theta _k = \frac{k+2}{2} + \frac{\log k}{4} + e_k$, then $e_k$ is convergent.

First step If $c_k < \frac{1}{4}$, then $c_{k+1}<\frac{1}{4}$.

For our convenience, let $c_0=0$ with $c_0 \log 0 = 0$. Plugging this in $\theta _{k+1}^2 - \theta _{k+1} - \theta _{k}^2=0$, we have

$$\begin{aligned} \left( \frac{k+2}{2} + c_{k+1} \log (k+1) \right) ^2 = \left( \frac{k+2}{2} + c_{k}\log k \right) ^2 + \frac{1}{4}, \end{aligned}$$

so

$$\begin{aligned} \left( c_{k+1} \log (k+1) - c_k \log k \right) \left( k+2+c_{k+1}\log (k+1) + c_k\log k \right) = \frac{1}{4}. \end{aligned}$$

Assume $c_{k+1}\ge 1/4$. Then

$$\begin{aligned} \frac{1}{4}&= \left( c_{k+1}\log (k+1) - c_k \log k \right) \left( k+2+c_{k+1}\log (k+1) + c_k \log k \right) \\&\ge \frac{1}{4}\log \left( 1+\frac{1}{k}\right) (k+2)\\&>\frac{1}{4}, \end{aligned}$$

which proves the first claim.

Second step $c_k \rightarrow \frac{1}{4}$ as $k \rightarrow \infty$.

Put $d_k = \frac{1}{4}-c_k$, then $0 < d_k \le \frac{1}{4}$.

$$\begin{aligned} \frac{1}{4}&=\left( \frac{1}{4}\log \left( 1+\frac{1}{k}\right) -d_{k+1}\log (k+1) + d_k \log k \right) \\&\quad \left( k+2+\frac{1}{4}\log k(k+1) -d_{k+1}\log (k+1) - d_k \log k \right) \\&\le \left( \frac{1}{4}\log \left( 1+\frac{1}{k}\right) -d_{k+1}\log (k+1) + d_k \log k \right) \left( k+2+\frac{1}{2}\log (k+1) \right) \end{aligned}$$

Therefore

$$\begin{aligned} d_{k+1} \log (k+1) - d_k \log k \le \frac{1}{4} \log \left( 1+\frac{1}{k}\right) - \frac{1}{4}\frac{1}{k+2+\frac{1}{2}\log (k+1)}. \end{aligned}$$

By talyor expansion,

$$\begin{aligned} d_{k+1} \log (k+1) - d_k \log k \le \frac{1}{4} \left( \frac{3+2\log k}{2k^2} + {\mathcal {O}}\left( \frac{1}{k^2}\right) \right) . \end{aligned}$$

So, By summing all the above inequality from 1 to k,

$$\begin{aligned} d_{k+1} \log (k+1) \le C \end{aligned}$$

so $d_{k+1} < \frac{C}{\log (k+1)}$. In conclusion, as $k \rightarrow \infty$, $d_k \rightarrow 0$.

Third step If $\theta _k = \frac{k+2}{2} + \frac{\log k}{4} + e_k$, then, $e_k$ converges.

From the previous claim, we can say that for some sufficiently large k, $|e_k| < \frac{1}{6}\log k$.

$$\begin{aligned} \left( \frac{k+2}{2} + \frac{1}{4}\log (k+1) + e_{k+1} \right) ^2 = \left( \frac{k+2}{2} + \frac{1}{4} \log k + e_{k} \right) ^2 + \frac{1}{4} \end{aligned}$$

Then,

$$\begin{aligned} \frac{1}{4}&=\left( \frac{1}{4}\log \left( 1+\frac{1}{k}\right) +e_{k+1} - e_k \right) \left( k+2+\frac{1}{4}\log k(k+1) + e_{k+1} + e_k \right) \\&\le \left( \frac{1}{4}\log \left( 1+\frac{1}{k}\right) +e_{k+1} - e_k \right) \left( k+2+\frac{5}{6}\log (k+1) \right) . \end{aligned}$$

So,

$$\begin{aligned} e_{k+1} - e_k \ge \frac{1}{4\left( k+2+ \frac{5}{6}\log (k+1) \right) } - \frac{1}{4}\log \left( 1+ \frac{1}{k} \right) = -\frac{\frac{5}{6}\log k + \frac{3}{2}}{k^2} + {\mathcal {O}}\left( \frac{1}{k^2}\right) . \end{aligned}$$

Summing this for $k=1,\dots ,k$, we get that $e_{k+1} > D$ for some constant D. Moreover,

$$\begin{aligned} \frac{1}{4}&=\left( \frac{1}{4}\log \left( 1+\frac{1}{k}\right) +e_{k+1} - e_k \right) \left( k+2+\frac{1}{4}\log k(k+1) + e_{k+1} + e_k \right) \\&\ge \left( \frac{1}{4}\log \left( 1+\frac{1}{k}\right) +e_{k+1} - e_k \right) \left( k+2\right) >\frac{1}{4} + (k+2)(e_{k+1} - e_k), \end{aligned}$$

which indicates that $e_{k+1} < e_k$. Since $\{e_k\}_{k=0}^\infty$ is a decreasing sequence with a lower bound, it converges. $\square$

Proof of equality in Section 2.1 We have

$$\begin{aligned} \frac{L\left\Vert x_0 - x_\star \right\Vert ^2}{2\theta _{k-1}^2}&= \frac{L\left\Vert x_0 - x_\star \right\Vert ^2}{2\left( \frac{k+\zeta }{2} + \frac{\log (k-1)}{4} + o(1)\right) ^2}\\&= \frac{2L\left\Vert x_0 - x_\star \right\Vert ^2}{(k+ \zeta )^2 \left( 1 + \frac{\log (k-1)}{2(k+\zeta )} + o(1/k)\right) ^2}\\&= \frac{2L\left\Vert x_0 - x_\star \right\Vert ^2}{(k+ \zeta )^2} \left( 1 - 2 \frac{\log (k-1)}{2(k+\zeta )} + o(1/k)\right) \\&= \frac{2L\left\Vert x_0 - x_\star \right\Vert ^2}{(k+\zeta )^2} - \frac{2L\left\Vert x_0 - x_\star \right\Vert ^2\log k}{(k+\zeta )^3} + o\left( \frac{1}{k^3}\right) , \end{aligned}$$

which verifies the equality in Sect. 2.1.