A Universal Accelerated Primal–Dual Method for Convex Optimization Problems

Abstract

This work presents a universal accelerated primal–dual method for affinely constrained convex optimization problems. It can handle both Lipschitz and Hölder gradients but does not need to know the smoothness level of the objective function. In line search part, it uses dynamically decreasing parameters and produces approximate Lipschitz constant with moderate magnitude. In addition, based on a suitable discrete Lyapunov function and tight decay estimates of some differential/difference inequalities, a universal optimal mixed-type convergence rate is established. Some numerical tests are provided to confirm the efficiency of the proposed method.

Appendices

Proof of Lemma 3.1

Let us first prove (18). Recall that $i_k$ is the smallest nonnegative integer such that (cf. (15))

$$\begin{aligned} h(x_{k,i_k})-\varDelta _{k,i_k}\le \frac{\delta _{k,i_{k}}}{2}. \end{aligned}$$

If $i_k = 0$, then $M_{k+1} = M_{k,0} /\rho _{_{\!d}}= M_{k}/\rho _{_{\!d}}$. Otherwise (i.e., $i_k\ge 1$), we claim that

$$\begin{aligned} M_{k,i_k}\le \rho _{\!u}\cdot M(\nu ,\delta _{k,i_{k}-1}). \end{aligned}$$

(57)

If this is violated, then $M_{k,i_{k}-1} = M_{k,i_k}/\rho _{\!u}>M(\nu ,\delta _{k,i_{k}-1})$. According to Proposition 2.1, this implies immediately that

$$\begin{aligned} h(x_{k,i_{k}-1})-\varDelta _{k,i_{k}-1}\le \frac{\delta _{k,i_{k}-1}}{2}, \end{aligned}$$

which yields a contradiction and thus verifies (57). Additionally, by (16), we have $\alpha _{k,i_{k}-1}\le \sqrt{\rho _{\! u}}\alpha _{k,i_{k}}$. Therefore, collecting (13,17,57) leads to

$$\begin{aligned} M_{k+1} = M_{k,i_k}/\rho _{_{\!d}}\le \frac{\sqrt{\rho _{\! u}} \rho _{\! u}}{\rho _{_{\!d}}}\cdot M(\nu ,\delta _{k,i_{k}}) =\frac{\sqrt{\rho _{\! u}} \rho _{\! u}}{\rho _{_{\!d}}}\cdot M(\nu ,\delta _{k+1}). \end{aligned}$$

(58)

This means that for all $k\in {\mathbb {N}}$, we have

$$\begin{aligned} M_{k+1}\le \frac{1}{\rho _{_{\!d}}} \max \left\{ M_{k},\,\sqrt{\rho _{\! u}} \rho _{\! u}\cdot M(\nu ,\delta _{k+1})\right\} . \end{aligned}$$

(59)

Since $\delta _{k+1} = \beta _{k+1}/{(k+1)}$ with $\{\beta _k\}_{k\in {\mathbb {N}}}$ being decreasing (cf.(17)), it follows that $\delta _{k+1}\le \delta _{\ell +1}$ and $M(\nu ,\delta _{\ell +1})\le M(\nu ,\delta _{k+1})$ for all $0\le \ell \le k$, which together with (59) indicates the estimate

$$\begin{aligned} \begin{aligned} M_{k+1}\le {}&\frac{1}{\rho _{_{\!d}}} \max \left\{ \frac{1}{\rho _{_{\!d}}}\max \left\{ M_{k-1},\,\sqrt{\rho _{\! u}} \rho _{\! u}\cdot M(\nu ,\delta _{k})\right\} ,\,\sqrt{\rho _{\! u}} \rho _{\! u}\cdot M(\nu ,\delta _{k+1})\right\} \\ \le {}&\frac{1}{\rho _{_{\!d}}}\max \left\{ \frac{M_{k-1}}{\rho _{_{\!d}}},\,\sqrt{\rho _{\! u}} \rho _{\! u}\cdot M(\nu ,\delta _{k+1})\right\} \\ \le {}&\cdots \le \frac{1}{\rho _{_{\!d}}}\max \left\{ \frac{M_{0}}{\rho _{_{\!d}}^{k}},\,\sqrt{\rho _{\! u}} \rho _{\! u}\cdot M(\nu ,\delta _{k+1})\right\} . \end{aligned} \end{aligned}$$

This proves the desired result (18).

Then, let us verify (19). Observing that $M_{k+1} = M_{k,i_k} /\rho _{_{\!d}}= \rho _{\! u}^{i_k}M_{k}/\rho _{_{\!d}}$, we have

$$\begin{aligned} i_k = \log _{\rho _{\!u}}\frac{\rho _{_{\!d}}M_{k+1}}{M_{k}} \quad \Longrightarrow \quad \sum _{j=0}^{k}i_j= \log _{\rho _{\! u}}\frac{\rho _{_{\!d}}^{k+1}M_{k+1}}{M_{0}}. \end{aligned}$$

Invoking the estimate of $M_{k+1}$ gives

$$\begin{aligned} \begin{aligned} \sum _{j=0}^{k}i_j \le {}&\max \left\{ 0, \frac{3}{2}+k\log _{\rho _{\!u}}\!\rho _{_{\!d}}+\log _{\rho _{\! u}}\frac{M(\nu ,\delta _{k+1})}{M_{0}}\right\} \\ \le {}&\frac{3}{2}+k\log _{\rho _{\!u}}\!\rho _{_{\!d}}+\left|{\log _{\rho _{\! u}}\frac{M(\nu ,\delta _{k+1})}{M_{0}}} \right|. \end{aligned} \end{aligned}$$

Since $M(\nu ,\delta _{k+1}) = \delta _{k+1}^{\frac{\nu -1}{\nu +1}}[M_\nu (h)]^{\frac{2}{\nu +1}}$, we obtain (19) and complete the proof of Lemma 3.1.

Derivation of the Reformulation (20)

Observing line 10 of Algorithm 1, we obtain (20d). Given $(x_k,v_k,\lambda _k)$ and $(\gamma _k,\beta _k,M_{k})$, $(y_k,x_{k+1},v_{k+1})$ are nothing but the output of Algorithm 2 with the input $(k,S_k,M_{k,i_k})$, where $S_k=\{x_k,v_k,\lambda _k,\beta _k,\gamma _k\}$. Hence, from lines 3 and 4, we obtain

$$\begin{aligned} \begin{aligned} {}&\frac{y_k-x_k}{\alpha _k} =v_k-y_k,\\ {}&\frac{x_{k+1}-x_k}{\alpha _k} ={} v_{k+1}-x_{k+1}, \end{aligned} \end{aligned}$$

which gives (20a) and (20c). Besides, we have

$$\begin{aligned} v_{k+1} = \mathop { {\mathop {\textrm{argmin}}\,}}\limits _{v\in Q}\left\{ g(v)+ \big \langle \nabla h(y_k)+A^\top \widetilde{\lambda }_k,v\big \rangle +\mu D_\phi (v,y_k)+ \frac{\gamma _k}{\alpha _k}D_\phi (v,v_k) \right\} , \end{aligned}$$

with $\widetilde{\lambda }_k= \lambda _k+\alpha _k/\beta _k(Av_k-b)$. The optimality condition reads as

$$\begin{aligned} \begin{aligned} 0\in {}&\nabla h(y_k)+\partial g(v_{k+1})+N_Q(v_{k+1})+A^\top \widetilde{\lambda }_k\\ {}&\quad +\mu \left[ \nabla \phi (v_{k+1})-\nabla \phi (y_k)\right] +\frac{\gamma _{k}}{\alpha _k}\left[ \nabla \phi (v_{k+1})-\nabla \phi (v_k)\right] . \end{aligned} \end{aligned}$$

After rearranging, we get (20b).

Proof of Lemma 5.1

1.1 The Case $\eta =\theta -1$

The estimate (47) becomes

$$\begin{aligned} \sqrt{\varphi (t)}\frac{y'(t)}{y(t)}+R\frac{y'(t)}{y^{\theta }(t)}\le -\sigma (t). \end{aligned}$$

(60)

Since $y(0) = 1$ and $y'(t)\le 0$, it holds that $0<y(t)\le 1$ for all $t\ge 0$. As $\varphi (t)$ is positive and nondecreasing, we obtain

$$\begin{aligned} \left( \sqrt{\varphi }\ln y\right) ' = \frac{\varphi '}{2\sqrt{\varphi }}\ln y + \sqrt{\varphi } \frac{y'}{y}\le \sqrt{\varphi } \frac{y'}{y}. \end{aligned}$$

Combining this with (60) gives

$$\begin{aligned} \begin{aligned} \left( \sqrt{\varphi (t)}\ln y(t)+\frac{R}{1-\theta }y^{1-\theta }(t)\right) '\le -\sigma (t), \end{aligned} \end{aligned}$$

and integrating over (0, t) leads to

$$\begin{aligned} \sqrt{\varphi (t)}\ln \frac{1}{y(t)}+\frac{R}{\theta -1}\left( y^{1-\theta }(t)-1\right) \ge \int _{0}^{t}\sigma (s)\,\textrm{d}s=\varSigma (t). \end{aligned}$$

(61)

Define

$$\begin{aligned} Y_1(t) := {}\exp \left( -\frac{\varSigma (t)}{2\sqrt{\varphi (t)}}\right) \quad \text {and}\quad Y_2(t) :={} \left( 1+\frac{\theta -1}{2R}\varSigma (t)\right) ^{\frac{1}{1-\theta }}. \end{aligned}$$

(62)

Then one finds that

$$\begin{aligned} \begin{aligned} {}&\sqrt{\varphi (t)}\ln \frac{1}{Y_1(t)}=\frac{1}{2}\varSigma (t),{} & {} Y_1(0) = 1,\\ {}&\frac{R}{\theta -1}\left( Y_2^{1-\theta }(t)-1\right) =\frac{1}{2}\varSigma (t),{} & {} Y_2(0) = 1. \end{aligned} \end{aligned}$$

This also implies

$$\begin{aligned} \sqrt{\varphi (t)}\ln \frac{1}{Y(t)}+\frac{R}{\theta -1}\left( Y^{1-\theta }(t)-1\right) \le \varSigma (t), \end{aligned}$$

(63)

where $Y(t):= Y_1(t)+Y_2(t)$. For fixed $t>0$, the function

$$\begin{aligned} v\longmapsto \sqrt{\varphi (t)}\ln \frac{1}{v}+\frac{R}{\theta -1}\left( v^{1-\theta }-1\right) \end{aligned}$$

is monotonously decreasing in terms of $v\in (0,\infty )$. Collecting (61,63) yields that

$$\begin{aligned} y(t)\le Y(t)=\exp \left( -\frac{\varSigma (t)}{2\sqrt{\varphi (t)}}\right) + \left( 1+\frac{\theta -1}{2R}\varSigma (t)\right) ^{\frac{1}{1-\theta }}. \end{aligned}$$

This completes the proof of Lemma 5.1 with $\eta =\theta -1$.

1.2 The Case $\eta <\theta -1$

The proof is in line with the previous case. With some elementary calculus computations, the estimate (61) now becomes

$$\begin{aligned} G(\varphi (t),y(t))\ge \varSigma (t), \end{aligned}$$

where $G:(0,\infty )\times (0,\infty )\rightarrow \,{{\mathbb {R}}}$ is defined by

$$\begin{aligned} G(w,v): = \frac{\sqrt{w}}{\theta -\eta -1}(v^{\eta +1-\theta }-1)+\frac{R}{\theta -1}(v^{1-\theta }-1), \end{aligned}$$

for all $w,\,v>0$. In addition to $Y_2(t)$ defined in (62), we need

$$\begin{aligned} Y_3(t):= \left( 1+\frac{\theta -\eta -1}{2\sqrt{\varphi (t)}}\varSigma (t)\right) ^{\frac{1}{\eta +1-\theta }}. \end{aligned}$$

Since $G(w,\cdot )$ is monotonously decreasing and

$$\begin{aligned} G(\varphi (t),Y_2(t)+Y_3(t))\le \varSigma (t)\le G(\varphi (t),y(t)), \end{aligned}$$

we obtain

$$\begin{aligned} y(t)\le \left( 1+\frac{\theta -1}{2R}\varSigma (t)\right) ^{\frac{1}{1-\theta }}+ \left( 1+\frac{\theta -\eta -1}{2\sqrt{\varphi (t)}}\varSigma (t)\right) ^{\frac{1}{\eta +1-\theta }}. \end{aligned}$$

This concludes the proof of Lemma 5.1 with $\eta <\theta -1$.