A Mirror Inertial Forward–Reflected–Backward Splitting: Convergence Analysis Beyond Convexity and Lipschitz Smoothness

Abstract

This work investigates a Bregman and inertial extension of the forward–reflected–backward algorithm (Malitsky and Tam in SIAM J Optim 30:1451–1472, 2020) applied to structured nonconvex minimization problems under relative smoothness. To this end, the proposed algorithm hinges on two key features: taking inertial steps in the dual space, and allowing for possibly negative inertial values. The interpretation of relative smoothness as a two-sided weak convexity condition proves beneficial in providing tighter stepsize ranges. Our analysis begins with studying an envelope function associated with the algorithm that takes inertial terms into account through a novel product space formulation. Such construction substantially differs from similar objects in the literature and could offer new insights for extensions of splitting algorithms. Global convergence and rates are obtained by appealing to the Kurdyka–Łojasiewicz property.

Appendices

Proof of Theorem 4.6

Throughout this appendix, we remind that the adoption of extended arithmetics is not necessary, since, as a result of Lemma 4.2.(iii), all variables are confined in the open set $C$ onto which both $h$ and $f$ (and consequently $\hat{h}$ and $\hat{f}_{\!\beta }$ as well, for any $\beta \in \mathbb {R}$) are finite-valued.

1.1 Proof of Theorem 4.6.(i) and 4.6.(ii)

We begin by proving a technical lemma in the setting of Theorem 4.6.(A).

Lemma A.1

Suppose that Assumption 1 holds and let $\gamma >0$ and $\beta \in \mathbb {R}$ be such that $\hat{f}_{\!\beta }\mathrel {{=}}f\mathbin {\dot{-}}\frac{\beta }{\gamma }h{}+{{\,\mathrm{\delta }\,}}_{{{\,\textrm{dom}\,}}h}$ is a convex function. Then, for every $x,x^-\in C$ and $\bar{x}\in {\text {T}}_{\gamma \!,\,\beta }^{h\text {-frb}}(x,x^-)$

$$\begin{aligned} \left( \phi _{\gamma \!,\,\beta }^{h\text {-frb}}+{{\,\textrm{D}\,}}_{\hat{f}_{\!\beta }}\right) ({\bar{x}},x) \le \left( \phi _{\gamma \!,\,\beta }^{h\text {-frb}}+{{\,\textrm{D}\,}}_{\hat{f}_{\!\beta }}\right) (x,x^-) {}-{} {{\,\textrm{D}\,}}_{\hat{h}-2\hat{f}_{\!\beta }}({\bar{x}},x). \end{aligned}$$

(A.1)

Proof

The claimed inequality follows from (4.6) together with the fact that ${{\,\textrm{D}\,}}_{\hat{f}_{\!\beta }}\ge 0$. $\square$

In the setting of Theorem 4.6.(A), recall that we set $c\mathrel {{:}{=}}1+2\beta +3\alpha p_{-f,h}>0$. Then, inequality (A.1) can equivalently be written in terms of as

where the second inequality owes to the fact that ${{\,\textrm{D}\,}}_{\hat{h}-2\hat{f}_{\!\beta }-\frac{c}{\gamma }h}\ge 0$, since $\hat{h}-2\hat{f}_{\!\beta }-\frac{c}{\gamma }h$ is convex, having

the coefficient of $h$ being null by definition of $c$, and $-f-\sigma _{-f,h}h$ being convex by definition of the relative weak convexity modulus $\sigma _{-f,h}$, cf. Definition 3.3. This proves (4.10a); inequality (4.10b) follows similarly by observing that

$$\begin{aligned} 0 \le {{\,\textrm{D}\,}}_{\hat{h}-2\hat{f}_{\!\beta }-\frac{c}{\gamma }h} = {{\,\textrm{D}\,}}_{\hat{h}-\hat{f}_{\!\beta }}-{{\,\textrm{D}\,}}_{\hat{f}_{\!\beta }}-\tfrac{c}{\gamma }{{\,\textrm{D}\,}}\le {{\,\textrm{D}\,}}_{\hat{h}-\hat{f}_{\!\beta }}-\tfrac{c}{\gamma }{{\,\textrm{D}\,}}_h, \end{aligned}$$

so that

This concludes the proof of Theorem 4.6.(i) and 4.6.(ii) in the setting of Theorem 4.6.(A).

Now we work under the setting of Theorem 4.6.(B), in which case the following lemma will be useful.

Lemma A.2

Additionally to Assumption 1, suppose that $\hat{f}_{\!\beta }$ is $L_{\hat{f}_{\!\beta }}$-Lipschitz differentiable for some $L_{\hat{f}_{\!\beta }}\ge 0$. Then, for every $x,x^-\in C$ and ${\bar{x}}\in {\text {T}}_{\gamma \!,\,\beta }^{h\text {-frb}}(x,x^-)$ we have

$$\begin{aligned} \left( \phi _{\gamma \!,\,\beta }^{h\text {-frb}}+{{\,\textrm{D}\,}}_{L_{\hat{f}_{\!\beta }}{\mathcal {j}}}\right) ({\bar{x}},x) \le \left( \phi _{\gamma \!,\,\beta }^{h\text {-frb}}+{{\,\textrm{D}\,}}_{L_{\hat{f}_{\!\beta }}{\mathcal {j}}}\right) (x,x^-) - {{\,\textrm{D}\,}}_{\hat{h}-2L_{\hat{f}_{\!\beta }}{\mathcal {j}}}(\bar{x},x). \end{aligned}$$

(A.2)

Proof

By means of the three-point identity, that is, by using (4.3a) in place of (4.3b), inequality (4.6) can equivalently be written as

$$\begin{aligned} \phi _{\gamma \!,\,\beta }^{h\text {-frb}}({\bar{x}},x) \le \varphi _{\overline{C}}({\bar{x}}) ={}&\phi _{\gamma \!,\,\beta }^{h\text {-frb}}(x,x^-) - {{\,\textrm{D}\,}}_{\hat{h}}(\bar{x},x) - \langle {\bar{x}}-x,\nabla \hat{f}_{\!\beta }(x)-\nabla \hat{f}_{\!\beta }(x^-)\rangle \end{aligned}$$

which by using Young’s inequality on the inner product and $L_{\hat{f}_{\!\beta }}$-Lipschitz differentiability yields

$$\begin{aligned} \le {}&\phi _{\gamma \!,\,\beta }^{h\text {-frb}}(x,x^-) - {{\,\textrm{D}\,}}_{\hat{h}}({\bar{x}},x) + \tfrac{L_{\hat{f}_{\!\beta }}}{2}\Vert \bar{x}-x\Vert ^2 + \tfrac{L_{\hat{f}_{\!\beta }}}{2}\Vert x-x^-\Vert ^2. \end{aligned}$$

(A.3)

Rearranging and using the fact that ${{\,\textrm{D}\,}}_{{\mathcal {j}}}(x,y)=\frac{1}{2}\Vert x-y\Vert ^2$ yields the claimed inequality. $\square$

Under Theorem 4.6.(B), recall that we define

$$\begin{aligned} c \mathrel {{:}{=}}(1+\alpha p_{-f,h}) {}-{} \tfrac{2\gamma L_{\hat{f}_{\!\beta }}}{\sigma _h} > 0. \end{aligned}$$

We will pattern the arguments of the previous case, and observe that inequality (A.2) can equivalently be written in terms of as

Once again, the fact that ${{\,\textrm{D}\,}}_{\hat{h}-2L_{\hat{f}_{\!\beta }}{\mathcal {j}}-\frac{c}{\gamma }h}\ge 0$ owes to the convexity of $\hat{h}-2L_{\hat{f}_{\!\beta }}{\mathcal {j}}-\frac{c}{\gamma }h$ on ${{\,\textrm{dom}\,}}h$, having

altogether proving (4.10a). Similarly, inequality (4.10b) follows from (A.3) together with the fact that ${{\,\textrm{D}\,}}_{\hat{h}-L_{\hat{f}_{\!\beta }}{\mathcal {j}}}\ge {{\,\textrm{D}\,}}_{\hat{h}-2L_{\hat{f}_{\!\beta }}{\mathcal {j}}}\ge \frac{c}{\gamma }{{\,\textrm{D}\,}}_h$, as shown above, having

This concludes the proof of Theorems 4.6.(i) and 4.6.(ii).

1.2 Proof of Theorem 4.6.(iii)

We first state a property of the Bregman distance ${{\,\textrm{D}\,}}_h$ that holds when $h$ is as in the assertion of the theorem. The proof is provided for completeness, though part of it is straightforward and the rest is an easy adaptation of [6, Lem. 7.3(viii)].

Lemma A.3

Let $h:\mathbb {R}^n\rightarrow {\overline{\mathbb {R}}}$ be a 1-coercive Legendre kernel. If either $h$ is strongly convex or ${{\,\textrm{dom}\,}}h=\mathbb {R}^n$, then ${{\,\textrm{D}\,}}_h(x,y)$ is level bounded in $y$ locally uniformly in $x$.

Proof

Let $(x^k)_{k\in \mathbb {N}}$ and $(y^k)_{k\in \mathbb {N}}$ be sequences in $\mathbb {R}^n$ such that ${{\,\textrm{D}\,}}_h(x^k,y^k)\le \ell$ for some $\ell \in \mathbb {R}$. Suppose that $(x^k)_{k\in \mathbb {N}}$ is bounded; then the proof reduces to showing that also $(y^k)_{k\in \mathbb {N}}$ is. If $h$ is, say, $\sigma _h$-strongly convex for some $\sigma _h>0$, the claim trivially follows from the fact that ${{\,\textrm{D}\,}}_h(x,y)\ge (\sigma _h/2)\Vert x-y\Vert ^2$ in this case.

Suppose that ${{\,\textrm{dom}\,}}h=\mathbb {R}^n$, then it follows from [6, Thm. 3.4] that $h^*$ is 1-coercive. Furthermore, observe that

$$\begin{aligned} \ell \ge {{\,\textrm{D}\,}}_h(x^k,y^k) ={}&h(x^k)-h(y^k)-\langle \nabla h(y^k),x^k-y^k\rangle \\ ={}&h(x^k)+h^*(\nabla h(y^k))-\langle \nabla h(y^k),x^k\rangle \\ \ge {}&c+h^*(\nabla h(y^k))-\Vert \nabla h(y^k)\Vert c', \end{aligned}$$

where $c\mathrel {{:}{=}}\inf h(x^k)$ and $c'\mathrel {{:}{=}}\sup \Vert x^k\Vert$ are finite. Since $h^*$ is 1-coercive, it follows that $(\nabla h(y^k))_{k\in \mathbb {N}}$ is bounded, and therefore so is $(y^k)_{k\in \mathbb {N}}$ by virtue of [6, Thm. 3.3]. $\square$

We now turn to the proof of Theorem 4.6.(iii). By contraposition, suppose that is not level bounded, and consider an unbounded sequence $(x_k,x_k^-)_{k\in \mathbb {N}}$ such that

for some $\ell \in \mathbb {R}$. Then, it follows from (4.10b) that

$$\begin{aligned} \inf \varphi _{\overline{C}}+ \tfrac{c}{\gamma }{{\,\textrm{D}\,}}_h({\bar{x}}_k,x_k) + \tfrac{c}{2\gamma }{{\,\textrm{D}\,}}_h(x_k,x_k^-) \le \varphi _{\overline{C}}({\bar{x}}_k) + \tfrac{c}{\gamma }{{\,\textrm{D}\,}}_h({\bar{x}}_k,x_k) \\+ \tfrac{c}{2\gamma }{{\,\textrm{D}\,}}_h(x_k,x_k^-) \le \ell , \end{aligned}$$

and in particular both $({{\,\textrm{D}\,}}_h({\bar{x}}_k,x_k))_{k\in \mathbb {N}}$ and $({{\,\textrm{D}\,}}_h(x_k,x_k^-))_{k\in \mathbb {N}}$ are bounded. Moreover, it follows from Lemma A.3 that if $(x_k^-)_{k\in \mathbb {N}}$ is unbounded then so is $(x_k)_{k\in \mathbb {N}}$, and similarly unboundedness of $(x_k)_{k\in \mathbb {N}}$ implies that of $({\bar{x}}_k)_{k\in \mathbb {N}}$. Since at least one among $(x_k)_{k\in \mathbb {N}}$ and $(x_k^-)_{k\in \mathbb {N}}$ is unbounded, it follows that $({\bar{x}}_k)_{k\in \mathbb {N}}$ is unbounded. Noticing that this sequence is contained in $[\varphi _{\overline{C}}\le \ell ]$, we conclude that $\varphi _{\overline{C}}$ is not level bounded.

Proof of Theorem 5.6

In the remainder of this section, we will make use of the norm on the product space $\mathbb {R}^n\times \mathbb {R}^n$ defined as . For a set $E\subseteq \mathbb {R}^n$, define $(\forall \varepsilon >0)$ $E_\varepsilon =\{x\in \mathbb {R}^n:{{\,\textrm{dist}\,}}(x,E)<\varepsilon \}$.

Let $(\forall k\in \mathbb {N})$ $z^k=(x^{k+1},x^k,x^{k-1})$, and let $\Omega$ be the set of limit points of $(z_k)_{k\in \mathbb {N}}$. Define

Set $(\forall k\in \mathbb {N})$ for simplicity. Then , $\delta _{k}\rightarrow \varphi ^\star$ decreasingly and ${{\,\textrm{dist}\,}}(x^k,\Omega )\rightarrow 0$ as $k\rightarrow +\infty$ by invoking Theorem 5.1. Assume without loss of generality that $(\forall k\in \mathbb {N})$ $\delta _{k}>\varphi ^\star$, otherwise we would have $(\exists k_0\in \mathbb {N})$ $x^{k_0}=x^{k_0+1}$ due to Theorem 5.1.(i), from which the desired result readily follows by simple induction. Thus, $(\exists k_0\in \mathbb {N})$ $(\forall k\ge k_0)$ . Appealing to Theorem 5.1.(iii) and Lemma 4.2.(i) yields that is constantly equal to $\varphi ^\star$ on the compact set $\Omega$. Note that satisfies the KL property under Assumption 5.6.A3; see, e.g., [2, §4.3]. In turn, appealing to Assumption 5.6.A3 and a standard uniformizing technique of the KL property, see, e.g., [13, Lem. 6.2], implies that there exists a concave $\psi \in \Psi _\eta$ such that for $k\ge k_0$

(B.1)

Define $(\forall k\in \mathbb {N})$

$$\begin{aligned} u^k ={}&\nabla ^2\hat{h}(x^k)(x^k-x^{k+1})+\nabla ^2\hat{f}_{\!\beta }(x^k)(x^{k+1}-x^k)+\tfrac{c}{2\gamma }\bigl (\nabla h(x^k)-\nabla h(x^{k-1})\bigr ) \\&+\nabla \hat{f}_{\!\beta }(x^{k-1})-\nabla \hat{f}_{\!\beta }(x^k)+\nabla \xi (x^k)-\nabla \xi (x^{k-1}), \\ v^k ={}&\nabla ^2\hat{f}_{\!\beta }(x^{k-1})(x^k-x^{k+1}) \\ {}&+\tfrac{c}{2\gamma }\nabla ^2h(x^{k-1})(x^{k-1}-x^k)+\nabla ^2\xi (x^{k-1})(x^{k-1}-x^k). \end{aligned}$$

Applying subdifferential calculus to yields that

which together with Lemma 4.2.(iv) entails that . In turn, Assumption 5.6.A2 implies that there exists $M>0$ such that

(B.2)

Finally, we show that $(x^k)_{k\in \mathbb {N}}$ is convergent. For simplicity, define $(\forall k,l\in \mathbb {N})$ $\Delta _{k,l}=\psi \left( \delta _{k}-\varphi ^\star \right) -\psi \left( \delta _{l}-\varphi ^\star \right)$. Then, combining (B.1) and (B.2) yields

where the second inequality is implied by concavity of $\psi$, the third one follows from (5.1), and the fourth one holds because $\sigma >0$ is the strong convexity modulus of h on a convex compact set that contains all the iterates. Hence,

(B.3)

Summing (B.3) from $k=k_0$ to an arbitrary $l\ge k_0+1$ yields that

where the second inequality holds as $\psi \ge 0$, from which one sees that $\sum _{k=0}^\infty \Vert x^{k+1}-x^k\Vert$ is finite as l is arbitrary. A similar procedure shows that $(x^k)_{k\in \mathbb {N}}$ is Cauchy, which together with Theorem 5.1.(iii) entails the rest of the statement.

Proof of Theorem 5.9

Assume without loss of generality that has desingularizing function $\psi (t)=t^{1-\theta }/(1-\theta )$ and let $(\forall k\in \mathbb {N})$ $\delta _{k}=\sum _{i=k}^\infty \Vert x^{i+1}-x^i\Vert$. We claim that

$$\begin{aligned} (\forall k\ge k_0+1) ~\delta _k\le \tfrac{4\gamma M}{(1-\theta )c\sigma }e_{k}^{1-\theta }. \end{aligned}$$

(C.1)

Indeed, summing (B.3) from every $k\ge k_0$ to $l\ge k+1$ and passing l to infinity give

from which the desired claim readily follows. It is routine to see that the desired sequential rate can be implied by those of $(e_k)_{k\in \mathbb {N}}$ through (C.1); see, e.g., [45, Thm. 5.3]; therefore, it suffices to prove convergence rate of $(e_k)_{k\in \mathbb {N}}$.

Recall from Theorem 5.1.(i) that $(e_k)_{k\in \mathbb {N}}$ is a decreasing sequence converging to 0. Then invoking the KL exponent assumption yields

where the first equality holds due to Lemma 4.5.(ii), which together with (B.2) implies that

(C.2)

Appealing again to Theorem 5.1.(i) gives

where the last inequality is implied by (C.2). Then [15, Lem. 10] justifies the desired rate of $(e_k)_{k\in \mathbb {N}}$.