Solving trust region subproblems using Riemannian optimization

Abstract

The Trust Region Subproblem is a fundamental optimization problem that takes a pivotal role in Trust Region Methods. However, the problem, and variants of it, also arise in quite a few other applications. In this article, we present a family of iterative Riemannian optimization algorithms for a variant of the Trust Region Subproblem that replaces the inequality constraint with an equality constraint, and converge to a global optimum. Our approach uses either a trivial or a non-trivial Riemannian geometry of the search-space, and requires only minimal spectral information about the quadratic component of the objective function. We further show how the theory of Riemannian optimization promotes a deeper understanding of the Trust Region Subproblem and its difficulties, e.g., a deep connection between the Trust Region Subproblem and the problem of finding affine eigenvectors, and a new examination of the so-called hard case in light of the condition number of the Riemannian Hessian operator at a global optimum. Finally, we propose to incorporate preconditioning via a careful selection of a variable Riemannian metric, and establish bounds on the asymptotic convergence rate in terms of how well the preconditioner approximates the input matrix.

Appendix A: Constructing \(\phi \)

We now show a simple way to construct a smooth $\phi (\cdot )$ fulfilling both requirements stated in Items 1 and 2 in Sect. 6.

First, we choose some time parameter $\epsilon > 0$ and set $d:= \lambda _{\min }(\textbf{M}) - \epsilon$. We construct $\phi (\alpha )$ to be a smoothed out over-estimation of $\max (\alpha , -d)$. We first construct an under estimation. Let $\gamma >1$ be another parameter, and define

$$\begin{aligned} \varphi (\alpha )=\frac{\alpha +d}{2}\left( 1-\tanh \left( -\gamma \left( \alpha +d\right) \right) \right) -d. \end{aligned}$$

Then $\varphi (\alpha )$ is a smooth function that approximates $\max (\alpha , -d)$, but it is an under estimation: $\varphi (\alpha ) \le \max (\alpha , -d)$.

While the difference between $\max (\alpha , -d)$ and $\varphi (\alpha )$ reduces significantly when $\gamma$ grows, we want to make sure that our approximation is greater than or equal to the $\max (\alpha , -d)$. To that end, let

$$\begin{aligned} \alpha _0:= - \frac{\textrm{W}\left[ 0,e^{-1}\right] +1}{2\gamma } - d, \end{aligned}$$

where $\textrm{W}\left[ 0,\cdot \right]$ is the zero branch of the Lambert-$\textrm{W}$ function. Now, set:

$$\begin{aligned} \phi (\alpha ):=\varphi (\alpha )-\varphi (\alpha _{0})-d. \end{aligned}$$

(A.1)

See Fig. 5 for a graphical illustration of $\phi$. It is possible to show that

$$\begin{aligned} 0 \le \phi (\alpha ) - \max (\alpha , -\lambda _{\min }(\textbf{M})) \le \frac{\textrm{W}\big [0,e^{-1}\big ]+1}{2\gamma } \big (1 - \tanh \big (\big (\textrm{W}\big [0,e^{-1}\big ]+1\big )/2\big )\big )+ \epsilon , \end{aligned}$$

(A.2)

so Item 1 holds, and the approximation error (Item 2) is small if $\epsilon$ is sufficiently small and $\gamma$ is sufficiently large. The proof of Eq. (A.2) is rather technical and does not convey any additional insight on the btrs and its solution, so we omit it.

Fig. 5

Illustration of the approximation’s behavior. Values of the variable $\alpha$ are depicted in the x axis. Note that $\phi (-\alpha ) > - \lambda _{\min }(\textbf{M})$ for all $\alpha$, while in addition $\phi (\alpha )$ well approximates $\alpha$ for values of $\alpha \ge -\lambda _{\min }(\textbf{M})$