In this paper, we introduce the notion of generalized \(\epsilon \)-stationarity for a class of nonconvex and nonsmooth composite minimization problems on compact Riemannian submanifold embedded in Euclidean space. To find a generalized \(\epsilon \)-stationarity point, we develop a family of Riemannian gradient-type methods based on the Moreau envelope technique with a decreasing sequence of smoothing parameters, namely Riemannian smoothing gradient and Riemannian smoothing stochastic gradient methods. We prove that the Riemannian smoothing gradient method has the iteration complexity of \({\mathcal {O}}(\epsilon ^{-3})\) for driving a generalized \(\epsilon \)-stationary point. To our knowledge, this is the best-known iteration complexity result for the nonconvex and nonsmooth composite problem on manifolds. For the Riemannian smoothing stochastic gradient method, one can achieve the iteration complexity of \({\mathcal {O}}(\epsilon ^{-5})\) for driving a generalized \(\epsilon \)-stationary point. Numerical experiments are conducted to validate the superiority of our algorithms.

在本文中，我们介绍了嵌入欧几里得空间的紧黎曼子流形上的一类非凸非光滑复合最小化问题的广义\(\epsilon \)平稳性概念。为了找到广义的\(\epsilon\) -平稳点，我们开发了一系列基于莫罗包络技术的黎曼梯度型方法，具有平滑参数递减序列，即黎曼平滑梯度和黎曼平滑随机梯度方法。我们证明黎曼平滑梯度方法的迭代复杂度为\({\mathcal {O}}(\epsilon ^{-3})\)用于驱动广义\(\epsilon \)- 静止点。据我们所知，这是流形上非凸非光滑复合问题最著名的迭代复杂度结果。对于黎曼平滑随机梯度方法，可以达到\({\mathcal {O}}(\epsilon ^{-5})\)的迭代复杂度来驱动广义\(\epsilon \) -驻点。进行数值实验来验证我们算法的优越性。