SIAM Journal on Optimization, Volume 31, Issue 1, Page 59-90, January 2021.
Recently, there has been a surge of interest in designing variants of the classical Newton-CG in which the Hessian of a (strongly) convex function is replaced by suitable approximations. This is mainly motivated by large-scale finite-sum minimization problems that arise in many machine learning applications. Going beyond convexity, inexact Hessian information has also been recently considered in the context of algorithms such as trust-region or (adaptive) cubic regularization for general nonconvex problems. Here, we do that for Newton-MR, which extends the application range of the classical Newton-CG beyond convexity to invex problems. Unlike the convergence analysis of Newton-CG, which relies on spectrum preserving Hessian approximations in the sense of Löwner partial order, our work here draws from matrix perturbation theory to estimate the distance between the range spaces underlying the exact and approximate Hessian matrices. Numerical experiments demonstrate a great degree of resilience to such Hessian approximations, amounting to a highly efficient algorithm in large-scale problems.

中文翻译：

---

不精确的黑森州信息下Newton-MR的收敛性

SIAM优化杂志，第31卷，第1期，第59-90页，2021年1月。
近来，在设计经典牛顿-CG的变体中引起了兴趣的涌现，其中（强）凸函数的Hessian被合适的近似值代替。这主要是由许多机器学习应用程序中出现的大规模有限和最小化问题引起的。除了凸性之外，最近还已经在诸如信任区域或（自适应）三次正则化等算法的上下文中考虑了不精确的Hessian信息，以解决一般的非凸问题。在这里，我们对Newton-MR进行了处理，这将经典的Newton-CG的应用范围从凸性扩展到了凸问题。与牛顿CG的收敛分析不同，牛顿CG依赖于在Löwner偏序意义上保持光谱的Hessian近似，我们在这里的工作来自矩阵摄动理论，用于估计精确和近似Hessian矩阵所基于的范围空间之间的距离。数值实验表明，这种Hessian近似具有很大的弹性，在大规模问题中相当于一种高效的算法。