In this work we investigate the practicality of stochastic gradient descent and its variants with variance-reduction techniques in imaging inverse problems. Such algorithms have been shown in the large-scale optimization and machine learning literature to have optimal complexity in theory, and to provide great improvement empirically over the deterministic gradient methods. However, in some tasks such as image deblurring, many of such methods fail to converge faster than the deterministic gradient methods, even in terms of epoch counts. We investigate this phenomenon and propose a theory-inspired mechanism for the practitioners to efficiently characterize whether it is beneficial for an inverse problem to be solved by stochastic optimization techniques or not. Using standard tools in numerical linear algebra, we derive conditions on the spectral structure of the inverse problem for being a suitable application of stochastic gradient methods. Particularly, if the Hessian matrix of an imaging inverse problem has a fast-decaying eigenspectrum, then our theory suggests that the stochastic gradient methods can be more advantageous than deterministic methods for solving such a problem. Our results also provide guidance on choosing appropriately the partition minibatch schemes, showing that a good minibatch scheme typically has relatively low correlation within each of the minibatches. Finally, we present numerical studies which validate our results.

中文翻译：

---

随机优化在成像逆问题中的实用性

在这项工作中，我们研究了随机梯度下降及其变体在成像逆问题中使用方差减少技术的实用性。此类算法在大规模优化和机器学习文献中已被证明在理论上具有最佳复杂度，并且在经验上比确定性梯度方法提供了很大的改进。然而，在图像去模糊等一些任务中，许多这样的方法无法比确定性梯度方法更快地收敛，即使在纪元计数方面也是如此。我们研究了这种现象，并为从业者提出了一种受理论启发的机制，以有效地表征通过随机优化技术解决逆问题是否有益。使用数值线性代数中的标准工具，我们推导出逆问题的谱结构的条件，作为随机梯度方法的合适应用。特别是，如果成像逆问题的 Hessian 矩阵具有快速衰减的特征谱，那么我们的理论表明，在解决此类问题时，随机梯度方法可能比确定性方法更有利。我们的结果还提供了在适当地选择分区小炉方案的指导，表明良好的小纤维方案通常在每个小匹匹匹匹配中具有相对较低的相关性。最后，我们提出了验证我们结果的数值研究。那么我们的理论表明，在解决此类问题时，随机梯度方法可能比确定性方法更有优势。我们的结果还提供了在适当地选择分区小炉方案的指导，表明良好的小纤维方案通常在每个小匹匹匹匹配中具有相对较低的相关性。最后，我们提出了验证我们结果的数值研究。那么我们的理论表明，在解决此类问题时，随机梯度方法可能比确定性方法更有优势。我们的结果还提供了在适当地选择分区小炉方案的指导，表明良好的小纤维方案通常在每个小匹匹匹匹配中具有相对较低的相关性。最后，我们提出了验证我们结果的数值研究。