In this paper, we analyze the generalization performance of the Iterative Hard Thresholding (IHT) algorithm widely used for sparse recovery problems. The parameter estimation and sparsity recovery consistency of IHT has long been known in compressed sensing. From the perspective of statistical learning, another fundamental question is how well the IHT estimation would predict on unseen data. This paper makes progress towards answering this open question by introducing a novel sparse generalization theory for IHT under the notion of algorithmic stability. Our theory reveals that: 1) under natural conditions on the empirical risk function over $n$ samples of dimension $p$ , IHT with sparsity level $k$ enjoys an $\tilde {\mathcal {O}}(n^{-1/2}\sqrt {k\log (n)\log (p)})$ rate of convergence in sparse excess risk; 2) a tighter $\tilde {\mathcal {O}}(n^{-1/2}\sqrt {\log (n)})$ bound can be established by imposing an additional iteration stability condition on a hypothetical IHT procedure invoked to the population risk; and 3) a fast rate of order $\tilde {\mathcal {O}}\left ({n^{-1}k(\log ^{3}(n)+\log (p))}\right)$ can be derived for strongly convex risk function under proper strong-signal conditions. The results have been substantialized to sparse linear regression and sparse logistic regression models to demonstrate the applicability of our theory. Preliminary numerical evidence is provided to support our theoretical predictions.

中文翻译：

---

迭代硬阈值的稳定性和风险界限


在本文中，我们分析了广泛用于稀疏恢复问题的迭代硬阈值（IHT）算法的泛化性能。 IHT 的参数估计和稀疏恢复一致性在压缩感知领域早已为人所知。从统计学习的角度来看，另一个基本问题是 IHT 估计对未见数据的预测效果如何。本文通过在算法稳定性的概念下引入一种新颖的 IHT 稀疏泛化理论，在回答这个悬而未决的问题方面取得了进展。我们的理论表明：1）自然条件下的经验风险函数$n$维度样本$p$ , 具有稀疏级别的 IHT $k$享有$\tilde {\mathcal {O}}(n^{-1/2}\sqrt {k\log (n)\log (p)})$稀疏超额风险的收敛率； 2）更紧$\tilde {\mathcal {O}}(n^{-1/2}\sqrt {\log (n)})$可以通过对调用总体风险的假设 IHT 过程施加额外的迭代稳定性条件来建立边界； 3）订单速度快$\tilde {\mathcal {O}}\left ({n^{-1}k(\log ^{3}(n)+\log (p))}\right)$可以在适当的强信号条件下导出强凸风险函数。结果已被具体化为稀疏线性回归和稀疏逻辑回归模型，以证明我们的理论的适用性。 提供了初步的数字证据来支持我们的理论预测。