Consider the classical supervised learning problem: we are given data $(y_i,{\mathit{x}}_i)$, $i\le n$, with $y_i$ a response and ${\mathit{x}}_i\in \mathcal{X}$ a covariates vector, and try to learn a model $\hat{f}:\mathcal{X}\to \mathbb{R}$ to predict future responses. Random feature methods map the covariates vector ${\mathit{x}}_i$ to a point $\mathit{\varphi }({\mathit{x}}_i)$ in a higher dimensional space ${\mathbb{R}}^N$, via a random featurization map ϕ. We study the use of random feature methods in conjunction with ridge regression in the feature space ${\mathbb{R}}^N$. This can be viewed as a finite-dimensional approximation of kernel ridge regression (KRR), or as a stylized model for neural networks in the so called lazy training regime.

We define a class of problems satisfying certain spectral conditions on the underlying kernels, and a hypercontractivity assumption on the associated eigenfunctions. These conditions are verified by classical high-dimensional examples. Under these conditions, we prove a sharp characterization of the error of random feature ridge regression. In particular, we address two fundamental questions: (1) What is the generalization error of KRR? (2) How big N should be for the random feature approximation to achieve the same error as KRR?

In this setting, we prove that KRR is well approximated by a projection onto the top ℓ eigenfunctions of the kernel, where ℓ depends on the sample size n. We show that the test error of random feature ridge regression is dominated by its approximation error and is larger than the error of KRR as long as $N\le n^{1-\delta }$ for some $\delta >0$. We characterize this gap. For $N\ge n^{1+\delta }$, random features achieve the same error as the corresponding KRR, and further increasing N does not lead to a significant change in test error.

考虑经典的监督学习问题：给定数据 $({是}_{\mathrm{一世}},{\mathit{X}}_{\mathrm{一世}})$, $\mathrm{一世}\le n$， 和 ${是}_{\mathrm{一世}}$ 一个回应和 ${\mathit{X}}_{\mathrm{一世}}\in \mathcal{X}$ 一个协变量向量，并尝试学习一个模型 $\widehat{F}：\mathcal{X}\to \mathbb{电阻}$预测未来的反应。随机特征方法映射协变量向量${\mathit{X}}_{\mathrm{一世}}$ 到一点 $\mathit{\phi }({\mathit{X}}_{\mathrm{一世}})$ 在高维空间 ${\mathbb{电阻}}^N$，通过一个随机特征图ϕ。我们在特征空间中研究了随机特征方法与岭回归的结合使用${\mathbb{电阻}}^N$. 这可以看作是核岭回归 (KRR) 的有限维近似，或者看作是所谓的惰性训练机制中神经网络的程式化模型。

我们在底层内核上定义了一类满足某些谱条件的问题，以及对相关特征函数的超收缩假设。这些条件由经典的高维例子验证。在这些条件下，我们证明了随机特征岭回归误差的尖锐特征。特别是，我们解决了两个基本问题：（1）KRR 的泛化误差是多少？(2)随机特征逼近应该有多大的N才能达到与 KRR 相同的误差？

在这种情况下，我们证明 KRR 可以通过投影到内核的顶部ℓ特征函数上来很好地近似，其中ℓ取决于样本大小n。我们表明随机特征岭回归的测试误差主要由其逼近误差决定，并且只要是大于 KRR 的误差$N\le n^{1-\delta }$ 对于一些 $\delta >0$. 我们描述了这种差距。为了$N\ge n^{1+\delta }$，随机特征达到与对应的 KRR 相同的误差，进一步增加N不会导致测试误差的显着变化。