当前位置:
X-MOL 学术
›
arXiv.cs.CG
›
论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Streaming Complexity of SVMs
arXiv - CS - Computational Geometry Pub Date : 2020-07-07 , DOI: arxiv-2007.03633 Alexandr Andoni, Collin Burns, Yi Li, Sepideh Mahabadi, David P. Woodruff
arXiv - CS - Computational Geometry Pub Date : 2020-07-07 , DOI: arxiv-2007.03633 Alexandr Andoni, Collin Burns, Yi Li, Sepideh Mahabadi, David P. Woodruff
We study the space complexity of solving the bias-regularized SVM problem in
the streaming model. This is a classic supervised learning problem that has
drawn lots of attention, including for developing fast algorithms for solving
the problem approximately. One of the most widely used algorithms for
approximately optimizing the SVM objective is Stochastic Gradient Descent
(SGD), which requires only $O(\frac{1}{\lambda\epsilon})$ random samples, and
which immediately yields a streaming algorithm that uses
$O(\frac{d}{\lambda\epsilon})$ space. For related problems, better streaming
algorithms are only known for smooth functions, unlike the SVM objective that
we focus on in this work. We initiate an investigation of the space complexity
for both finding an approximate optimum of this objective, and for the related
``point estimation'' problem of sketching the data set to evaluate the function
value $F_\lambda$ on any query $(\theta, b)$. We show that, for both problems,
for dimensions $d=1,2$, one can obtain streaming algorithms with space
polynomially smaller than $\frac{1}{\lambda\epsilon}$, which is the complexity
of SGD for strongly convex functions like the bias-regularized SVM, and which
is known to be tight in general, even for $d=1$. We also prove polynomial lower
bounds for both point estimation and optimization. In particular, for point
estimation we obtain a tight bound of $\Theta(1/\sqrt{\epsilon})$ for $d=1$ and
a nearly tight lower bound of $\widetilde{\Omega}(d/{\epsilon}^2)$ for $d =
\Omega( \log(1/\epsilon))$. Finally, for optimization, we prove a
$\Omega(1/\sqrt{\epsilon})$ lower bound for $d = \Omega( \log(1/\epsilon))$,
and show similar bounds when $d$ is constant.
中文翻译:
SVM 的流式传输复杂性
我们研究了在流模型中解决偏置正则化 SVM 问题的空间复杂度。这是一个经典的监督学习问题,引起了很多关注,包括开发用于近似解决问题的快速算法。用于近似优化 SVM 目标的最广泛使用的算法之一是随机梯度下降 (SGD),它只需要 $O(\frac{1}{\lambda\epsilon})$ 随机样本,并立即产生流算法使用 $O(\frac{d}{\lambda\epsilon})$ 空间。对于相关问题,更好的流算法仅适用于平滑函数,这与我们在这项工作中关注的 SVM 目标不同。我们开始研究空间复杂度,以找到该目标的近似最优值,以及相关的“点估计” 绘制数据集以评估任何查询 $(\theta, b)$ 上的函数值 $F_\lambda$ 的问题。我们证明,对于这两个问题,对于维度 $d=1,2$,可以得到空间多项式小于 $\frac{1}{\lambda\epsilon}$ 的流算法,这是 SGD 的复杂度凸函数类似于偏置正则化 SVM,并且众所周知它通常是紧的,即使对于 $d=1$。我们还证明了点估计和优化的多项式下界。特别是,对于点估计,我们获得了 $\Theta(1/\sqrt{\epsilon})$ 的紧边界,对于 $d=1$ 和 $\widetilde{\Omega}(d/{ \epsilon}^2)$ 为 $d = \Omega( \log(1/\epsilon))$。最后,为了优化,我们证明了 $\Omega(1/\sqrt{\epsilon})$ 下界 $d = \Omega( \log(1/\epsilon))$,
更新日期:2020-07-08
中文翻译:
SVM 的流式传输复杂性
我们研究了在流模型中解决偏置正则化 SVM 问题的空间复杂度。这是一个经典的监督学习问题,引起了很多关注,包括开发用于近似解决问题的快速算法。用于近似优化 SVM 目标的最广泛使用的算法之一是随机梯度下降 (SGD),它只需要 $O(\frac{1}{\lambda\epsilon})$ 随机样本,并立即产生流算法使用 $O(\frac{d}{\lambda\epsilon})$ 空间。对于相关问题,更好的流算法仅适用于平滑函数,这与我们在这项工作中关注的 SVM 目标不同。我们开始研究空间复杂度,以找到该目标的近似最优值,以及相关的“点估计” 绘制数据集以评估任何查询 $(\theta, b)$ 上的函数值 $F_\lambda$ 的问题。我们证明,对于这两个问题,对于维度 $d=1,2$,可以得到空间多项式小于 $\frac{1}{\lambda\epsilon}$ 的流算法,这是 SGD 的复杂度凸函数类似于偏置正则化 SVM,并且众所周知它通常是紧的,即使对于 $d=1$。我们还证明了点估计和优化的多项式下界。特别是,对于点估计,我们获得了 $\Theta(1/\sqrt{\epsilon})$ 的紧边界,对于 $d=1$ 和 $\widetilde{\Omega}(d/{ \epsilon}^2)$ 为 $d = \Omega( \log(1/\epsilon))$。最后,为了优化,我们证明了 $\Omega(1/\sqrt{\epsilon})$ 下界 $d = \Omega( \log(1/\epsilon))$,