With recent emergence of machine learning problems with massive number of features, feature selection (FS) has become an ever-increasingly important tool to mitigate the effects of the so-called curse of dimensionality. FS aims to eliminate redundant and irrelevant features for models that are faster to train, easier to understand, and less prone to overfitting. This study presents a wrapper FS method based on Simultaneous Perturbation Stochastic Approximation (SPSA) with Barzilai and Borwein (BB) non-monotone gains within a pseudo-gradient descent framework wherein performance is measured via cross-validation. We illustrate that SPSA with BB gains (SPSA-BB) provides dramatic improvements in terms of the number of iterations for convergence with minimal degradation in cross-validated error performance over the current state-of-the art approach with monotone gains (SPSA-MON). In addition, SPSA-BB requires only one internal parameter and therefore it eliminates the need for careful fine-tuning of numerous other internal parameters as in SPSA-MON or comparable meta-heuristic FS methods such as genetic algorithms (GA). Our particular implementation includes gradient averaging as well as gain smoothing for better convergence properties. We present computational experiments on various public datasets with Nearest Neighbors and Naive Bayes classifiers as wrappers. We present comparisons of SPSA-BB against full set of features, SPSA-MON, as well as seven popular meta-heuristics based FS algorithms including GA and particle swarm optimization. Our results indicate that SPSA-BB converges to a good feature set in about 50 iterations on the average regardless of the number of features (whether a dozen or more than 1000 features) and its performance is quite competitive. SPSA-BB can be considered extremely fast for a wrapper method and therefore it stands as a high-performing new feature selection method that is also computationally feasible in practice.

随着最近出现的具有大量特征的机器学习问题，特征选择（FS）已成为缓解所谓的“维数诅咒”的影响的越来越重要的工具。FS的目标是消除模型的冗余和不相关的特征，这些特征的训练速度更快，更易于理解，并且不易过度拟合。这项研究提出了一种基于伪随机下降框架内的Barzilai和Borwein（BB）非单调增益同时扰动随机逼近（SPSA）的包装器FS方法，其中性能是通过交叉验证来衡量的。我们说明，与当前具有单调增益的最新方法（SPSA-MON）相比，具有BB增益的SPSA（SPSA-BB）在收敛的迭代次数方面提供了显着的改进，并且交叉验证的错误性能降幅最小。 ）。此外，SPSA-BB仅需要一个内部参数，因此无需像SPSA-MON或类似的元启发式FS方法（例如遗传算法（GA））中那样，对许多其他内部参数进行仔细的微调。我们的特定实现包括梯度平均以及增益平滑，以实现更好的收敛特性。我们介绍了以最近邻居和朴素贝叶斯分类器作为包装器的各种公共数据集的计算实验。我们将SPSA-BB与完整功能SPSA-MON，以及基于GA和粒子群优化的7种流行的基于元启发式的FS算法。我们的结果表明，SPSA-BB平均可以在大约50次迭代中收敛到一个好的功能集，而与功能的数量（无论是12个还是1000个以上的功能）无关，并且其性能具有相当的竞争力。对于包装方法，SPSA-BB可以认为是非常快的，因此它是一种高性能的新特征选择方法，在实践中在计算上也是可行的。