Feature selection by maximizing high-order mutual information between the selected feature vector and a target variable is the gold standard in terms of selecting the best subset of relevant features that maximizes the performance of prediction models. However, such an approach typically requires knowledge of the multivariate probability distribution of all features and the target, and involves a challenging combinatorial optimization problem. Recent work has shown that any joint Probability Mass Function (PMF) can be represented as a naive Bayes model, via Canonical Polyadic (tensor rank) Decomposition. In this paper, we introduce a low-rank tensor model of the joint PMF of all variables and indirect targeting as a way of mitigating complexity and maximizing the classification performance for a given number of features. Through low-rank modeling of the joint PMF, it is possible to circumvent the curse of dimensionality by learning ‘principal components’ of the joint distribution. By indirectly aiming to predict the latent variable of the naive Bayes model instead of the original target variable, it is possible to formulate the feature selection problem as maximization of a monotone submodular function subject to a cardinality constraint – which can be tackled using a greedy algorithm that comes with performance guarantees. Numerical experiments with several standard datasets suggest that the proposed approach compares favorably to the state-of-art for this important problem.

中文翻译：

---

通过张量分解和子模性进行信息论特征选择


通过最大化所选特征向量和目标变量之间的高阶互信息来进行特征选择是选择相关特征的最佳子集以最大化预测模型性能的黄金标准。然而，这种方法通常需要了解所有特征和目标的多元概率分布，并且涉及具有挑战性的组合优化问题。最近的工作表明，任何联合概率质量函数 (PMF) 都可以通过正则多元（张量秩）分解表示为朴素贝叶斯模型。在本文中，我们引入了所有变量的联合 PMF 和间接目标的低秩张量模型，作为减轻复杂性并最大化给定数量特征的分类性能的一种方法。通过联合 PMF 的低秩建模，可以通过学习联合分布的“主成分”来避免维数灾难。通过间接预测朴素贝叶斯模型的潜在变量而不是原始目标变量，可以将特征选择问题表述为受基数约束的单调子模函数的最大化——这可以使用贪心算法来解决附带性能保证。对几个标准数据集的数值实验表明，所提出的方法在这个重要问题上优于现有技术。