Modeling feature interactions is of crucial importance to predict click-through rate (CTR) in industrial recommender systems. However, manually crafting cross features usually requires extensive domain knowledge and labor-intensive feature engineering to obtain the desired cross features. To alleviate this problem, the factorization machine (FM) is proposed to model feature interactions from raw features automatically. In particular, it embeds each feature in a vector representation and discovers second-order interactions as the product of two feature representations. In order to learn nonlinear and complex patterns, recent works, such as NFM, PIN, and DeepFM, exploited deep learning techniques to capture higher-order feature interactions. These approaches lack guarantees about the effectiveness of high-order pattern as they model feature interactions in a rather implicit way. To address this limitation, xDeepFM is recently proposed to generate high-order interactions of features in an explicit fashion, where multiple interaction networks are stacked. Nevertheless, xDeepFM suffers from rather high complexity which easily leads to overfitting.

In this paper, we develop a more expressive but lightweight solution based on FM, named High-order Attentive Factorization Machine (HoAFM), by accounting for the higher-order sparse feature interactions in an explicit manner. Beyond the linearity of FM, we devise a cross interaction layer, which updates a feature’s representation by aggregating the representations of other co-occurred features. In addition, we perform a bit-wise attention mechanism to determine the different importance of co-occurred features on the granularity of dimensions. By stacking multiple cross interaction layers, we can inject high-order feature interactions into feature representation learning, in order to establish expressive and informative cross features. Extensive experiments are performed on two benchmark datasets, Criteo and Avazu, to demonstrate the rationality and effectiveness of HoAFM. Empirical results suggest that HoAFM achieves significant improvement over other state-of-the-art methods, such as NFM and xDeepFM. https://github.com/zltao/HoAFM.

对功能交互进行建模对于预测工业推荐系统中的点击率（CTR）至关重要。但是，手动制作十字形特征通常需要广泛的领域知识和劳动密集型的特征工程才能获得所需的十字形特征。为了缓解这个问题，提出了分解机（FM），可以自动根据原始特征对特征交互进行建模。特别是，它将每个特征嵌入矢量表示中，并发现二阶交互作为两个特征表示的乘积。为了学习非线性和复杂的模式，近来的作品（例如NFM，PIN和DeepFM）利用深度学习技术来捕获高阶特征交互。这些方法缺乏对高阶模式有效性的保证，因为它们以相当隐式的方式对特征交互进行建模。为了解决此限制，最近提出了xDeepFM以显式方式生成要素的高阶交互，其中堆叠了多个交互网络。但是，xDeepFM具有相当高的复杂性，很容易导致过度拟合。

在本文中，我们开发了一种基于FM的更具表现力但轻巧的解决方案，称为高阶注意力分解机（HoAFM），通过以显式方式考虑高阶稀疏特征交互。除了FM的线性之外，我们还设计了一个交叉交互层，该层通过汇总其他共同出现的要素的表示来更新要素的表示。此外，我们执行按位关注机制以确定共同出现的功能在维度粒度上的不同重要性。通过堆叠多个交叉交互层，我们可以将高阶特征交互注入特征表示学习中，以建立表达性和信息性交叉特征。在两个基准数据集Criteo和Avazu上进行了广泛的实验，以证明HoAFM的合理性和有效性。实验结果表明，HoAFM相对于其他最新方法（例如NFM和xDeepFM）取得了显着改善。https://github.com/zltao/HoAFM。