In recent years, with the rapid rise of social networks, such as Weibo and Twitter, multimodal social network rumors have also spread. Unlike traditional unimodal rumor detection, the main difficulty of multimodal rumor detection is in avoiding the generation of noise information while using the complementarity of different modal features. In this article, we propose a multimodal online social network rumor detection model based on the multilevel attention residual neural network (MARN). First, the features of text and image are extracted by Bert and ResNet-18, respectively, and the cross-attention residual mechanism is used to enhance the representation of images with a text vector. Second, the enhanced image vector and text vector are concatenated and fused by the self-attention residual mechanism. Finally, the fused image–text vectors are classified into two categories. Among them, the attention mechanism can effectively enhance the image representation and further improve the fusion effect between the image and the text, while the residual mechanism retains the unique attributes of each original modal feature while using different modal features. To assess the performance of the MARN model, we conduct experiments on the Weibo dataset, and the results show that the MARN model outperforms the state-of-the-art models in terms of accuracy and F1 value.

近年来，随着微博、推特等社交网络的迅速崛起，多模态社交网络谣言也随之蔓延。与传统的单模态谣言检测不同，多模态谣言检测的主要难点在于在利用不同模态特征的互补性的同时避免噪声信息的产生。在本文中，我们提出了一种基于多级注意残差神经网络（MARN）的多模态在线社交网络谣言检测模型。首先，分别通过 Bert 和 ResNet-18 提取文本和图像的特征，并使用交叉注意力残差机制来增强文本向量对图像的表示。其次，增强的图像向量和文本向量通过自注意力残差机制进行连接和融合。最后，融合的图像-文本向量分为两类。其中，注意力机制可以有效增强图像表示，进一步提高图像与文本的融合效果，而残差机制在使用不同的模态特征的同时保留了每个原始模态特征的独特属性。为了评估 MARN 模型的性能，我们在微博数据集上进行了实验，结果表明 MARN 模型在准确性和 F1 值方面优于最先进的模型。而残差机制在使用不同的模态特征的同时保留了每个原始模态特征的独特属性。为了评估 MARN 模型的性能，我们在微博数据集上进行了实验，结果表明 MARN 模型在准确性和 F1 值方面优于最先进的模型。而残差机制在使用不同的模态特征的同时保留了每个原始模态特征的独特属性。为了评估 MARN 模型的性能，我们在微博数据集上进行了实验，结果表明 MARN 模型在准确性和 F1 值方面优于最先进的模型。