Sarcasm is a sophisticated construct to express contempt or ridicule. It is well-studied in multiple disciplines (e.g., neuroanatomy and neuropsychology) but is still in its infancy in computational science (e.g., Twitter sarcasm detection). In contrast to previous methods that are usually geared toward a single discipline, we focus on the multidisciplinary cross-innovation, i.e., improving embryonic sarcasm detection in computational science by leveraging the advanced knowledge of sarcasm cognition in neuroanatomy and neuropsychology. In this work, we are oriented toward sarcasm detection in social media and correspondingly propose a multimodal, multi-interactive, and multihierarchical neural network ( $M_{3}N_{2} $ ). We select Twitter, image, text in image, and image caption as the input of $M_{3}N_{2} $ since the brain’s perception of sarcasm requires multiple modalities. To reasonably address the multimodalities, we introduce singlewise, pairwise, triplewise, and tetradwise modality interactions incorporating gate mechanism and guide attention (GA) to simulate the interactions and collaborations of involved regions in the brain while perceiving multiple modes. Specifically, we exploit a multihop process for each modality interaction to extract modal information multiple times using GA for obtaining multiperspective information. Also, we adopt a two-hierarchical structure leveraging self-attention accompanied by attention pooling to integrate multimodal semantic information from different levels mimicking the brain’s first- and second-order comprehensions of sarcasm. Experimental results show that $M_{3}N_{2} $ achieves competitive performance in sarcasm detection and displays powerful generalization ability in multimodal sentiment analysis and emotion recognition.

中文翻译：

---

从多学科模仿大脑对讽刺的认知，用于 Twitter 讽刺检测

讽刺是表达蔑视或嘲笑的复杂结构。它在多个学科（例如，神经解剖学和神经心理学）中得到了充分研究，但在计算科学（例如，Twitter 讽刺检测）中仍处于起步阶段。与以往通常针对单一学科的方法不同，我们专注于多学科交叉创新，即通过利用神经解剖学和神经心理学中讽刺认知的先进知识，改进计算科学中的胚胎讽刺检测。在这项工作中，我们面向社交媒体中的讽刺检测，并相应地提出了一个多模态、多交互和多层次的神经网络（ $M_{3}N_{2} $ ). 我们选择 Twitter、图像、图像中的文本和图像标题作为输入 $M_{3}N_{2} $因为大脑对讽刺的感知需要多种方式。为了合理地解决多模态问题，我们引入了结合门机制和引导注意力 (GA) 的单向、成对、三向和四向模态交互，以模拟大脑中相关区域在感知多种模式时的交互和协作。具体来说，我们为每个模态交互利用多跳过程，使用 GA 多次提取模态信息以获得多视角信息。此外，我们采用双层次结构，利用自注意力和注意力池来整合来自不同层次的多模态语义信息，模仿大脑对讽刺的一阶和二阶理解。实验结果表明 $M_{3}N_{2} $在讽刺检测方面取得了竞争优势，并在多模态情感分析和情感识别方面表现出强大的泛化能力。