Purpose

Despite the extensive academic interest in social media sentiment for financial fields, multimodal data in the stock market has been neglected. The purpose of this paper is to explore the influence of multimodal social media data on stock performance, and investigate the underlying mechanism of two forms of social media data, i.e. text and pictures.

Design/methodology/approach

This research employs panel vector autoregressive models to quantify the effect of the sentiment derived from two modalities in social media, i.e. text information and picture information. Through the models, the authors examine the short-term and long-term associations between social media sentiment and stock performance, measured by three metrics. Specifically, the authors design an enhanced sentiment analysis method, integrating random walk and word embeddings through Global Vectors for Word Representation (GloVe), to construct a domain-specific lexicon and apply it to textual sentiment analysis. Secondly, the authors exploit a deep learning framework based on convolutional neural networks to analyze the sentiment in picture data.

Findings

The empirical results derived from vector autoregressive models reveal that both measures of the sentiment extracted from textual information and pictorial information in social media are significant leading indicators of stock performance. Moreover, pictorial information and textual information have similar relationships with stock performance.

Originality/value

To the best of the authors’ knowledge, this is the first study that incorporates multimodal social media data for sentiment analysis, which is valuable in understanding pictures of social media data. The study offers significant implications for researchers and practitioners. This research informs researchers on the attention of multimodal social media data. The study’s findings provide some managerial recommendations, e.g. watching not only words but also pictures in social media.

中文翻译：

---

探索多模式社交媒体数据对股票绩效的影响：实证研究与分析

目的

尽管学术界对金融领域的社交媒体情绪有着广泛的兴趣，但股票市场中的多模式数据却被忽略了。本文的目的是探讨多模式社交媒体数据对股票表现的影响，并研究文本和图片两种形式的社交媒体数据的潜在机制。

设计/方法/方法

这项研究采用面板向量自回归模型来量化源自社交媒体中两种方式（即文本信息和图片信息）的情感效果。通过这些模型，作者检查了社交媒体情绪与股票表现之间的短期和长期关联性，并通过三个指标进行了衡量。具体来说，作者设计了一种增强的情感分析方法，通过用于单词表示的全局矢量（GloVe）整合随机行走和单词嵌入，以构造特定于域的词典并将其应用于文本情感分析。其次，作者利用基于卷积神经网络的深度学习框架来分析图片数据中的情感。

发现

从向量自回归模型得出的经验结果表明，从社交媒体中的文字信息和图片信息中提取的情感量度都是股票表现的重要领先指标。此外，图形信息和文本信息与股票表现具有相似的关系。

创意/价值

据作者所知，这是第一项将多模式社交媒体数据纳入情感分析的研究，这对于理解社交媒体数据的图片非常有价值。该研究对研究人员和从业人员具有重大意义。这项研究告知研究人员多模式社交媒体数据的关注。该研究的发现提供了一些管理建议，例如，不仅要看单词，还要看社交媒体中的图片。