Appraisal theories explain how the cognitive evaluation of an event leads to a particular emotion. In contrast to theories of basic emotions or affect (valence/arousal), this theory has not received a lot of attention in natural language processing. Yet, in psychology it has been proven powerful: Smith and Ellsworth (1985) showed that the appraisal dimensions attention, certainty, anticipated effort, pleasantness, responsibility/control and situational control discriminate between (at least) 15 emotion classes. We study different annotation strategies for these dimensions, based on the event-focused enISEAR corpus (Troiano et al., 2019). We analyze two manual annotation settings: (1) showing the text to annotate while masking the experienced emotion label; (2) revealing the emotion associated with the text. Setting 2 enables the annotators to develop a more realistic intuition of the described event, while Setting 1 is a more standard annotation procedure, purely relying on text. We evaluate these strategies in two ways: by measuring inter-annotator agreement and by fine-tuning RoBERTa to predict appraisal variables. Our results show that knowledge of the emotion increases annotators' reliability. Further, we evaluate a purely automatic rule-based labeling strategy (inferring appraisal from annotated emotion classes). Training on automatically assigned labels leads to a competitive performance of our classifier, even when tested on manual annotations. This is an indicator that it might be possible to automatically create appraisal corpora for every domain for which emotion corpora already exist.

中文翻译：

---

情绪感知，不可知或自动：获取认知事件评估注释的语料库创建策略

评估理论解释了事件的认知评估如何导致特定的情感。与基本情绪或情感（价/声）理论相比，该理论在自然语言处理中并未引起广泛关注。然而，在心理学上，它已被证明是有效的：Smith和Ellsworth（1985）表明，评估维度的关注，确定性，预期的努力，愉悦，责任/控制和情境控制可区分（至少）15个情绪类别。我们基于以事件为中心的enISEAR语料库研究了这些维度的不同注释策略（Troiano et al。，2019）。我们分析了两个手动注释设置：（1）在掩盖经验丰富的情感标签的同时显示要注释的文本；（2）揭示与文字有关的情感。设置2使注释者能够对所描述的事件产生更真实的直觉，而设置1是更标准的注释过程，仅依赖于文本。我们通过两种方式评估这些策略：通过测量注释者之间的协议以及通过微调RoBERTa来预测评估变量。我们的结果表明，对情感的了解增加了注释者的可靠性。此外，我们评估了一种基于规则的纯自动标注策略（从带注释的情感类别中推断出评估结果）。即使在手动注释上进行测试，对自动分配的标签进行的培训也可以使我们的分类器具有竞争优势。这表明有可能为已经存在情感语料库的每个域自动创建评估语料库。