Sentiment analysis methods have become popular for investigating human communication, including discussions related to software projects. Since general-purpose sentiment analysis tools do not fit well with the information exchanged by software developers, new tools, specific for software engineering (SE), have been developed. We investigate to what extent off-the-shelf SE-specific tools for sentiment analysis mitigate the threats to conclusion validity of empirical studies in software engineering, highlighted by previous research. First, we replicate two studies addressing the role of sentiment in security discussions on GitHub and in question-writing on Stack Overflow. Then, we extend the previous studies by assessing to what extent the tools agree with each other and with the manual annotation on a gold standard of 600 documents. We find that different SE-specific sentiment analysis tools might lead to contradictory results at a fine-grain level, when used off-the-shelf. Conversely, platform-specific tuning or retraining might be needed to take into account differences in platform conventions, jargon, or document lengths.

情感分析方法已成为调查人类交流的流行方法，包括与软件项目相关的讨论。由于通用情感分析工具不能很好地适应软件开发人员交换的信息，因此开发了专门用于软件工程 (SE) 的新工具。我们调查了用于情绪分析的现成的 SE 特定工具在多大程度上减轻了对软件工程实证研究结论有效性的威胁，这在以前的研究中得到了强调。首先，我们复制了两项研究，探讨情绪在 GitHub 上的安全讨论和 Stack Overflow 上的问题编写中的作用。然后，我们通过评估工具彼此之间以及与 600 份文档的黄金标准手动注释的一致程度来扩展先前的研究。现成的。相反，可能需要特定于平台的调整或重新培训以考虑平台约定、行话或文档长度的差异。