Hate speech is an important problem that is seriously affecting the dynamics and usefulness of online social communities. Large scale social platforms are currently investing important resources into automatically detecting and classifying hateful content, without much success. On the other hand, the results reported by state-of-the-art systems indicate that supervised approaches achieve almost perfect performance but only within specific datasets, most of them in English language. In this work, we analyze this apparent contradiction between existing literature and actual applications. We study closely the experimental methodology used in prior work and their generalizability to other datasets. Our findings evidence methodological issues, as well as an important dataset bias. As a consequence, performance claims of the current state-of-the-art have become significantly overestimated. The problems that we have found are mostly related to data overfitting and sampling issues. We discuss the implications for current research and re-conduct experiments to give a more accurate picture of the current state-of-the art methods. Moreover, we design some baseline approaches to perform cross-lingual experiments, using English and Spanish datasets.

仇恨言论是一个重要问题，严重影响在线社交社区的动态和实用性。当前，大型社交平台正在将重要资源投入到自动检测和分类可恨内容方面，但收效甚微。另一方面，最新系统报告的结果表明，受监督的方法几乎可以实现完美的性能，但只能在特定的数据集中使用，大多数是英语。在这项工作中，我们分析了现有文献与实际应用之间的这种明显矛盾。我们仔细研究了先前工作中使用的实验方法及其对其他数据集的一般性。我们的发现证明了方法论上的问题以及重要的数据集偏差。作为结果，当前最新技术的性能要求已被大大高估。我们发现的问题主要与数据过拟合和抽样问题有关。我们讨论了对当前研究和再进行实验的意义，以便更准确地了解当前最先进的方法。此外，我们使用英语和西班牙语数据集设计了一些基准方法来执行跨语言实验。