With the world’s interaction moving more and more toward using online social media platforms, the advent of cyberbullying has also raised its head. Multiple forms of cyberbullying exist from the more common text based to images or even videos, and this paper will explore the context of textual comments. Even in the niche area of considering only text-based data, several approaches have already been worked upon such as n-grams, recurrent units, convolutional neural networks (CNNs), gated recurrent unit (GRU) and even a combination of the mentioned architectures. While all of these produce workable results, the main point of contention is that true contextual understanding is quite a complex concept. These methods fail due to two simple reasons: (i) lack of large datasets to properly utilize these architectures and (ii) the fact that understanding context requires some mechanism of remembering history that is only present in the recurrent units. This paper explores some of the recent approaches to the difficulties of contextual understanding and proposes an ALBERT-based fine-tuned model that achieves state-of-the-art results. ALBERT is a transformer-based architecture and thus even at its untrained form provides better contextual understanding than other recurrent units. This coupled with the fact that ALBERT is pre-trained on a large corpus allowing the flexibility to use a smaller dataset for fine-tuning as the pre-trained model already has deep understanding of the complexities of the human language. ALBERT showcases high scores in multiple benchmarks such as the GLUE and SQuAD showing that high levels of contextual understanding are inherently present and thus fine-tuning for the specific case of cyberbullying allows to use this to our advantage. With this approach, we have achieved an F1 score of 95% which surpasses current approaches such as the CNN + wordVec, CNN + GRU and BERT implementations.

中文翻译：

---

基于 ALBERT 的网络欺凌分析微调模型

随着世界的互动越来越倾向于使用在线社交媒体平台，网络欺凌的出现也引起了人们的注意。网络欺凌存在多种形式，从更常见的文本到图像甚至视频，本文将探讨文本评论的上下文。即使在仅考虑基于文本的数据的利基领域，也已经研究了几种方法，例如 n-gram、循环单元、卷积神经网络 (CNN)、门控循环单元 (GRU) 甚至上述架构的组合. 虽然所有这些都产生了可行的结果，但主要争论点是真正的上下文理解是一个相当复杂的概念。由于两个简单的原因，这些方法失败了：(i) 缺乏正确利用这些架构的大型数据集，以及 (ii) 理解上下文需要某种记忆历史的机制，而这种机制仅存在于循环单元中。本文探讨了最近解决上下文理解困难的一些方法，并提出了一种基于 ALBERT 的微调模型，该模型可实现最先进的结果。ALBERT 是一种基于转换器的架构，因此即使在未经训练的形式下，也能提供比其他循环单元更好的上下文理解。再加上 ALBERT 是在大型语料库上预先训练的，因此可以灵活地使用较小的数据集进行微调，因为预先训练的模型已经对人类语言的复杂性有了深入的了解。ALBERT 在 GLUE 和 SQuAD 等多个基准测试中展示了高分，表明高水平的上下文理解是固有的，因此针对网络欺凌的特定情况进行微调可以利用这一点。通过这种方法，我们取得了 95% 的 F1 分数，超过了当前的方法，例如 CNN + wordVec、CNN + GRU 和 BERT 实现。