With the growing use of information technology in all life domains, hacking has become more negatively effective than ever before. Also with developing technologies, attacks numbers are growing exponentially every few months and become more sophisticated so that traditional IDS becomes inefficient detecting them. This paper proposes a solution to detect not only new threats with higher detection rate and lower false positive than already used IDS, but also it could detect collective and contextual security attacks. We achieve those results by using Networking Chatbot, a deep recurrent neural network: Long Short Term Memory (LSTM) on top of Apache Spark Framework that has an input of flow traffic and traffic aggregation and the output is a language of two words, normal or abnormal. We propose merging the concepts of language processing, contextual analysis, distributed deep learning, big data, anomaly detection of flow analysis. We propose a model that describes the network abstract normal behavior from a sequence of millions of packets within their context and analyzes them in near real-time to detect point, collective and contextual anomalies. Experiments are done on MAWI dataset, and it shows better detection rate not only than signature IDS, but also better than traditional anomaly IDS. The experiment shows lower false positive, higher detection rate and better point anomalies detection. As for prove of contextual and collective anomalies detection, we discuss our claim and the reason behind our hypothesis. But the experiment is done on random small subsets of the dataset because of hardware limitations, so we share experiment and our future vision thoughts as we wish that full prove will be done in future by other interested researchers who have better hardware infrastructure than ours.

中文翻译：

---

大数据分析和分布式深度学习，用于下一代入侵检测系统优化

随着信息技术在所有生活领域中的广泛使用，黑客变得比以往任何时候都更加负面。同样，随着技术的发展，攻击的数量每隔几个月就呈指数增长，并且变得越来越复杂，因此传统的IDS对其进行检测变得效率低下。本文提出了一种解决方案，该解决方案不仅可以检测比已使用的IDS具有更高的检测率和更低的误报率的新威胁，而且还可以检测集体和上下文安全攻击。我们通过使用深度聊天神经网络Networking Chatbot来获得这些结果：Apache Spark Framework之上的长期短期记忆（LSTM），具有输入流量和流量聚合的输入，并且输出是两个单词的语言，即正常语言或不正常。我们建议合并语言处理的概念，上下文分析，分布式深度学习，大数据，流量分析异常检测。我们提出了一个模型，该模型从其上下文中的数百万个数据包序列中描述网络抽象正常行为，并以近实时的方式对其进行分析，以检测点，集合和上下文异常。在MAWI数据集上进行了实验，它不仅显示出比签名IDS更好的检测率，而且还显示出比传统异常IDS更好的检测率。实验显示出较低的假阳性率，较高的检测率和更好的点异常检测。至于上下文异常和集体异常检测的证明，我们讨论了我们的主张以及我们的假设背后的原因。但是由于硬件限制，该实验是在数据集的随机小子集上完成的，