The visualization and detection of anomalies (outliers) are of crucial importance to many fields, particularly cybersecurity. Several approaches have been proposed in these fields, yet to the best of our knowledge, none of them has fulfilled both objectives, simultaneously or cooperatively, in one coherent framework. Moreover, the visualization methods of these approaches were introduced for explaining the output of a detection algorithm, not for data exploration that facilitates a standalone visual detection. This is our point of departure in introducing UN-AVOIDS, an unsupervised and nonparametric approach for both visualization (a human process) and detection (an algorithmic process) of outliers, that assigns invariant anomalous scores (normalized to [0, 1]), rather than hard binary-decision. The main aspect of novelty of UN-AVOIDS is that it transforms data into a new space, which is introduced in this paper as neighborhood cumulative density function (NCDF), in which both visualization and detection are carried out. In this space, outliers are remarkably visually distinguishable, and therefore the anomaly scores assigned by the detection algorithm achieved a high area under the ROC curve (AUC). We assessed UN-AVOIDS on both simulated and two recently published cybersecurity datasets, and compared it to three of the most successful anomaly detection methods: LOF, IF, and FABOD. In terms of AUC, UN-AVOIDS was almost an overall winner with a margin that varied between −0.028 and 0.125, depending on the data. The article concludes by providing a preview of new theoretical and practical avenues for UN-AVOIDS. Among them is designing a visualization aided anomaly detection (VAAD), a type of software that aids analysts by providing UN-AVOIDS’ detection algorithm (running in a back engine), NCDF visualization space (rendered to plots), along with other conventional methods of visualization in the original feature space, all of which are linked in one interactive environment.

中文翻译：

---

不可避免：用于可视化异常值和不变检测评分的无监督和非参数方法


异常（异常值）的可视化和检测对于许多领域（尤其是网络安全）至关重要。在这些领域已经提出了几种方法，但据我们所知，没有一种方法能够在一个连贯的框架中同时或合作地实现这两个目标。此外，引入这些方法的可视化方法是为了解释检测算法的输出，而不是为了促进独立视觉检测的数据探索。这是我们引入 UN-AVIODS 的出发点，这是一种无监督和非参数的方法，用于异常值的可视化（人类过程）和检测（算法过程），分配不变的异常分数（标准化为 [0, 1]），而不是硬二元决策。 UN-AVOODS的新颖之处主要在于它将数据转换到一个新的空间，本文将其引入为邻域累积密度函数（NCDF），在该空间中同时进行可视化和检测。在这个空间中，异常值在视觉上非常容易区分，因此检测算法分配的异常分数达到了较高的 ROC 曲线下面积 (AUC)。我们在模拟数据集和最近发布的两个网络安全数据集上评估了 UN-AVIODS，并将其与三种最成功的异常检测方法进行了比较：LOF、IF 和 FABOD。就 AUC 而言，UN-AVIODS 几乎是总体赢家，其优势在 -0.028 到 0.125 之间变化，具体取决于数据。本文最后概述了联合国避税措施的新理论和实践途径。 其中包括设计可视化辅助异常检测（VAAD），这是一种通过提供 UN-AVIODS 的检测算法（在后台引擎中运行）、NCDF 可视化空间（呈现为绘图）以及其他传统方法来帮助分析人员的软件原始特征空间中的可视化，所有这些都链接在一个交互式环境中。