Historical network traffic retrieval, both at the packet and flow level, has been applied in many fields of network security, such as network traffic analysis and network forensics. To retrieve specific packets from a vast number of packet traces, it is an effective solution to build indexes for the query attributes. However, it brings challenges of storage consumption and construction time overhead for packet indexing. To address these challenges, we propose an efficient indexing scheme called IndexWM based on the wavelet matrix data structure for packet indexing. Moreover, we design a packet storage format based on the PcapNG format for our network traffic collection and retrieval system, which can speed up the extraction of index data from packet traces. Offline experiments on randomly generated network traffic and actual network traffic are performed to evaluate the performance of the proposed indexing scheme. We choose an open-source and widely used bitmap indexing scheme, FastBit, for comparison. Apart from the native bitmap compression method Word-Aligned Hybrid (WAH), we implement an efficient bitmap compression method Scope-Extended COMPAX (SECOMPAX) in FastBit for performance evaluation. The comparison results show that our scheme outperforms the selected bitmap indexing schemes in terms of time consumption, storage consumption and retrieval efficiency.

中文翻译：

---

一种高效的网络流量收集与检索系统索引方案

在数据包级别和流级别的历史网络流量检索已应用于网络安全的许多领域，例如网络流量分析和网络取证。要从大量数据包跟踪中检索特定数据包，这是为查询属性建立索引的有效解决方案。然而，这带来了存储消耗和分组索引的构造时间开销的挑战。为了解决这些挑战，我们提出了一种基于小波矩阵数据结构的高效索引方案IndexWM，用于数据包索引。此外，我们为网络流量收集和检索系统设计了基于PcapNG格式的数据包存储格式，可以加快从数据包跟踪中提取索引数据的速度。对随机生成的网络流量和实际网络流量进行脱机实验，以评估所提出的索引方案的性能。我们选择一种开源且使用广泛的位图索引方案FastBit进行比较。除了本机位图压缩方法Word-Aligned Hybrid（WAH）外，我们还在FastBit中实现了一种有效的位图压缩方法Scope-Extended COMPAX（SECOMPAX）以进行性能评估。比较结果表明，我们的方案在时间消耗，存储消耗和检索效率方面均优于所选的位图索引方案。我们在FastBit中实现了有效的位图压缩方法Scope-Extended COMPAX（SECOMPAX），以进行性能评估。比较结果表明，我们的方案在时间消耗，存储消耗和检索效率方面均优于所选的位图索引方案。我们在FastBit中实现了有效的位图压缩方法Scope-Extended COMPAX（SECOMPAX），以进行性能评估。比较结果表明，我们的方案在时间消耗，存储消耗和检索效率方面均优于所选的位图索引方案。