Complex storage stacks providing data compression, indexing, and analytics help leverage the massive amounts of data generated today to derive insights. It is challenging to perform this computation, however, while fully utilizing the underlying storage media. This is because, while storage servers with large core counts are widely available, single-core performance and memory bandwidth per core grow slower than the core count per die. Computational storage offers a promising solution to this problem by utilizing dedicated compute resources along the storage processing path. We present DeltaFS Indexed Massive Directories (IMDs), a new approach to computational storage. DeltaFS IMDs harvest available (i.e., not dedicated) compute, memory, and network resources on the compute nodes of an application to perform computation on data. We demonstrate the efficiency of DeltaFS IMDs by using them to dynamically reorganize the output of a real-world simulation application across 131,072 CPU cores. DeltaFS IMDs speed up reads by 1,740× while only slightly slowing down the writing of data during simulation I/O for in situ data processing.

中文翻译：

---

使用 DeltaFS 索引的海量目录大规模流式数据重组

提供数据压缩、索引和分析的复杂存储堆栈有助于利用当今生成的大量数据来获得洞察力。然而，在充分利用底层存储介质的同时执行这种计算具有挑战性。这是因为，虽然具有大量核心数的存储服务器广泛可用，但单核性能和每个核心的内存带宽的增长速度要慢于每个裸片的核心数。计算存储通过利用存储处理路径上的专用计算资源为这个问题提供了一个有前途的解决方案。我们介绍了 DeltaFS 索引海量目录 (IMD)，这是一种计算存储的新方法。DeltaFS IMD 在应用程序的计算节点上收集可用（即非专用）计算、内存和网络资源，以对数据执行计算。我们通过使用 DeltaFS IMD 来动态重组跨 131,072 个 CPU 内核的真实世界模拟应用程序的输出，展示了 DeltaFS IMD 的效率。DeltaFS IMD 将读取速度提高了 1,740 倍，同时在模拟 I/O 期间仅略微减慢了数据写入速度原位数据处理。