Modern cosmological simulations have reached the trillion-element scale, rendering data storage and subsequent analysis formidable tasks. To address this circumstance, we present a new MPI-parallel approach for analysis of simulation data while the simulation runs, as an alternative to the traditional workflow consisting of periodically saving large data sets to disk for subsequent ‘offline’ analysis. We demonstrate this approach in the compressible gasdynamics/N-body code Nyx, a hybrid $\mbox{MPI}+\mbox{OpenMP}$ code based on the BoxLib framework, used for large-scale cosmological simulations. We have enabled on-the-fly workflows in two different ways: one is a straightforward approach consisting of all MPI processes periodically halting the main simulation and analyzing each component of data that they own (‘in situ’). The other consists of partitioning processes into disjoint MPI groups, with one performing the simulation and periodically sending data to the other ‘sidecar’ group, which post-processes it while the simulation continues (‘in-transit’). The two groups execute their tasks asynchronously, stopping only to synchronize when a new set of simulation data needs to be analyzed. For both the in situ and in-transit approaches, we experiment with two different analysis suites with distinct performance behavior: one which finds dark matter halos in the simulation using merge trees to calculate the mass contained within iso-density contours, and another which calculates probability distribution functions and power spectra of various fields in the simulation. Both are common analysis tasks for cosmology, and both result in summary statistics significantly smaller than the original data set. We study the behavior of each type of analysis in each workflow in order to determine the optimal configuration for the different data analysis algorithms.

中文翻译：

---

宇宙学模拟的原位和传输分析。

现代宇宙学模拟已达到万亿元素规模，使得数据存储和后续分析任务艰巨。为了解决这种情况，我们提出了一种新的 MPI 并行方法，用于在仿真运行时分析仿真数据，作为传统工作流程的替代方案，该传统工作流程包括定期将大型数据集保存到磁盘以供后续“离线”分析。我们在可压缩气体动力学/N 体代码 Nyx 中演示了这种方法，这是一个基于 BoxLib 框架的混合 $\mbox{M​​PI}+\mbox{OpenMP}$ 代码，用于大规模宇宙学模拟。我们以两种不同的方式启用动态工作流程：一种是一种简单的方法，由所有 MPI 进程组成，定期停止主模拟并分析它们拥有的数据的每个组成部分（“原位”）。另一种方法是将进程划分为不相交的 MPI 组，其中一个组执行模拟并定期将数据发送到另一个“sidecar”组，后者在模拟继续时（“传输中”）对数据进行后处理。两组异步执行任务，仅在需要分析一组新的模拟数据时停止同步。对于原位和传输方法，我们尝试了两种具有不同性能行为的不同分析套件：一种在模拟中使用合并树计算等密度轮廓内包含的质量来发现暗物质晕，另一种则计算模拟中各个领域的概率分布函数和功率谱。两者都是宇宙学的常见分析任务，并且两者都会导致汇总统计数据显着小于原始数据集。我们研究每个工作流程中每种类型分析的行为，以确定不同数据分析算法的最佳配置。