Large-scale simulations of blood flow that resolve the 3D deformation of each comprising cell are increasingly popular owing to algorithmic developments in conjunction with advances in compute capability. Among different approaches for modeling cell-resolved hemodynamics, fluid structure interaction (FSI) algorithms based on the immersed boundary method are frequently employed for coupling separate solvers for the background fluid and the cells within one framework. GPUs can accelerate these simulations; however, both current pre-exascale and future exascale CPU-GPU heterogeneous systems face communication challenges critical to performance and scalability. We describe, to our knowledge, the largest distributed GPU-accelerated FSI simulations of high hematocrit cell-resolved flows with over 17 million red blood cells. We compare scaling on a fat node system with six GPUs per node and on a system with a single GPU per node. Through comparison between the CPU- and GPU-based implementations, we identify the costs of data movement in multiscale multi-grid FSI simulations on heterogeneous systems and show it to be the greatest performance bottleneck on the GPU.

由于算法的发展以及计算能力的提高，解决每个组成细胞的3D变形的大规模血流模拟越来越受欢迎。在用于建模细胞解析的血流动力学的不同方法中，基于沉浸边界方法的流体结构相互作用（FSI）算法经常用于在一个框架内耦合背景流体和细胞的单独求解器。GPU可以加速这些仿真。但是，当前的亿万次级和将来的亿万亿级CPU-GPU异构系统都面临着对性能和可伸缩性至关重要的通信挑战。据我们所知，我们对具有超过1700万个红细胞的高血细胞比容的高分辨率流进行了最大的分布式GPU加速FSI模拟。我们将比较每个节点具有六个GPU的胖节点系统和每个节点具有单个GPU的系统的缩放比例。通过比较基于CPU和GPU的实现，我们确定了异构系统上多尺度多网格FSI仿真中数据移动的成本，并表明它是GPU上最大的性能瓶颈。