X-ray computed tomography is a commonly used technique for noninvasive imaging at synchrotron facilities. Iterative tomographic reconstruction algorithms are often preferred for recovering high quality 3D volumetric images from 2D X-ray images, however, their use has been limited to small/medium datasets due to their computational requirements. In this paper, we propose a high-performance iterative reconstruction system for terabyte(s)-scale 3D volumes. Our design involves three novel optimizations: (1) optimization of (back)projection operators by extending the 2D memory-centric approach to 3D; (2) performing hierarchical communications by exploiting "fat-node" architecture with many GPUs; (3) utilization of mixed-precision types while preserving convergence rate and quality. We extensively evaluate the proposed optimizations and scaling on the Summit supercomputer. Our largest reconstruction is a mouse brain volume with 9Kx11Kx11K voxels, where the total reconstruction time is under three minutes using 24,576 GPUs, reaching 65 PFLOPS: 34% of Summit's peak performance.

中文翻译：

---

Petascale XCT：在多 GPU 节点上使用分层通信进行 3D 图像重建

X 射线计算机断层扫描是在同步加速器设施中进行无创成像的常用技术。迭代断层扫描重建算法通常更适合从 2D X 射线图像中恢复高质量的 3D 体积图像，但是，由于其计算要求，它们的使用仅限于中小型数据集。在本文中，我们提出了一种用于 TB 级 3D 卷的高性能迭代重建系统。我们的设计涉及三个新颖的优化：（1）通过将 2D 以内存为中心的方法扩展到 3D 来优化（反）投影算子；(2) 利用具有多个 GPU 的“胖节点”架构进行分层通信；(3) 在保持收敛速度和质量的同时利用混合精度类型。我们在 Summit 超级计算机上广泛评估了建议的优化和扩展。我们最大的重建是具有 9Kx11Kx11K 体素的小鼠大脑体积，其中使用 24,576 个 GPU 的总重建时间不到三分钟，达到 65 PFLOPS：Summit 峰值性能的 34%。