The massive amount of data produced during simulation on high-performance computers has grown exponentially over the past decade, exacerbating the need for streaming compression and decompression methods for efficient storage and transfer of this data—key to realizing the full potential of large-scale computational science.

Lossy compression approaches such as JPEG when applied to scientific simulation data realized as a stream of images can achieve good compression rates but at the cost of introducing compression artifacts and loss of information. This paper develops a unified framework for in situ compression artifact removal in which the fully convolutional neural network architectures are combined with scalable training, transfer learning, and experience replay to achieve superior accuracy and efficiency while significantly decreasing the storage footprint as compared with the traditional optimization-based approaches.

We demonstrate the proposed approach and compare it with compressed sensing postprocessing and other baseline deep learning models using climate simulations and nuclear reactor simulations, both of which are driven by hyperbolic partial differential equations. Our approach when applied to remove the compression artifacts on the JPEG-compressed nuclear reactor simulation data (using a transfer-trained model that was pretrained on the climate simulation data and updated incrementally as the nuclear reactor simulation progressed), achieved a significant improvement—mean peak signal-to-noise ratio of 42.438 as compared with 27.725 obtained with the compressed sensing approach.

在过去的十年中，在高性能计算机上模拟过程中生成的大量数据呈指数增长，这加剧了对流压缩和解压缩方法进行有效存储和传输数据的需求，这是实现大规模计算的全部潜力的关键科学。

当将有损压缩方法（如JPEG）应用于以图像流形式实现的科学模拟数据时，可以实现良好的压缩率，但以引入压缩伪像和信息丢失为代价。本文开发了一个用于原位压缩伪影消除的统一框架，该框架中，全卷积神经网络体系结构与可扩展的训练，传递学习和体验重播相结合，以实现卓越的准确性和效率，同时与传统优化相比可显着减少存储空间基于方法。

我们演示了提出的方法，并将其与使用气候模拟和核反应堆模拟的压缩感测后处理以及其他基线深度学习模型进行了比较，二者均由双曲型偏微分方程驱动。我们的方法在去除JPEG压缩核反应堆模拟数据上的压缩伪影（使用在气候模拟数据上经过预训练并随着核反应堆模拟的进行而逐步更新的传递训练模型）上时，取得了显着的进步-平均值峰值信噪比为42.438，而压缩感测方法的峰值信噪比为27.725。