Development and application of an HDF5 schema for SKA-scale image cube visualization

Abstract

In this paper, we describe an HDF5 schema created to support the efficient visualization of the large image cubes that will be produced by SKA Phase 1 and precursor radio telescopes. We demonstrate how the “HDF5-IDIA” schema’s features can improve the performance of visualization software, using both low-level metrics and real-world tests of the schema’s implementation in CARTA, an image viewer that is being developed to replace the existing CyberSKA and CASA viewers.

Introduction

Data produced by radio telescopes such as MeerKAT (Jonas et al., 2018), as well as other Square Kilometer Array (SKA) pathfinders, is now being used to create multi-dimensional image cubes which are large enough to pose multiple challenges to visualization software, requiring an increase in storage space, memory and computation resources. In our ongoing effort to facilitate efficient access to these images, we realized that the way in which the data was represented on storage systems created a considerable limitation to the performance that we could achieve. In particular, we wanted the ability to store additional pre-calculated representations and access them through a well-defined hierarchy.

Most astronomical image files are currently packaged using the FITS standard¹  (Wells and Greisen, 1979). However, the FITS standard is primarily used for image transport and archiving, and is not well-suited for storing or defining additional derived data products in a hierarchical structure. The HDF5 technology suite (Folk et al., 2011) provides a data model, file format, API, library, and tools to enable the creation of structured schemas for different applications. We will show how these can be beneficial in packaging of large radio astronomy image cubes for the purpose of visualization and visual analytics.

We initially attempted to utilize existing HDF5 schemas developed for image cubes, but found that they did not meet our needs. The LOFAR HDF5 schema (Anderson et al., 2010, Alexov et al., 2012) did not meet performance requirements: each 2D image plane is stored in a separate group, therefore a separate group and dataset must be opened for each pixel when data is read along the third axis of the image cube.

The HDFITS schema (Price et al., 2015) serves as a starting point for an HDF5 schema that maintains round-trip compatibility with the FITS format, but lacks the additional structures required for precalculated and cached datasets. We have therefore created a new schema tailored to our application, with a hierarchy similar to that of HDFITS, but extensions have been added to support a number of features required for efficient visualization of large datasets.

The rest of the paper is laid out as follows: Section 2 details the requirements we have for the new schema. Section 3 considers the types of workloads that datasets using this schema will commonly be used for, in the context of image cube visualization. Section 4 describes the optional datasets that are defined in the schema in order to accelerate these workloads, as well as an outline of the schema hierarchy and naming conventions. Section 5 details integration of support for the schema into CARTA: The Cube Analysis and Rendering Tool for Astronomy (Comrie et al., 2018), as of Version 1.2 of the software package. Section 6 shows performance metrics of the schema with low-level benchmarks, and compares its performance to that of FITS within CARTA.