A generic, cluster-centred lossless compression framework for joint auroral data

Abstract

Studying the well-known phenomenon “aurora” plays a pivotal role in investigating the solar–terrestrial coupling mechanism. A special auroral spectrograph in Antarctic Zhongshan Station constitutes a auroral observation joint system with satellite-borne sensors of the Defense Meteorological Satellite Program. Multipoint observation by this system provides more essential information for relevant studies than single observation by each instrument, but also results in a multifold increased volume of data that are difficult to be either stored or transmitted. To address this difficulty, we develop a clustering-based, generic lossless data compression framework that combines the usage of various ultimate compressors with a hierarchical clustering algorithm to exert the strength of all the compressors in data reduction. This framework achieves an always-best compression performance for different-sized datasets with a reasonable time consumption, which promises the design of pipelines using it for real-time data transmission.

Introduction

Aurora is representative of energetic particle precipitation into the upper atmosphere of Earth’s polar regions. It is the visible manifestation of spontaneous emissions concerning some excited atmospheric compositions by collision with the precipitating electrons/protons originated from solar wind or magnetosphere [1], [2]. Relevant studies are crucial for uncovering the secrets of multiscale coupling of the Sun–Earth system.

Auroral observing can be conducted by various devices independently, e.g., optical imagers measuring electromagnetic energy from a distance and particle detectors capable of in-situ energy measurements, some of which are operated on the ground, and some are satellite-borne that pass over fixed locations at different times, even in a day. It is also feasible to carry out the multi-point observation, i.e., exploit different sensors to measure the auroral characteristics of interest in a simultaneous way, in which case these devices constitute a joint observing system. That collects information on a larger scale than the observations by individual devices, facilitating a comprehensive understanding of relevant physical and chemical processes.

Up to now, several cooperative observing networks including United States Antarctic Program [3], Scientific Commission on Antarctic Research (SCAR) programs [4] and Chinese Meridian Project [5] have been established for auroral observations. In a different way, virtual observation platforms can be built for economic considerations under the premise that all involved instruments concurrently observe the same event or phenomenon. An example of joint auroral observing system consists of a customized high-resolution auroral spectrograph (ASG) in Antarctic Zhongshan Station (ZHS) [6] and Special Sensor Ultraviolet Spectrographic Imager (SSUSI) [7] and Special Sensor J5 (SSJ5) [8] positioned on a Defense Meteorological Satellite Program (DMSP) satellite. In Fig. 1, one DMSP satellite is passing the magnetic meridian over ZHS, and the intersection of blue and green curves manifesting the trajectory of satellite and the effective field of view of ASG, respectively.

There is no doubt that a combined use of jointly observed data from different sources brings benefits to auroral research in simulating the auroral particle transport in the atmosphere [9], [10], [11], reversely tracing the origins of aurora [12], [13], [14], and etc. However, it associates with a great difficulty in the preservation and manipulation of numerous files that have various formats. Lossless compression is widely used for reversible data reduction, Besides, the problem caused by great variety of data (which refers to the diversity of data types and data sources) can be overcome by several means.

In this regard, this paper presents a lossless compression framework for the mixed data concerning aurora observations by using multiple remarkably-performing compressors with the help of hierarchical clustering. That is, raw data are allocated into several clusters regardless of their types, then losslessly compressed cluster by cluster. The data of each cluster is processed by the same compressor, albeit more than one compressor is adopted. This framework has the generality of application to any dataset, and its compression performance is supposed to be always better than all used compressors. For validation, we combine the natural data collected by satellite-borne detectors SSUSI and SSJ5 with auroral spectral data (ASD) simultaneously obtained from ZHS, concentrating on the two-hour and two-day periods for experiments. Results demonstrate that our proposed framework improves the encoding efficiency based on other compressors and effectively utilizes the similarity between data, though slightly compromising the computation efficiency due to the usage of hierarchical clustering.

More details about this implementation are described in the rest of paper. Section 2 introduces some background knowledge. Section 3 provides the procedures about how to combine hierarchical clustering and lossless compression for efficiently storing a great variety of data. Section 4 discusses the detailed experimental design and analysis. At last, the summary and prospect is given.