Simplified square to hexagonal lattice conversion based on 1-D multirate processing

Highlights

• Analysis of the square to hexagonal lattice conversion.

• Simplified and efficient square to hexagonal conversion method.

• Efficiency analysis and experimental validation.

Abstract

Hexagonal image sampling and processing are theoretically superior to the most commonly used square lattice based sampling and processing, but due to the lack of commercial image sensors, current research mainly relies on virtually hexagonally sampled data through square to hexagonal lattice conversion, which is a typical 2-D interpolation problem. This paper presents a simplified and efficient square to hexagonal lattice conversion method. The method firstly utilizes the separable nature of the interpolation kernel to simplify the original 2-D interpolation into 1-D interpolation along the horizontal direction only, and then it applies the 1-D multirate technique to further simplify the shift-variant 1-D interpolation into shift-invariant 1-D convolutions. Compared with the original 2-D interpolation version, the proposed method becomes both simple and computationally efficient, and it is also suitable for implementation with parallel processing and hardware. Finally, experiments are performed and the results are consistent with the analysis.

Introduction

Currently, digital imaging devices (including image sensors and displays) are predominantly based on the square lattice, but the multidimensional sampling theory has shown that [1], [2], for a circularly band-limited analog image, the hexagonal lattice sampling can provide 13.4% fewer samples than the square lattice sampling. Since most practical imaging systems are circularly symmetric, their modulation transfer functions (MTFs) exhibit circularly low-pass nature and the outputs will become circularly band-limited, thus the hexagonal lattice is actually the optimal sampling scheme for practical sampled imaging systems. In addition to sampling efficiency, hexagonal lattice has superior geometric properties, such as higher degree of symmetry, equal distance and uniform connectivity with its six neighbors [3], as shown in Fig. 1. More important, hexagonal lattice is also common in the structures of biological visual sensors, such as the compound eyes of insects [4] and the retina of human eyes [5], and for this reason, hexagonal image processing has attracted the researchers of computer vision since the early days.

In the seminal paper [2], Mersereau has established fundamental theories of hexagonal image processing, including the hexagonal sampling, the hexagonal linear shift-invariant system, the hexagonal Fourier transform, the hexagonal discrete Fourier transform (HDFT), the hexagonal recursive system, and the hexagonal FIR filter design. This seminal paper further demonstrates the superiority of computational efficiency and performance. Since then, especially with the rise of biologically inspired image processing, hexagonal image processing has begun to attract more researchers’ attention, and in recent years the research has spread to broad applications like image restoration [6], image registration [7], [8], edge detection [9], [10], image labeling [11], morphological processing [12], [13], shape retrieval [14], hexagonal Gabor filtering [15], ultrasound image processing [16], computed tomography (CT) image reconstruction [17], [18], [19], [20], image sensor design [21], [22], [23], and convolutional neural network [24], [25], [26].

Despite the attractive advantages, due to the lack of practical imaging sensors (CCD and CMOS), for the hexagonal image processing at the present time, it needs to obtain the equivalent hexagonally sampled data by resampling from the commonly used square lattice data, i.e., by performing the square to hexagonal lattice conversion. Speake and Mersereau [27] have outlined the general lattice conversion theory using matrix description, and Mersereau [28] also developed the lattice conversion theory with the 2-D multirate processing techniques. Generally speaking, square to hexagonal lattice conversion is a 2-D interpolation problem. That is, according to the sampling theory, if the Nyquist criterion has been fulfilled, the original analog image can be completely reconstructed, and then the lattice conversion can be done by resampling the reconstructed image [29]. In practice, simple interpolation kernels are often used [30], [31], e.g., nearest-neighbor [32] and bilinear [33]. To reduce the aliasing artifacts in the classical interpolation methods, Ville et al. [34] proposed to use the hexagonal spline functions. Condat et al. [35] proposed a reversible conversion method that decomposes the lattice conversion process into three successive shear operations, and they implemented the method using 1-D fractional delay filters. Besides, an intuitive approach is based on sub-pixel clustering [9], [10], [36], which firstly upsamples the image to obtain a more denser version and then merges several square sub-pixels of the intermediate data into the equivalent hexagonal pixel. For evaluation purpose, Li et al. [37] proposed an ideal lattice conversion method in the frequency domain.

Due to the 2-D nature, the common square to hexagonal lattice conversion is a bit complex. Moreover, the hexagonal image processing research inevitably involves comparing with the original square lattice, and for fair comparison, it is preferable that the two sets of samples are equivalent with each other, thus high quality lattice conversion is important. However, in this case, the 2-D interpolation kernel will be much larger, and the computational costs will increase dramatically. Regarding to this practical matter, this paper aims to simplify the common 2-D interpolation approach. The proposed method converts the original 2-D interpolation into 1-D interpolation along one direction only, and then further simplifies the 1-D interpolation with the 1-D multirate processing techniques.

On the one hand, for a given circularly band-limited image that is sampled with the square lattice and the hexagonal lattice separately, the sampling intervals of the two lattices will have the same value on the one direction and only have different values on the other direction (the horizontal direction in this paper). On the other hand, for the square lattice sampling, the reconstruction filters are generally separable. Then, the common 2-D interpolation can be simplified into 1-D interpolation along the horizontal direction only [38]. We notice that, the 1-D interpolation itself is still shift-variant, for which the coefficients need to be computed for each interpolation, and thus the complexity as well as the computational costs are still a practical matter. In this paper, we propose to substitute the 1-D interpolation with the 1-D multirate processing, and the problem will be simplified to shift-invariant convolutions. Then, the square to hexagonal lattice conversion will be more efficient in both computation and implementation.

The remainder of this paper is organized as follows. For the better understanding the motivation, Section 2 gives the related theories, including the 2-D sampling, the 2-D separable processing, and the 1-D multirate signal processing. Then, Section 3 presents the proposed method in detail and Section 4 gives the experimental results and the discussion. Finally, Section 5 concludes the paper.