Full-color computational ghost imaging using a chromatic LED array and image interpolation

In computational ghost imaging based on a single-color LED array, the high-speed LED illumination module generates the illumination pattern P_i (i represents the ith pattern), which is projected on the object. A single pixel detector with no spatial resolution is used to collect and measure the total light intensity reflected or transmitted from the object. Given the light intensity distribution of the pattern P_i and the value of total light intensity S_i obtained by the single-pixel detector, a pixelated computational image whose pixel resolution equals to that of the illumination pattern, can be obtained by many algorithms with different complexity [1, 2, 8, 23]. One of the simplest of these algorithms is to add the weighted sum of the total light intensity value and the corresponding light intensity distribution.

$$\begin{equation}I = \sum\limits_{i{\text{ = 1}}}^N {{S_i} \cdot {P_i}} \end{equation} \tag{ 1 }$$

where N is number of measurements.

Hadamard basis [24], one of the common choices for single-pixel imaging [25, 26], are displayed as the illumination patterns on the LED array using a display strategy described in our previous work [21]. The process of sampling can be regarded as

$$\begin{equation}S = \left[ {\begin{array}{*{20}{c}} {{S_1}} \\ {{S_2}} \\ \cdots \\ {{S_n}} \end{array}} \right] = \left[ {\begin{array}{*{20}{l}} {{h_{11}}}&{{h_{12}}}& \cdots &{{h_{1n}}} \\ {{h_{21}}}&{{h_{22}}}& \cdots &{{h_{2n}}} \\ \vdots & \vdots & \vdots & \vdots \\ {{h_{n1}}}&{{h_{n2}}}& \cdots &{{h_{nn}}} \end{array}} \right] \times \left[ {\begin{array}{*{20}{c}} {{O_1}} \\ {{O_2}} \\ \cdots \\ {{O_n}} \end{array}} \right] = H \times O,\end{equation} \tag{ 2 }$$

where S= [S₁, S₂, ... S_n ,]^T is a column vector of intensity measurements, O= [O₁, O₂, ... O_n ,]^T represents the object with its elements in one-dimension. h_ij is the element of a n-order Hadamard matrix H at ith row and jth column.

Normally, the image of the object is reconstructed by perform matrix inversion on equation (2), i.e. I = O= H⁻¹ × S. However, because Hadamard matrix is Hermitian, meaning H= H⁻¹ = H^T. As a result, the reconstructed image is I = H × S, and can be rewritten in a similar form of equation (1) as

$$\begin{equation}I = \sum\limits_{i{\text{ = 1}}}^N {{S_i} \cdot {H_i}} \end{equation} \tag{ 3 }$$

where H_i = [h_{i 1}, h_{i 2}, ... h_in ,]^T is the ith row of the n-order Hadamard matrix H.

Considering about the fact that the switching time of a single LED is less than 1 μs and a digital full-color image consists of the red, green, and blue (RGB) channels, a trichromatic LED array module is developed to provide high-speed structured illumination for computational ghost imaging. The trichromatic LED illumination module, as shown in figure 1(a), consists of a field-programmable gate array (FPGA, Xilinx Spartan-6 XC6SLX100), a drive circuit and an LED array with 64 × 64 LED chips (EVERLIGHT, SMD 18-038/RSGHBHC1-S02/2T). The chips on LED array are arranged in basket-weave way as shown in figure 1(b), each LED is comprised of one red DIE, one green DIE and one blue DIE. The chromatic LED array gives three independent single-color illuminations and each monochromatic group of measurements are used to reconstruct RGB images of the object, respectively, which in turn are fused into a chromatic image.

Zoom In Zoom Out Reset image size

There is a main reason for choosing basket-weave arrangement for sampling. Compared to regular Cartesian arrangement of LED chips, as shown in figure 1(c), our proposed arrangement has some irregular characteristics while maintaining the original duty ratio in red and blue channel. When the sampling frequency is not more than twice the highest frequency of the image, moderate irregular sampling can expand the frequency band, thus improving the image quality. It means our proposed arrangement can have a better performance in terms of restoring colored high frequency.

When the chromatic LED array illuminates, the lens images the chromatic LED array on the object. The light emitted from three colored dies, respectively, samples the object in the corresponding spatial position of the object. However, for the 64 × 64 full-color sampling image obtained, its RGB values in a one pixel are from the same LED chip, that is, the digital sampling image does not contain the spatial location information of sampling. For example, there is a gap of 0.3 mm between two adjacent dies in one LED chip. Considering this situation, it is possible for us to reconstruct the 64 × 64 chromatic sampling image to reflect the spatial information and have an improvement on the quality of image. Modeling the 64 × 64 LED array and interpolating the 64 × 64 sampling image can help us reconstruct a more accurate image. Bilinear algorithm and bicubic algorithm are the most commonly used algorithms for digital image interpolation. However, before interpolation, these two algorithms simply regard the location of sampling points of all channels as being the center of the LED chip and do not take into account the different spatial locations of sampling points in R and B channels. Therefore, we design a targeted interpolation algorithm after modeling the LED array with basket-weave arrangement.

It is worth mentioning that the LED array can perform structured illumination up to 2.5 MHz after the drive circuit was optimized, which is five times faster than we achieved in our previous work [21].

Figure 2(a) shows the physical structure of four single LED chips with basket-weave arrangement, and we divide the chromatic LED array into grids. The length of gap between two adjacent LED chips is 0.375 mm, the size of each LED chip is 1 mm × 1 mm, the size of each die is 0.3 mm × 0.2 mm. We draw a line between the center of the green dies in two adjacent chips, divide it into equal parts, and take the length of each part as the side length of the grid dividing the LED array. After calculation, the whole LED array can be accurately modeled when N = 28 with a resolution of 1792 × 1792, and each colored die is accurately divided into 4 × 6 or 6 × 4. Importantly, the process of sampling the object is equivalent to adding the values of 4 × 6 or 6 × 4 grids and calculating the averaged value in the simulation, then we can get 64 × 64 sampling image.

Zoom In Zoom Out Reset image size

A targeted algorithm is proposed to interpolate the sampling image. There are two steps to interpolate. Firstly, for grids at the position of each colored die, they are filled with the same value of the 64 × 64 sampling image in the corresponding color channel. Note that we interpolate the RGB channels of the image separately. For example, we fill the grids in the red channel of interpolation image with the values of the red channel in the 64 × 64 sampling image. Secondly, for the other blank grids in different channels, instead of using common image interpolation method like bilinear algorithm and bicubic algorithm, we use the modified Shepard's method algorithm [27] for the special distribution of our sampling points.

In order to facilitate the presentation of our interpolation process, the display of grids is omitted and only schematic of red points is shown in figure 2(b). However, the interpolation is applied to all three channels. The black points are the center of the known sampling patches with the basket-weave distribution. For an unknown interpolated point P, with a radius R, only the nearest five sampling points D_i to P are used to evaluate the intensity value of P. d_i is the distance from D_i to P. Weighting factor s_i is defined as,

$$\begin{equation}{s_i}({d_i}) = \left\{ \begin{array}{*{20}{l}} \displaystyle {\frac{1}{{{d_i}}}}&{{\text{if}}\;\;0 < {d_i} \leqslant \frac{R}{3}} \\[10pt] \displaystyle {\frac{{27}}{{4R}}{{\left( {\frac{{{d_i}}}{R} - 1} \right)}^2}}&{{\text{if}}\;\;\frac{R}{3} < {d_i} \leqslant R} \\[10pt] 0&{{\text{if}}\;\;{d_i} > R} \end{array}. \right.\end{equation} \tag{ 4 }$$

In practice, R is operationally defined as 2N. The intensity value (red only in this instance) of P is determined as,

$$\begin{equation}f(P) = \left\{ {\begin{array}{*{20}{l}} {\left[ {\mathop \sum \limits_{i = 1}^5 {{({s_i})}^2}{z_i}} \right]/\left[ {\mathop \sum \limits_{i = 1}^5 {{({s_i})}^2}} \right]}&{{\textrm{if}}{d_i} \ne 0}\\ {{z_i}}&{{\textrm{if}}{d_i} = 0} \end{array}} \right.\end{equation} \tag{ 5 }$$

where z_i is the measured intensity value of D_i . Since our targeted algorithm for image reconstruction reflects the spatial position information of the sampling points, the resolution of the interpolated image is improved theoretically compared with bilinear algorithm and bicubic algorithm, especially for colored image with complex texture. This is because in the red channel, the sampling values at the position of red die have big difference from the values at the position of green. Similarly, in the blue channel, the sampling values at the position of blue die have big difference from the values at the position of green.

The experimental system, as shown in figure 3, includes the chromatic LED array, a camera lens (Sigma, f = 30 mm, F/1.4D, DC) to project the patterns onto the object, a single-pixel detector (Thorlabs PDA100A-EC) to measure the transmitting light intensities collected by a singlet lens (BOCIC, OCL3-050-050, f = 50 mm, F/1), and a computer to reconstruct the images using the intensity data transferred from a high dynamic range digitizer (PicoScope 6404D). The evolutionary compressive sensing algorithm is applied during the data acquisition and the image reconstruction, the frame-rate of the experimental system reaches 83 frame per second.

Zoom In Zoom Out Reset image size

In the simulation, firstly, we scale an original high-resolution image I_o(i, j) with a resolution of 3840 × 3840 into an image with a resolution of 1792 × 1792 using bicubic algorithm, this step is to simulate the process of dividing an image of object with infinite resolution into grids when N = 28. Secondly, the 1792 × 1792 image is sampled to obtain a 64 × 64 sampling image using the method mentioned in section 2. Finally, we interpolate the 64 × 64 image with our targeted algorithm to obtain the interpolation image I_r(i, j) with a resolution of 3840 × 3840 (N = 60). RMSE is introduced as the evaluation indexes of image quality for each channel as

$$\begin{equation}{\text{RMS}}{{\text{E}}_{{\text{channel}}}} = \sqrt {\frac{1}{{m \times n}}\sum\limits_{i = 1,\,j = 1}^{m,n} {{{({I_{\text{r}}}(i,j) - {I_{\text{o}}}(i,j))}^2}} } \end{equation} \tag{ 6 }$$

where m = n = 3840 are the dimensions of the image and channel are the RGB channels of the image. In our work, we use the averaged RMSE of the three monochromatic image RMSEs to evaluate the quality for full-color image.

$$\begin{equation}{\text{RMS}}{{\text{E}}_{{\text{RGB}}}}{\text{ = (RMS}}{{\text{E}}_{\text{R}}}{\text{ + RMS}}{{\text{E}}_{\text{G}}}{\text{ + RMS}}{{\text{E}}_{\text{B}}}{\text{)/3}}{\text{.}}\end{equation} \tag{ 7 }$$

Thirty-five different original high-resolution images with high frequency information are sampled and reconstructed using our targeted algorithm. Meanwhile, we use Cartesian sampling for comparison. Figure 4(a) shows the RMSEs of the reconstructed image based on these two kinds of sampling ways. The RMSEs are in descending order of Cartesian arrangement RMSE values (red square dots). There is a 4.6% reduction in averaged RMSE when compared to Cartesian arrangement which is coincident with the conclusion that basket-weave arrangement has a better performance than Cartesian arrangement for full-color pictures with high frequency information. Figure 4(b) demonstrates two examples of reconstructed results with their RMSEs. Besides RMSE measurement, our proposed arrangement slightly reduces artifacts and get a better visual effect.

Zoom In Zoom Out Reset image size

Thirty-five different 3840 × 3840 images with complex details are chosen and sampled, then four algorithms are applied on each image for reconstruction. Figure 5(a) shows the RMSEs of the reconstruction images using four different interpolation algorithms, which are mosaic algorithm, bicubic algorithm, bilinear algorithm, and our proposed algorithm. The RMSEs are in descending order of Mosaic RMSE values (red square dots). Our targeted algorithm yields the lowest averaged RMSE value among the four groups' data with a drop about 2%, which is coincident with the conclusion that our targeted algorithm improves the quality of full-color image with complex details by reflecting the spatial information. Figures 5(b) and (c) demonstrate two examples of the reconstructed results for visual case and detailed numerical information.

Zoom In Zoom Out Reset image size

Experiments are performed with a single-pixel system using an LED array with basket-weave arrangement. We scale the resolution of the original high-resolution image to 1792 × 1792 and print the image as the object in the experimental system. For each monochromatic 64 × 64 image reconstructed, 16 384 measurements (4096 × 2 × 2, 4096 refers to 64 × 64 pixel resolution resulting images, the first ×2 refers to differential measurement [19], the second ×2 refers to display strategy [21]) are performed. We interpolate the monochromatic image using four different interpolation algorithms and calculate the RMSE between the original high-resolution image and the interpolated ones. At the same time, an LED array with Cartesian arrangement is also developed for comparison. Two examples of the reconstructed results and their numerical information are shown in figure 6. There is an improvement in RMSE for basket-weave sampling over Cartesian arrangement. Besides, the RMSE value of the image reconstructed using the proposed algorithm is the smallest. Experimental results are consistent with the simulation, demonstrating the feasibility of our proposed idea.

Zoom In Zoom Out Reset image size

It is worth mentioning that three monochromatic images are normalized before fusing into a full-color resulting image. However, the minimum value and the maximum value may exceed the true range of original image values because of the existence of noise, resulting in the problem of incorrect color matching and incorrect values of the resulting image. To address this problem, we draw two 300 × 300 patches in the original image with different values of 200 and 50 used for calibration. After imaging, the resulting image is obtained with two 5 × 5 patches in corresponding location, as shown in figure 6. In the process of linear normalization, we average the pixel values, respectively, in patches and make them 200 and 50 through linear normalization. The reason for choosing 200 and 50 is to consider that some values in the resulting image are below 0 or above 255 after normalization.