2.1.

Forward Imaging of the FPM

To demonstrate the importance of LED position error correction, the basic concepts of FPM imaging model are introduced. The typical FPM system setup is shown in Fig. 1(a), which mainly includes CCD, tube lens and low NA microscope objective lens, sample, and the LED array. The LED array is shown in Fig. 1(b), which is an enlarged view of the parts within the black dotted box in Fig. 1(a). The distance between LEDs and sample is $h$. If the light emitted by the LED is an ideal plane wave, and its incident wave vector $v_x^{m,n},v_y^{m,n}$ can be expressed as

Fig. 1

Schematic diagram of a typical FPM imaging process. (a) The classic setup of FPM. (b) The illumination wave-vector of the LED located in the circle.

The LEDs are turned on sequentially to realize the illumination of the sample from various incident angles. Synchronously, the LR intensity image of the sample illuminated at the corresponding angle is collected by the CCD. This is explained further using mathematics. In this paper, it is assumed that the sample is very thin. When the LED located at $(m,n)$ the is turned on, under the action of its illumination wave vector$v_x^{m,n},v_y^{m,n}$, the complex amplitude of the sample is$o(x,y)$written as

When the LED in different positions is turned on, the LR intensity image sequence of the corresponding samples is saved. In this way, the sample spectra in different sub-regions can bypass the physical limitations of the imaging system. Then, the FPM reconstruction algorithm that iteratively switches between the spatial domain and the Fourier domain stitches these spectra together correctly. After iterative convergence, the high-resolution complex amplitude result of the object is obtained.

2.2.

Analysis of LED Positional Misalignment

In the FPM iterative reconstruction process, it is necessary to match the space position coordinates of the real LED and the sub-aperture spectrum position coordinates. Therefore, the position coordinates of the LED directly affect the accuracy of spectrum splicing and determine the quality of the complex amplitude of the reconstructed sample. There are two sources of LED position error: the first one is the deviation between the actual LED position and the design position produced during the manufacturing process, and the second is the relative position error of the LED array and the optical axis during installation. The offset between LEDs is a tiny random variable that cannot be ignored, which will increase the uncertainty of the reconstruction result. The latter is the global offset of the LED array, which will cause the quality of the reconstruction result to drop rapidly.

Figure 2 shows the simulated amplitude and phase results of LED position error, respectively. The high-resolution amplitude and phase images used in the simulation are shown in Fig. 2(a). Figure 2(b) shows the reconstructed image without LED position error, which is not much different from Fig. 2(a). As mentioned above, there are two types of LED position error. Figure 2(c) is the reconstructed amplitude and phase image with the slight offset between LEDs. It can be seen that this tiny, random LED position variable will have a corresponding impact on the reconstruction quality. In addition, the reconstructed amplitude and phase image, which is demonstrated in Fig. 2(d), corresponds to the second type of LED position error. The relative position error of the LED array and the optical axis will cause the global shift of the spectrum, which will have a great influence on the reconstruction result. The combined effect of these two errors is shown in Fig. 2(e), which can lead to worse reconstruction results of intensity and phase. Thus, it is necessary to correct these two types of position errors in FPM.

Fig. 2

The simulation results of LED position misalignment in the FPM. (a) The original HR amplitude and phase images, respectively. (b) Without LED position error, the HR amplitude and phase graphics restored by traditional algorithms. (c) The reconstructed amplitude and phase image with the slight offset between LEDs. (d) In the case of the relative position error of the LED lighting module and the optical axis, the reconstructed intensity and phase image. (e) The reconstructed intensity and phase simulation results, include (c) and (d) the two LED position errors.

2.3.

Proposed Misalignment Correction Algorithm

Before introducing the tcFPM algorithm process, the position distribution model of the LED array is defined. In the simulation and experiment of this paper, the LED array is a non-linear annular distribution, which can effectively avoid the pathological problem of the grid,^26,27 as shown in Fig. 3(a). The LED array is composed of a central LED and five LED rings. The number of LEDs included in each LED ring is $N$, where $N\in \{\mathrm{8,12},\mathrm{16,16},16\}$. The distance between two adjacent LED rings is $r=5\mathrm{mm}$. For example, the LED marked in red in Fig. 3(a), which is located at the $m=3(m\in \{\mathrm{1,2},\mathrm{3,4},5\})$, $n=3$$(n\in \{\mathrm{1,2},\mathrm{3,4},\mathrm{5,6},\mathrm{7,8},\mathrm{9,10},\mathrm{11,12}\})$. In Fig. 3(b), the black dashed axis represents the ideal space coordinates, and the ideal LED spatial distribution is demonstrated by white transparent LEDs. The real coordinates and the spatial distribution of LEDs are indicated by the red solid line coordinate axis and the red transparent LED. The $\mathrm{\Delta }{x}_{m,n}$ and $\mathrm{\Delta }{y}_{m,n}$, respectively, represent the deviation of the LED position in two directions. In addition, the displacement error of the relative position of the LED array and the optical axis is distinguished by $\mathrm{\Delta }\theta ,\mathrm{\Delta }X$, and $\mathrm{\Delta }Y$. Therefore, the coordinates of the LED including the deviation of the LED position and the error of the relative position of the LED array and the optical axis can be expressed as

Fig. 3

The position distribution model of the LED array. (a) The schematic diagram of the nonlinear distribution of LEDs. (b) The parameter representation of the LED array misalignment model.

Aiming at the above two different LED position errors, we propose tcFPM, which is a system scheme for initial correction to obtain the global offset of the LED array, and then precise correction to obtain the slight offset between LEDs. The algorithm flow chart of tcFPM is illustrated in Fig. 4.

Fig. 4

The flow chart of tcFPM method.

The algorithm process is mainly composed of two parts: the blue dashed frame in Fig. 4 is the correction process of the global offset of the LED array, and the correction process of the slight offset between LEDs is marked with the brown dashed frame. The LR image corresponding to the LED array in the bright field range contains a large amount of sample information, which has a greater impact on the image quality of FPM. Therefore, First, the global offset of the LED array in the bright field is preliminarily corrected. In this article, the initial correction LED range, which includes a center LED and two adjacent LED rings. The reason why the range is selected is to be able to achieve a large-scale global correction capability and to ensure the robustness of the algorithm.

First, the global error variable $\mathrm{\Delta }\theta ,\mathrm{\Delta }X,\mathrm{\Delta }Y$, the Fourier spectrum of the HR sample image $O_0(v_x,v_y)$and the pupil function $P_0(v_x,v_y)$are initialized. Generally, the initial value of the global error variable is set to$\mathrm{\Delta }\theta =0,\mathrm{\Delta }X=0,\mathrm{\Delta }Y=0$. When the bright field LEDs are simultaneously illuminated, the corresponding sample image is collected. The obtained sample image is subjected to bilinear interpolation operation, and then the result after Fourier transform is used as the initial sample spectrum $O_0(v_x,v_y)$. The pupil function $P_0(v_x,v_y)$ is initialized as a circular low-pass filter, and the radius of the circle is$2\pi \mathrm{NA}/\lambda$, where NA is the numerical aperture of the brightfield microscope objective and $\lambda$ is the central wavelength of the illumination.

Next, the incident wave vector $v_X^{m,n},v_Y^{m,n}$ is calculated according to Eqs. (1) and (5), which includes the LED global offset error$\mathrm{\Delta }\theta ,\mathrm{\Delta }X,\mathrm{\Delta }Y$. Based on Eqs. (2) and (3), the spectrum information${\mathrm{\Psi }}^{m,n}(v_x,v_y)$and intensity information ${\psi }^{m,n}(x,y)$ of the sample after the objective pupil function can be obtained as follows:

Further, we compare the calculated intensity distribution $o^{m,n}(x,y)$ with $I_{S_1}^{m,n}(x,y)$ using the following formula to find the smallest cost function $E_1$.

The brown dashed box in Fig. 4 shows the process of correcting the slight offset between LEDs. The spatial position coordinates of the LED are updated, which is uses the global offset obtained by the preliminary calibration. The incident wave vector $v_x^{m,n},v_y^{m,n}$ including the LED random position deviation $\mathrm{\Delta }{x}_{m,n},\mathrm{\Delta }{y}_{m,n}$ is calculated according to expressions Eqs. (1) and (5). Before the iteration starts, the Fourier spectrum $O_i(v_x,v_y)$ of the high-resolution sample and the pupil function $P_i(v_x,v_y)$ need to be initialized. Similarly, the spectrum information ${\mathrm{\Psi }}^{m,n}(v_x,v_y)$ and intensity information ${\psi }^{m,n}(x,y)$ of the sample after the objective pupil function can be written as:

Further, following the principle of phase retrieval, the phase information of the target image remains unchanged. The amplitude component of the target image is replaced by the square root of the actual measurement value. In this way, the updated LR target image ${\varphi }_i^{m,n}(x,y)$ and its Fourier spectrum ${\mathrm{\Phi }}_i^{m,n}(x,y)$ are calculated as follows:

Next, we update the spectral information in the corresponding sub-aperture in the HR spectrum of the object and the pupil function:²⁵

Finally, the calculated intensity distribution ${\psi }^{m,n}(x,y)$ is compared with $I_{S_2}^{m,n}(x,y)$ using the following formula. The cost function for finding LED position error $\mathrm{\Delta }{x}_{m,n},\mathrm{\Delta }{y}_{m,n}$ is expressed as

The tcFPM algorithm carries out a two-step correction process according to the difference of LED position error, which has good stability. The problem of the LED position error is solved, and a better high-resolution reconstruction result of the sample can be obtained. The simulation in Sec. 3.1 and the experiment in Sec. 3.2 will prove that the reconstruction results of tcFPM can be more accurate and stable for a set of LR images containing position errors.

3.1.

Simulation

In this section, the effectiveness of the tcFPM scheme is further illustrated by calculation simulation. The system parameters used in the simulation are consistent with the component parameters adopted in the experiment. The NA of the microscope objective is 0.3, and its magnification is 10. The pixel size of the CCD is$2.75\mu \mathrm{m}$, and the simulated image size is $512\times 512\mathrm{pixels}$. The center wavelength of the LED is set to 632 nm, and its spatial distribution is consistent with the model mentioned above. The distance between the LED array plane and the sample is 80 mm. The slight offset between LEDs and the global offset of the LED array are simulated separately. These two errors are represented by random variables uniformly distributed within a certain range, to ensure the adaptability and stability of the tcFPM algorithm. The reconstruction process of simulation and experiment are performed in MATLAB R2020a.

The intensity and phase images of the sample used in the simulation are shown in Fig. 2(a). In the simulation, 69 LR intensity images are used. During the reconstruction of tcFPM, the number of iterations of the global correction process is 6, and the number of iterations of the precise correction process is 14. The reconstruct results of the traditional FPM, pcFPM scheme and tcFPM are demonstrated in Figs. 5(a), 5(b), and 5(c), respectively. It is shown that with the same LED array position error, the high-resolution intensity image and phase recovered by the tcFPM proposed in this paper have more details.

Fig. 5

The Results of simulation recovery of different algorithms. (a) The intensity and phase images restored by traditional FPM. (b) The intensity and phase images reconstructed with pc-FPM. (c) Intensity and phase images calculated with tcFPM.

The reconstruction intensity, phase correlation coefficient and RMSE of each iteration are shown in Figs. 6(a), 6(b), 6(c), and 6(d), respectively. The iterative process of the traditional FPM reconstruction algorithm is represented by the yellow curve in Fig. 6. The blue curve in Fig. 6 demonstrates the iterative reconstruction process of pcFPM. The iterative reconstruction process of tcFPM is clearly marked with the red curve, in Fig. 6. In addition, the ideal iterative reconstruction result of the known LED error is shown by the green curve in Fig. 6. In Figs. 6(a) and 6(b), the correlation coefficient of the intensity and phase of the traditional FPM iteration process (yellow line) is poor. The correlation coefficients of intensity are obviously optimized in the iterative process of pcFPM (blue line), but the result of the phase recovery is relatively poor. That the result of tcFPM (red line) is more stable, compared to traditional FPM algorithm and the pcFPM. As shown in the RMSE results of Figs. 6(c) and 6(d), it can well perceive that the result of tcFPM is more stable and accurate than the original FPM recovery algorithm and pcFPM position correction method. From what has been mentioned above, the tcFPM proposed in this paper is very close to the ideal iterative recovery process, which means that the algorithm is very stable.

Fig. 6

(a) The intensity correlation coefficient between the restored complex amplitude image and the original high-resolution image, corresponding to four different reconstruction iteration processes. (b) The phase correlation coefficient between the restored complex amplitude image and the original high-resolution image, corresponding to four different reconstruction iterations. Panels (c) and (d) show the intensity and phase reconstruction accuracy of versus iteration time in different reconstruction processes.

3.2.

Experiment

To prove the feasibility and performance of the tcFPM scheme, an experimental system was developed with reference to Fig. 1, as shown in Fig. 7. The main parameters of the system are as follows: the LED array (MP-1616, $1.6\mathrm{mm}\times 1.6\mathrm{mm}$, center wavelength is 593 nm, Luminus), a microscope objective lens ($10\times$, $\mathrm{NA}=0.3$, Nikon, Japan), a tube lens ($1\times$, focal length = 200 mm, Nikon, Japan), and a monochrome CCD camera with a maximum frame rate of 55 fps ($\mathrm{acA}2040\text{-}55\mathrm{um}$, $2048\times 1536\mathrm{pixels}$, Basler, Germany). The LED array is controlled by Arduino Mega 2560, and the CCD is triggered to collect the corresponding LR image data. Similar to the simulation, we use this system to collect 69 LR images. Without loss of generality, we choose the region of interest of the LR image to verify the performance of the system.

Fig. 7

The experimental system of the tcFPM.

First, the classic USAF resolution board is selected as the experimental sample. According to the FOV of the collected LR image, a part of USAF data is selected for reconstruction operation, the results are shown in Fig. 8. The LR images are captured under the illumination of LEDs in the brightfield area, which is used as a contrast image. The interpolated result of the image is as indicated in the Fig. 8(a). Figure 8(b) shows an enlargement of the green frame area in Figure 8(a). The result of using traditional FPM recovery without correcting the LED misalignment is shown in Fig. 8(c). Figure 8(d) shows the recovery of the pcFPM program. The reconstruction of tcFPM proposed in this article is demonstrated in Fig. 8(e).

Fig. 8

The experimental results of the USAF resolution target. (a) The full field FOV of LR image collected by CCD. The segment (b) is the enlargement of the green frame in image (a). (c) The experimental results of traditional FPM (without LED position correction). (d) The recovery results corresponding to the pcFPM scheme. (e) The corresponding reconstruction results of the tcFPM.

To show the comparison of the reconstruction results, a line of intensity data of the third element of the ninth group [as demonstrated by the solid green line in Fig. 8(b)] is plotted, as shown in Fig. 9(a). The green curve in Fig. 9(a) shows that the LR intensity images collected by the CCD cannot be distinguished even after interpolation. Reconstruction through traditional FPM can improve the resolution of the result [the yellow line in the Fig. 9(a)], but the reconstructed result has low contrast and high background noise, which is the result of LED position errors. The pcFPM scheme can help improve the reconstruction resolution [the blue line in the Fig. 9(a)]. Compared with pcFPM, the tcFPM can effectively improve the resolution of the reconstructed image [the red line in the Fig. 9(a)]. The reconstruction intensity correlation coefficient and RMSE of each iteration are shown in Figs. 9(b) and 9(c), respectively. In Fig. 9(b), it can be seen that the intensity correlation coefficient of the traditional FPM iteration process [the yellow line in the Fig. 9(b)] has little improvement. Compared with FPM and pcFPM, the intensity correlation coefficient of tcFPM [the red line in the Fig. 9(b)] performs well. In addition, the iterative process [the blue line in the Fig. 9(b)] of the pcFPM scheme is unstable. The iterative process of tcFPM is relatively stable. As shown in Fig. 9(c), the RMSE of the experimental results shows that the result of tcFPM is more stable and accurate, which is compared to the FPM algorithm without LED position correction and pcFPM method. These experimental results are consistent with the previous simulation process.

Fig. 9

(a) The intensity curve of the third element of the ninth group [shown as the green line in Fig. 8(b)] corresponds to (b), (c), (d), and (e) in Fig. 8, respectively. (b) The intensity correlation coefficient of the reconstructed high-resolution image corresponds to three different iterative processes. (c) The intensity reconstruction accuracy of versus iteration time in different reconstruction processes.

Additionally, the stained human blood cell samples are measured and reconstructed using different algorithms. Similarly, the sample image in the imaging FOV is shown in Fig. 10(a). Figures 10(b), 10(c), and 10(d) show the AOI magnified areas of the HR intensity and phase images reconstructed by traditional FPM, pcFPM, and tcFPM, respectively. Figure 10(e) shows the intensity curve of the red blood cell cross-sectional profile at the solid green line in Figure 10(a). The result of traditional FPM restoration has serious artifacts due to the uncorrected LED position error. The results of pcFPM recovery significantly improved the impact of LED misalignment. As shown by the red intensity curve, compared with pcFPM, the cell edges in the HR image recovered by tcFPM are clearer. Overall, the reconstructed image provides more different details and eliminates obvious background noise. The results of biological samples also verified the reconstruction accuracy and stability of tcFPM.

Fig. 10

The experimental recovery image of the human blood biological sample smear. (a) LR image with full FOV. (b) The amplitude and phase of high-resolution images reconstructed by traditional FPM. (c) The AOI of the high-resolution complex amplitude image reconstructed by pcFPM. (d) The amplitude and phase of high-resolution images reconstructed by tcFPM. (e) The intensity curve of the cross-sectional profile of red blood cells.