1.

Introduction

In biomedical applications, it is desirable to obtain complex images with both high resolution and wide field. Regardless of advancements in sophisticated mechanical scanning microscope systems and lensless microscopy setups, the modification of conventional microscopes to obtain ideal high-resolution results has been a hot topic of recent research work. Inspired by ptychography,^1,2 Fourier ptychography (FP) in particular is a simple and cost-effective analytical method for this application.^3–7 As a newly proposed computational imaging method, FP integrates principles of phase retrieval^8,9 and aperture synthesizing^10,11 to achieve both high resolution and large field of view (FOV). By introducing a programmable light-emitting diode (LED) array as an angle-varied coherent illumination source, higher-frequency information is shifted into the passband of the objective lens and detected by the image sensor. Then FP can stitch these images together in the Fourier domain to enlarge the system bandwidth via aperture synthesizing. The lost phase information is further recovered via the phase retrieval technique using intensity-only measurements. As such, FP can reconstruct a sample with both high-equivalent numerical aperture (NA) and large FOV. So far, FP has received increasing attention over the past few years, and many types of research have been proposed on the system setup and reconstruction algorithm to correct various system aberrations,^12,13 provide robust optimization methods against noise,^14,15 report innovative system setup designs,^16,17 and so on.^18–20

Recently, convolutional neural networks (CNNs) have been proven to reliably provide inductive answers to the inverse problem in computational imaging,¹¹ and many biomedical imaging problems have been solved with CNN, such as computed tomography,²¹ image super-resolution,²² and holography.^11,23 The purpose of FP is to converge a high-resolution target from the acquired image sequence. It is natural to introduce the idea of CNN into this inverse problem and learn an underlying mapping from the low-resolution input to a high-resolution output.^24–27 However, there exist some specific issues in biomedical applications. First, different from natural image applications, it is hard for biomedical imaging problems to access large amounts of images. Therefore, supervised CNNs, which rely on large datasets, have encountered obstacles in training data. Second, computational imaging methods like FP are sensitive to system parameters. Once the system setup is changed or the system aberration is introduced, the performance of the deep-learning-based network will be degraded. Toward the first dilemma, Zhang et al.²⁷ generated datasets with simulations to train the network. However, there is a lack of related networks to solve the second issue. To address these dilemmas properly, it is effective to add constraints based on the application characteristics. Jiang et al.²⁸ proposed a physics-based framework that they established using a forward imaging model of FP via TensorFlow and utilized the back-propagation to optimize the high-resolution sample. This method uses the imaging model of FP as the constraint and solves the above two issues. The neural network is only utilized to imitate the traditional iterative recovery process. Following this idea, related methods can work with limited data^29,30 and can work stably toward system aberration.^31–34 However, these works are still essentially the iterative-based algorithm, and the automatic differentiation (AD) property of the neural network is not fully utilized. It would be much more desirable to design a new neural network to further degrade the noise in the reconstruction and estimate the optical aberration with higher accuracy.

In this paper, we report a neural network model to solve Fourier ptychographic problems. The Zernike aberration recovery and total variation (TV) constraint are introduced as the augmenting modules to ensure aberration corrected reconstruction and robustness against noise. As such, we name this model the integrated neural network model (INNM). INNM is essentially a TensorFlow-based trainable network mimicking the iterative reconstruction of FP. We model the forward imaging process of FP via TensorFlow at first. To estimate the optical aberration properly, the optical aberration of the employed objective lens is modeled as a pupil function in our model and optimized along with the sample through backpropagation. Then we introduce the alternate updating (AU) mechanism to achieve better performance and use the Zernike mode to make a robust and proper aberration estimation. As such, our method can recover the optical aberration with high accuracy and achieve better performance simultaneously. To further eliminate the noise, we incorporated the TV on both the amplitude and phase of the sample image. It turns out that this application can improve the image quality. Our experiments demonstrate that INNM outperforms other methods with higher contrast, especially when validated in a severe aberration condition.

This paper is structured as follows. In Sec. 2, we discuss the fundamental principles and reconstruction procedure of FP. In Sec. 3, we describe the model structure and introduced mechanisms step by step. Then in Sec. 4, we validate our method on both simulated and experimental datasets under variant acquisition conditions and analyze the benefits of introduced methods in detail. Finally, we provide concluding remarks and discuss our ongoing efforts in Sec. 5.

2.

Fourier Ptychographic Microscopy

As a classic analytical method, FP is mainly composed of the explicit forward imaging model and the decomposition procedure. Considering the generalized FP schematic diagram setup shown in Fig. 1(a), the sample is successively illuminated by plane waves from the LED matrix at different angles. The exit waves are then captured by the image sensor (CCD) through the objective lens. Sequentially lighting distinct LEDs on the matrix, FP can obtain a low-resolution intensity image sequence to recover a high-resolution complex one.

Fig. 1

Fundamental principles of FP: (a) the schematic diagram of the FP experimental setup and the physical comparison and (b) the iterative decomposition procedure with pupil recovery.

The forward imaging procedure is shown in the left column of Fig. 1(a). We denote the thin sample as its transmission function $o(\mathbf{r})$, where $\mathbf{r}=(x,y)$ represents the 2D spatial coordinates with its Fourier expression as $\mathbf{k}=(k_x,k_y)$. When obliquely illuminated by the $n$’th monochromatic LED, the exit wave at object plane can be denoted as $o(\mathbf{r})\odot \exp (i{\mathbf{k}}_n)$, where $\odot$ denotes the element-wise multiplication and ${\mathbf{k}}_n=(k_x^n,k_y^n)$ denotes the $n$’th wave vector corresponding to the angle of incident illumination. The final wave captured by the CCD can be expressed as

As for the decomposition procedure shown in Fig. 1(b), FP will first initialize a high-resolution sample image. Then FP will shift the confined CTF into distinct apertures and utilize a phase retrieval method called alternating projection (AP) to obtain a self-consistent complex image. In AP, the pinned aperture in the Fourier domain will be updated by keeping its spatial phase unchanged and replacing its spatial amplitude with the square root of the corresponding captured image. In addition, since optical aberration is a common issue in practical applications,³⁵ it is valuable to embed an extra pupil recovery called ePIE⁴ after AP to compensate for this interference. The whole decomposition process with pupil recovery is shown in Fig. 1(b) (the red flowchart denotes the original FP and the green one denotes the extended pupil recovery).

3.

Method

3.1.

Framework of INNM

To solve FP problems via the neural network, we model the forward imaging process via TensorFlow, which could obtain the sample and optical aberration simultaneously. Figure 2 illustrates the detailed pipeline workflow and the framework of our method. As shown in Fig. 2(a), the upsampled central measurement and the standard CTF without pupil aberration are set to the initial guess of the sample and CTF, respectively. Then the optimized targets can be obtained through several training stages. Each training stage consists of a few epochs (for simplicity, only one epoch is shown). As shown in Fig. 2(b), we take the pair of captured image $I_{\ln }(\mathbf{r})$ and corresponding plane wave vector ${\mathbf{k}}_n$ as a single batch. In each epoch, all batches are fed into the model, and the model parameters are updated through backpropagation. To avoid confusion, we termed these batches as varying angle illumination units (VAIUs) and represented them as a complete epoch.

Fig. 2

Illustration of the INNM framework: (a) schematics of pipeline procedure with the AU mechanism; (b) schematics of a training epoch; and (c) the basic framework of INNM with embedded pupil recovery.

The detailed framework of our model with the embedded pupil recovery is shown in Fig. 2(c). This model is transferred from the forward imaging process in the Fourier domain, and the whole process can be formulated as follows:

Eq. (3)

$${\phi }_{\ln }(\mathbf{k})=O(\mathbf{k}+{\mathbf{k}}_n)\odot C(\mathbf{k}),$$

Eq. (4)

$${\phi }_{hn}(\mathbf{k})=\mathcal{F}\{\sqrt{I_{\ln }(\mathbf{r})}\odot \angle \{{\mathcal{F}}^{-1}\{{\phi }_{\ln }(\mathbf{k})\}\}\},$$

where $O(\mathbf{k})$ denotes the Fourier function of the sample, $\angle$ denotes the phase, ${\phi }_{\ln }$ and ${\phi }_{hn}$ represent the original and updated apertures in Fourier domain during the AP.

Reflected in Fig. 2(c), the sample and CTF are naturally treated as two-channel (with the real and imaginary parts) learnable feature maps. Then the sample $O(\mathbf{k})$ in the Fourier domain is forward propagated through the well-designed framework following Eq. (3), where the sample is first shifted according to ${\mathbf{k}}_n$ and multiplied with the CTF $C(\mathbf{k})$. Then the generated spectrum ${\phi }_{\ln }(\mathbf{k})$ is transmitted via the AP defined in Eq. (4) with the preupsampled input $I_{\ln }(\mathbf{r})$. Thus we can generate the updated spectrum ${\phi }_{hn}(\mathbf{k})$. Based on the phase retrieval mechanism, these two exit wave series of spectra ${\phi }_{\ln }(\mathbf{k})$ and ${\phi }_{hn}(\mathbf{k})$ should contain the same frequency information when converged. Therefore, the whole framework can obtain optimized results by minimizing the differences between these spectra. The first loss function used in INNM can be expressed as

Eq. (5)

$$\text{loss}={\mathrm{\Sigma }}_{n=0}^N{\Vert {\phi }_{hn}(\mathbf{k})-{\phi }_{\ln }(\mathbf{k})\Vert }_2^2,$$

where L2-norm is used to measure the differences and $N$ denotes the number of VAIUs.

As for the detailed composition of the network, self-designed fixed -layers and multiplication layers are widely used to perform operations such as AP. More specifically, the -layer can be flexibly customized to define operations, such as using the roll function in TensorFlow to perform the shifting operation. As such, these layers can be designed to perform the forward imaging model of FP in TensorFlow. We should note that INNM is defined in the complex field, and all the layers consist of the real and imaginary parts. For example, we should rewrite Eq. (3) in the complex domain as

Eq. (6)

$$\begin{gathered} {\phi }_{\ln }(\mathbf{k})=\{O_r(\mathbf{k}+{\mathbf{k}}_n)+i·O_i(\mathbf{k}+{\mathbf{k}}_n)\}\odot \{C_r(\mathbf{k})+i·C_i(\mathbf{k})\} \\ =(O_r\odot C_r-O_i\odot C_i)+i·(O_r\odot C_i+O_i\odot C_r), \end{gathered}$$

where the subscripts $r$ and $i$ denote the real and imaginary parts, respectively. At last, customized network layers are used to represent the learnable targets, like sample and CTF. As such, the model is lightweight with limited trainable parameters (sample and CTF). After backpropagation, we can extract parameters in the learnable layers shown in the blue boxes of Fig. 2(c) to get the optimized results. Our Jupyter notebook source code for this algorithm has been made available; a link is provided in the Data, Materials, and Code Availability section, below.

4.

Experiments

We validated our INNM on both simulated and experimental datasets. For comparison, we mostly compared our results with the widely used method ePIE,⁴ which is a robust and effective FP method with pupil recovery. We also compared with other methods over the experimental dataset, such as Jiang’s method²⁸ and AS.⁴²

4.2.

Results on Simulated Datasets

The ground truth of the simulated dataset is shown in Fig. 3(f). The cameraman and street map are used as the amplitude and phase images. The defocus aberration is set to be the optical aberration. The goal of reconstructed methods is to extract the sample and CTF from 225 $32\times 32$ intensity images.

Fig. 3

Comparison of recovered results and some decomposed Zernike amplitudes. (a)–(f) Recovered results of the sample and pupil function under different methods (the amplitude are normalized into 0 to 1). (g) Decline curve of the INNM. (h) A scatter plot of some decomposed Zernike polynomial coefficients $c_l$ (piston coefficient $Z_0^0$ is not present).

4.2.1.

Alternate updating process

For clarity, we call the framework of INNM in Sec. 3.1 as B-INNM. Then recovered results under different settings of INNM are represented in Figs. 3(a)–3(d), and INNM stands for B-INNM with all the attributes. Considering the influence of the AU, we compare the recovered results shown in Figs. 3(a) and 3(b). It is apparent that the former sample images are quite ambiguous, and the estimated aberration is also meaningless. Instead, the latter one contains fewer artifacts and achieves higher fidelity, owing to the introduction of the AU mechanism. In addition, we represent the training curve with AU shown in Fig. 3(g). Based on the above discussion, we can demonstrate that AU can lead to the expected optimum by controlling the optimized target and compensate for the aberration to some extent. However, the recovered aberration contains unexpected periodic speckles [Fig. 3(b3)]. This problem is caused by the periodic grid sampling pattern in the Fourier domain and would degrade the high-resolution FP reconstruction.¹⁸ To solve this problem, additional principles are incorporated.

4.2.2.

Zernike polynomial function

As shown in Fig. 3(c), the greatest contribution made by the Zernike mode is the proper correction of the aberration. Compared to the aberration shown in Fig. 3(b3), the new aberration with the Zernike modes completely removes the pollution of raster-grid speckles. Furthermore, the decomposed Zernike coefficients under various conditions are illustrated in Fig. 3(h). We can find that INNM can fit the polynomial coefficients with high accuracy in all order terms, whereas B-INNM with AU cannot restore these parameters. As such, the biased coefficients in tilt Zernike modes ($Z_1^{-1}$ and $Z_1^1$) lead to the raster-grid speckles in Fig. 3(b3). When this issue is resolved, the Zernike mode helps to achieve the higher reconstruction quality shown in Fig. 3(c). Such a phenomenon proves that taking the optical aberration as a power series expansion is better for optimization than taking it as a whole. In addition, we can further find that the decomposed coefficient amplitudes obtained from ePIE are differential from the ground truth in the tilt Zernike modes. This disadvantage will hinder the high-quality reconstruction shown in Fig. 3(e).

4.2.3.

Total variation loss

By introducing the above two mechanisms into the B-INNM, our method can predict the optical aberration with high accuracy and eliminate its interference. Here we will provide the results of integrating the TV loss in the final objective function. In FP, the recovered amplitude and phase can always crosstalk with each other and cause background artifacts. As shown in Fig. 3(c), even though we properly estimate the aberration, the recovered sample images still suffer from this noise. To degrade unexpected noises, we introduce the TV regularization and the modified result is shown in Fig. 3(d). It is obvious that the background noise is greatly degraded and achieves higher image quality. In the end, we compare our results [Fig. 3(d)] with ePIE [Fig. 3(e)]. We can clearly find that our reconstructed phase completely eliminates the amplitude interference while ePIE cannot, owing to the introduction of advanced mechanisms.

In summary, by reproducing the imaging system through neural network tools and incorporating advanced mechanisms, our method provides a new perspective for FP reconstruction and can achieve higher image quality against optical aberrations and artifacts.

4.3.

Results on Experimental Datasets

In this section, we implement our method over several experimental datasets captured under normal and severe conditions.

4.3.1.

Normal condition

We first implement our framework over two general datasets where the samples are well placed on the focal plane. Figure 4 shows the reconstructions of a tissue section stained by immunohistochemistry methodology and a blood smear. In Fig. 4(a), we show the three recovered images of the tissue section illuminated by red, green, and blue monochromatic LEDs. By combining the three images together, the recovered color amplitude is shown in Fig. 4(b1) with its phase shown in Fig. 4(b2). The corresponding results from ePIE are shown in Fig. 4(c). The tissue section dataset is carefully captured so that the aberration could be ignored during the reconstruction. As such, we can find that both INNM and ePIE can obtain images with high quality.

Fig. 4

Reconstruction of two datasets in a low-aberration condition. (a) Recovered amplitudes at 632, 532, and 470 nm wavelengths from INNM. (b), (c) The combined color intensity and phase images of a tissue section stained by immunohistochemistry methodology from INNM and ePIE. (d)–(f) Recovered results of blood cells.

In addition, we test the two methods in another blood smear dataset shown in Figs. 4(d)–4(f) and draw the same conclusion. The recovered sample images from ePIE and INNM share highly similarly detailed information such as cellular structures and phase distribution, and we can clearly see the performance improvement from these reconstructions compared to raw images shown in Fig. 1(b).

4.3.2.

Severe condition

Although the image reconstruction capabilities of INNM and ePIE are similar in generic datasets, the introduced mechanism introduced by INNM greatly enhances the potential to alleviate the optical aberration. We deliberately crop a tile close to the edge of the entire FOV, in which the system aberration is so sophisticated that seriously affects the qualities of captured intensity images. To make our results more credible, we additionally test INNM under different network settings, adjust the number of Zernike modes $L$ to 50, and make an extra comparison with Jiang’s method²⁸ and AS.⁴²

The reconstructed amplitude images with some magnified areas are shown in Fig. 5 (the ground truth is obtained from the aberration-free dataset). First, compared with the ground truth shown in Fig. 5(a), we can easily find that ePIE cannot recover the amplitude with distinguishable cell structures [Fig. 5(e)], which also happens in the reconstruction of INNM without Zernike function [Fig. 5(d)]. Then when focusing on the center of the secondary magnification area, we can find that reconstruction obtained by the INNM without TV is rough in detail due to the noise [Fig. 5(c)], and the information of small cellular structure is totally lost. In contrast, the complete INNM can distinguish this small cell characteristic from surrounding tissue structures, and the overall image is high quality with a smooth background [Fig. 5(b)].

Fig. 5

Recovered amplitude images of a tissue slide in condition of severe aberration: (a) high-resolution ground truth; (b)–(d) recovered amplitudes from INNM under different conditions; (e) recovered amplitude from ePIE; and (f), (g) aberrations restored from INNM with and without TV.

In Fig. 6, we further compare the phase images recovered from INNM, ePIE, Jiang’s method, and AS, in which the first two methods take optical aberration into account and the latter two do not (all the methods start with the zero-aberration hypothesis). It is clear that reconstructions from Jiang’s method and AS are highly blurred and cannot provide instructive information due to the influence of aberrations. For more recognizable details shown in the secondary magnification area, we can find that INNM achieves higher contrast than ePIE, compared with GT shown in Fig. 6(a). Also the cell structure of ePIE is quite blur compared with INNM.

Fig. 6

Recovered phase images in condition of severe aberration: (a) ground truth; (b)–(e) recovered phases from INNM, ePIE, Jiang’s method, and AS;⁴² and (f), (g) aberrations restored from INNM without Zernike mode and ePIE.

Reflected in the recovered optical aberration, the results of ePIE and INNM without the Zernike mode [Figs. 6(g) and 6(f)] are quite different from those of INNM with and without TV [Figs. 5(f) and 5(g)], resulting in poor performance on recovered images shown in Fig. 5. This phenomenon further proves the superiority of INNM in compensating aberration.

In conclusion, INNM can reconstruct sample images with high quality in both simulated and experimental datasets. Especially, when validated in severe aberration conditions, INNM performs higher robustness against sophisticated optical aberrations and achieves better performance than other methods. This improved performance relies on the augmenting modules like Zernike aberration recovery on the one hand, and the AD of optimization tools in neural networks on the other hand. The AD approach allows solving FP problems without finding an analytical derivation of the update function since the derivation could be challenging to obtain. In addition, AD is directly benefited from the progress made in the machine-learning community in terms of hardware, software tools, and algorithms. We can expect INNM to be further improved thanks to the fast-paced progress in AD.

5.

Conclusion

In this work, we reported a reconstruction method based on the neural network model to solve FP problems and achieved artifacts-free performance. By reproducing the imaging process into the neural network and modeling the sample and aberration as learnable weights of multiplication layers, INNM can obtain an aberration-free complex sample. Advanced tools are further introduced to ensure good performance. The AU can help INNM optimize the sample and aberration in an appropriate alternative way, the introduced Zernike mode can estimate the sophisticated optical aberration with high accuracy and the TV terms is useful to reduce artifacts by encouraging spatial smoothness. We tested our method over both the simulated and experimental datasets, and the results demonstrated that INNM could reconstruct images with smooth backgrounds and detailed information. We believe our recovered aberration can be used as a good estimation toward system optical transmission and we hope our method can provide a neural-network perspective to solve the iterative-based coherent or incoherent imaging problem. Moreover, we can expect such techniques can be further improved thanks to the fast-paced progress in deep-learning toolboxes like TensorFlow or powerful tools like novel regularizations and optimizers.

There are many works worth trying in the future. For example, we can replace some layers of INNM with advanced architectures so that the model can learn some advanced features in advance. In addition, although we can foresee the effect of different coefficient combinations in TV terms, manual adjustment is still required for redundant operation in practice. As such, how to optimize the image evaluation system, with which the network model can automatically adjust the performance, is also a feasible direction of research.