Advanced Biological Imaging for Intracellular Micromanipulation: Methods and Applications

Abstract

Intracellular micromanipulation assisted by robotic systems has valuable applications in biomedical research, such as genetic diagnosis and genome-editing tasks. However, current studies suffer from a low success rate and a large operation damage because of insufficient information on the operation information of targeted specimens. The complexity of the intracellular environment causes difficulties in visualizing manipulation tools and specimens. This review summarizes and analyzes the current development of advanced biological imaging sampling and computational processing methods in intracellular micromanipulation applications. It also discusses the related limitations and future extension, providing an important reference about this field.

1. Introduction

Intracellular organelles and molecules governing the liveness [1,2], dynamics [3,4], organizations [5,6], and functionalization [7,8] at cellular, multicellular, and tissue levels have attracted considerable interests from the field of biomedical research. Intracellular manipulation, measurement, and other surgical operations were conducted to unveil underlying mysteries inside cells [9]. For example, intracellular physical and chemical properties, such as Young’s modulus [10], viscosity [11], pressure [12], pH [13], and temperature [14], were measured to study biological processes and biomedical reactions in living cells. Some tracking macromolecules were delivered inside mammalian cells for the exploration of intracellular mechanisms, such as the entry of cytoplasmic molecules into primary cilia [15] and the release of spliced mRNAs from nuclear speckle domains [16]. Furthermore, genome-editing tasks have attracted research interest, particularly those that involve the transfer of DNA [17,18] and RNA molecules [19,20] to different cell types for cell transfection.

Although many intracellular manipulation techniques have been proposed, traditional clinical operations still depend on well-trained operators to fulfill various tasks [21]. In practice, operators laterally align manipulation tools with targeted specimens, manually adjust the relative height positions between manipulation tools and targeted specimens, and then initiate a given operation. Fatigue- and experience-induced failures always occur and result in low operation success rates and large damage, thus degrading the performance of intracellular manipulation [22,23]. Generally, intracellular objects are fragile [24,25] and usually have irregular shapes [26,27], distribution [28,29], and small sizes [30,31]. For example, mitochondria in mammalian animals are usually less than 2 μm and randomly distribute among the cytoplasm [32]. An inappropriate operation position not only leads to operation failure but also largely damages the targeted cells and even kills them. Therefore, the acquired position should be precisely manipulated through computational imaging during intracellular operations.

The complexity of intracellular environments and culture media inevitably causes difficulties in localizing operation positions [33,34,35]. The identification of targeted biological specimens is easily disturbed by unavoidable surroundings, such as counterpart cells and auxiliary manipulation tools, in an imaging field. Importantly, the morphological characteristics of most organelles are similar to those of their intracellular neighbors; as such, they become undistinguishable under optical observation [36]. Optical observation should be carried out several times to ensure that the imaging plane contains the target organelles because of the random distribution [37]. Meanwhile, most organelles have an irregular shape, so their appropriate operation position should be selected through comprehensive considerations [38]. Moreover, imaging blurs and noises during an optical observation deteriorate the imaging performance and make the organelles boundary unclear [39]. Apart from the automation demand of existing intracellular micromanipulation systems, two main challenges are encountered in current studies: how to obtain the sampled images of targeted intracellular specimens with high resolution and contrast and how to precisely extract operation information from the sampled images for a successful intracellular micromanipulation. The first challenge can be overcome with emerging advanced imaging technologies. With the help of some start-of-the-art imaging instruments and post-computation procedures, high-quality, real-time images during manipulation can be obtained. For the second challenge, many intelligent image computational processing methods have been proposed to restore the sampled images, analyze the targeted specimens, and provide precise visual feedback. Thus, the success rates of intracellular manipulation tasks can be improved, and operation damage can be reduced.

Reviews on biological imaging and computational processing methods in intracellular micromanipulation applications are rarely reported. In this review, advanced imaging technologies are described in terms of methods and applications in intracellular micromanipulation. First, sampling methods for deriving high-quality images are summarized in terms of imaging principles. Second, the intelligent computational image processing methods of the sampled images in different intracellular micromanipulation tasks are discussed and analyzed. Third, the limitations and drawbacks of each method are also explored. Lastly, conclusions and future outlook are presented. This systematic summary of this field can provide valuable references for academic and industrial fields.

2. Image Sampling Methods

Imaging sampling is essential for a successful intracellular micromanipulation through which the recognition, localization, tracking, reconstruction, motion, and orientation control of target specimens and manipulation tools heavily rely on the quality of sampled images. In other words, image sampling methods must be selected on the basis of several aspects, such as the size, morphology, distribution of specimens, imaging resolution, sampling speed and accessibility of the adopted instruments as well as their integration capability with other manipulation tools. For example, computer tomography (CT) and magnetic resonance imaging (MRI) are widely used in the visualization of bones, tissues, and organs, but they are unsuitable for imaging small-scaled specimens, such as cells and organelles [40,41].

This section mainly focuses on advanced biological imaging technologies at a cellular or subcellular level, such as commercially available state-of-the-art microscopes, including confocal fluorescence microscope (CFM), two-photon fluorescence microscope (TPFM), light sheet fluorescence microscope (LSFM), lab-built photoacoustic microscope (PAM), traditional wide-field fluorescence microscope (WFFM), advanced structured illumination microscope (SIM), and stimulated emission depletion microscope (STEDM) with a super-resolution imaging ability.

2.1. State-of-the-Art Microscopes

2.1.1. Confocal Fluorescence Microscope (CFM)

CFM has become a routine imaging tool in biological and biomedical sciences. The basic concept of confocal microscopes was first proposed in a previous study [42]. In brief, a pinhole aperture placed in front of a microscope photodetector can focus illuminated light on specimens, thereby rejecting most out-of-focus blurs during observation, as shown in Figure 1a. As commercially available instruments, CFM is widely adopted in imaging living cells, intracellular organelles, and components [43]. HeLa cells and porous CaCO₃ particles were recorded with the aid of a CFM, which revealed the number of internalized vaterite particles and their crystalline structures [44], as shown in Figure 1b. The actin cytoskeleton architecture of Leukemia cell lines, Jurkat, and K562 cells were fully observed with a CFM to probe the underlying F-actin remodeling mechanism [45]. With the ability to control the imaging depth of a field and reducing background blurs away from the focal plane, CFMs can be used to measure the 3D structure of specimens. Polystyrene beads were delivered into human hepatocellular carcinoma (HCC) cells and verified with a 3D measurement from CFM [46], as shown in Figure 1c. In Figure 1d, the CFM 3D reconstruction crosses the dashed red line after microinjection for 15 s [47]. The spatial resolution of CFMs is restricted above 200 nm because of the diffraction limit of optical microscopes [48]. Research efforts have been devoted to improving the CFM resolution beyond the diffraction limit. A pinhole displacement method has been proposed to increase the contrast up to 80% with a diffraction-limited resolution of 250 nm [49]. The light intensity profile of Bessel beams follows Bessel function of different orders. This intensity distribution gives it a tighter focal spot than that of classic Gaussian beams, and its use in CFM can lead to a resolution enhancement of up to 20%. A CFM using Bessel–Gauss beams improved lateral resolution to 219 nm [50]. Furthermore, a fluorescence emission difference method illuminating specimens with a solid beam and a donut beam has been established. The final measured image was the subtraction of the acquired solid image and the donut one, and its lateral resolution was 210 nm, which was much higher than that of a conventional CFM [51]. In a resolution-doubling method, a point detector was replaced with an imaging detector via a recording algorithm, and the resolution of CFM can be increased to 162 nm [52]. With these methods, minor intracellular components such as actin filaments, microtubes, and proteins can be optically discriminated in CFM observations. However, CFM has the disadvantages of slow scanning speed [53] and high-intensity light [54], which may lead to short fluorescence lifetime and phototoxicity in living cells [55]. Therefore, the imaging time should be strictly controlled during CFM operation.

2.1.2. Two-Photon Fluorescence Microscope (TPFM)

TPFM principle is based on two-photon absorption and excitation phenomena, that is, the combined energy of two quasi-simultaneously absorbed photons is sufficient to match the energy gap between the ground and excited states [56]. The typical structure of a TPFM is illustrated in Figure 1e. With the reduction of photo-bleaching effect, TPFM has become a powerful instrument for long-term measurement [57]. For example, TPFM can be used to monitor the movement of transferred T-cells in mice for 48 h [58], as shown in Figure 1f. TPFM is not susceptible with interferences to fluorescence concentration and light sources. TPFM can be used to accurately detect fluorescent probes to track dynamic changes of hydrogen peroxide (H₂O₂) and adenosine triphosphate (ATP) in neurons [59]. In addition, TPFM can perform tomography to measure specimens at different optical depths. A TPFM-based rapid bacteria localization platform was proposed. This technique produced z-stack images to distinguish the intracellular and extracellular bacteria [60]. However, TPFM is mainly limited by the low imaging speed (about 0.5 Hz), and it take tens of minutes to obtain a high-resolution image stack for the traditional TPEM [61]. Recently, an advanced TPFM with a high sampling speed of 2.7 kHz has been developed using a multilfocus compressive sensing method and a digital micromirror device [62]. Besides, the spatiotemporal resolution of TPFM is relatively low, which is theoretically 500 nm laterally and 1 µm axially; it also follows the diffraction limitation [63]. Other studies have proposed a fast high-resolution miniaturized two-photon microscope (FHIRM-TPM) with a lateral resolution of 640 nm and an axial resolution of 3.35 µm [64], and a photomultiplier TPFM with a lateral resolution of 550 nm [65]. The measured lateral resolutions of a TPFM with a double-wavelength meta-surface object lens at light wavelengths of 820 and 605 nm are 930 and 680 nm, respectively [66]. For TPFM, the visualization of biological objects smaller than hundreds of nanometers in size seems challenging. However, TPFM, which benefits from high-energy photons, can drive the electronic excitation of deep fluorescent molecules and is suitable for deep tissue imaging [67]. Specifically, TPFM has attracted extensive interest in studies of the structure and function of the mammalian brain in vivo [68]. Last but not least, research should note the high purchase costs of TPFM, which are normally more expensive than others [69].

2.1.3. Light Sheet Fluorescence Microscope (LSFM)

LSFM is another instrument with a long-term measurement capability. A light sheet generated from a thin laser sampling beam is focused only on a portion of specimens, and fluorescence is then collected perpendicularly from the illumination axis [70], as shown in Figure 1g. The highly efficient excitation and collection scheme ensure that the phototoxic and photobleaching effects of LSFM are minimized compared with those of CFM and TPFM [71]. Specifically, exposure light energy was found to be 5600 and 10⁶ times less than that of CFM and TPFM, respectively [72]. Therefore, LSFM is suitable for such applications requiring long-term measurement and careful exposure. Morphology changes of Drosophila embryos at different biological stages, such as germ band retraction, segment formation, and dorsal closure, were imaged with a LSFM [73]. Cardiomyocytes were imaged based on a customized LSFM for up to 1.5 days to investigate the holistic cell behaviors sculpting the four-chambered mammalian heart [74]. It should be noted that sampled images of LSFM are invulnerable to out-of-focus blurs, and they can exhibit more detailed features than normal wide-field microscopes [75]. Figure 1h shows the mitochondria visualization of HeLa cells labeled with the mitochondrial import receptor subunit TOM-20 by using a LSFM [76]. The image demonstrates a better contrast for LSFM than for wide-field illumination. The lateral resolution of LSFM is regulated primarily by the numerical aperture of the detection objective, while the axial resolution is additionally determined by the thickness of the light sheet [77]. Specifically, LSFM has a comparable lateral resolution as CFM. A single-objective LSFM with a resolution about 300 nm was proposed, which was based on the concept that combining objective with a coverslip-walled immersion chamber. Parallel recording of 16 cell lines and their cellular responses to drug treatment was conducted with this instrument [78]. Utilizing a LSFM with a scheme of two orthogonal detection and illumination objectives, the neural wiring during Caenorhabditis elegans brain development was imaged with a lateral resolution of 330 nm [79]. However, the axial resolution of LSFM is not as high as CFM, which is typically not better than 1 µm [72,80,81,82]. It is worthwhile to note that the complicated sample preparation is a problem for LSFM, because its orthogonal configuration hinders standard Petri dishes and multi-well plates for sample preservation [83]. In such a narrow space, the integration of LSFM with other manipulation tools is a huge challenge. Therefore, its applicability in intracellular micromanipulation is greatly limited.

2.2. Photoacoustic Microscope (PAM)

Photoacoustic imaging is inspired by the photoacoustic effect converting optical energy to measurable pressure waves [84]. This effect is initiated when specimens are irradiated with a modulated laser source, resulting in thermoelastic expansion and photoacoustic wave formation corresponding to the geometric characters of specimens [85]. Figure 2a shows the two main configurations of a photoacoustic microscope (PAM) in terms of imaging modes. One is the optical resolution (OR) mode in which the focused laser light is illuminated on specimens. The optical region of an OR-PAM is restricted, but its lateral resolution is improved. The other one is the acoustic resolution (AR) mode that adopts uniform illumination. The unfocused light enables deeper penetration but yields lower resolution [86].

Photoacoustic imaging technology is suitable for deep depth imaging, and because acoustic waves can transmit across soft tissues and skins, high-resolution imaging can be achieved at depths far exceeding the optical diffusion limit. In fact, photoacoustic imaging technology has been widely adopted for navigating microrobots in vivo. The migration of capsule micromotors was controlled to move to the targeted region in the intestines under the guidance of photoacoustic imaging [87], as shown in Figure 2b. PAM also has the advantages of label-free imaging and out-of-focus blur suppression, and has become an ideal technology for non-toxic biological imaging. Many subcellular practices have been imaged with enhanced contrast. This instrument can achieve a high lateral resolution of 250 nm. Figure 2c illustrates the sampled images of lipids, proteins, and nucleic acids in fibroblast cells by using an ultraviolet-localized mid-infrared PAM [88]. However, it lacks an optical sectioning capability. Theoretically, the axial resolution of PAM is controlled with the bandwidth of a photoacoustic signal and the central frequency of an ultrasonic transducer [89]. Studies have been performed to improve the axial resolution of PAM without losing its lateral accuracy. For instance, a Grueneisen relaxation photoacoustic microscope (GR-PAM) delivering two identical laser pulses has been proposed [90], where the axial resolution can be improved to 2.3 µm, and the lateral resolution was 410 nm. Figure 2d illustrates the optically sampled images of red blood cells at three different depths and their 3D reconstruction model. Another study has developed an isometric PAM based on a transparent micro-ring resonator ultrasonic detector, whose axial resolution was 2.12 µm [91]. In addition, an axial resolution improvement method has been established on the basis of configuration modification [92]. Figure 2e shows the dual-view PAM and sampled images of a mouse brain slice. Two split lasers are focused onto the specimens, and photoacoustic waves are collected with the ultrasonic transducer. This design can improve the axial resolution to 1.8 µm. Generally, due to its relative low axial resolution, PAM is more frequently to measure specimen profiles in 2D sampled planes instead of 3D space. Besides, PAM has several other limitations that hinder its applications in biological imaging. First, the sampling speed of PAM is low. There is a trade-off between resolution and sampling speed, that is, it takes several minutes to measure an area of 1 m × 1 m² with an acceptable optical resolution using using OR-PAM [93]. Second, PAM requires complex light configurations, and there are almost no commercial tools available, which limits the accessibility of researchers who do not have expertise in this field. Finally, the potential damage from deposited laser energy is another issue that should be avoided in high-resolution imaging [94,95].

2.3. Wide-Field Fluorescence Microscope (WFFM)

With low purchase expenses and high accessibility [96], WFFM is popular in biological imaging. Figure 3a shows the WFFM observations of a burr-like porous spherical microrobot loaded with MC3T3-E1 fibroblasts, and the microrobot movement inside the yolk of a zebrafish embryo was measured for automation control [97]. Similar to CFM, WFFM suffer from a diffraction barrier. The resolution of WFFM can be expressed as $\frac{\mathsf{\lambda }}{2NA}$ (lateral) and $\frac{2\mathsf{\lambda }}{N{A}^2}$ (axial) [98], where λ is the wavelength of the emission light, and NA is the numerical aperture of the objective lens. Even for a WFFM with a high-NA objective lens, its resolution hardly reaches 230 nm laterally and 800 nm axially because the visible light wavelength varies from 390 to 780 nm [99]. During WFFM operation, the whole specimens are illuminated with emission light, and sampled images contain in-plane features, out-of-focus blurs projected from adjoined depths, and noises, which undoubtedly deteriorate the imaging quality, especially for depths that are far from the focused plane [100]. Such optical behavior can be modeled as a convolution process between point spread functions (PSFs) and the measured specimen [101]. To restore the sampled images of WFFM, many methods have been proposed on the basis of deconvolution algorithms that numerically invert blur processes, such as the Landweber method [102], Tikhonov–Miller gradient descent method [103], and Richardson–Lucy maximum likelihood method [104,105]. Figure 3b shows an unfocused sampled image of Caenorhabditis elegans and deconvoluted images with different deconvolution methods [106]. The processed images exhibit sharper features with more details than the original one, providing more accurate information for intracellular micromanipulation tasks. Through focal plane movement, the whole specimen can be scanned, and the sampled images via a WFFM are optically sectioned and stored in a computer. The sampled images can also be reconstructed into a 3D model after the restoration processes of deconvolution, segmentation, and reconstruction [107]. Figure 3c presents the sampled image series of a mitochondrion in a THP-1 cell and its 3D reconstruction model.

Due to advantages of great compatibility, controllability, and programmability, WFFM is widely used to visualize cells, intracellular organelles, microrobots, and other micro-objects. It should be noted that the size of measured specimens is usually larger than 1 µm, because WFFM resolution is limited even if the recovery algorithm is implemented. On the other hand, due to its simple configuration and large operation space, robot-aided manipulation tools are easy to integrate with WFFM. Therefore, WFFM is frequently applied in intracellular micromanipulation.

2.4. Super-Resolution Microscopy

Several super-resolution microscopy technologies are developed to break the diffraction limit of optical microscopes during recent years. Figure 4a illustrates the configuration of a structured illumination microscope (SIM), whose spatially patterned excitation light can be generated by multiple laser beams; as such, SIM can be used for super-resolution observation [108]. Specifically, the resolution of SIM can reach approximately less than 100 nm laterally and 300 nm axially [109]. The optical section can be sampled by generating either a grating at the conjugate plane of specimens or a fringe pattern onto them [110]. Figure 4b shows the sampled images of cytoskeletal and nuclear components in a uterine endometrial epithelial cell [109]. The images obtained with a SIM have better contract and resolution than the images sampled via a WFFM. However, the high-resolution sampling of SIM is prone to small position movement and intensity changes, Therefore, in case of a poor signal-to-noise ratio, the specimen should be fixed during SIM measurement [111], which will reduce the application range of SIM in intracellular microscopy.

Another exciting phenomenon for super-resolution microscopy is stimulated emission through which PSFs are scaled down to a subdiffraction level [112]. Figure 4c shows a typical configuration of a stimulated emission depletion microscope (STEDM); here, excited-state electrons can return to their ground state by interacting with a depletion laser beam of a certain frequency [113]. The lateral resolution of STEDM can reach 30–80 nm for most commercial systems, and the axial resolution of less than 100 nm of STEDM can be achieved with specialized depletion schemes [114]. Figure 4d illustrates the STEDM-sampled images of living hippocampal primary neurons; the enlarged view proves its ability to provide a nanoscale resolution that is superior to that of CFM [115]. However, stimulated emission is obtained by using a high-powered depletion laser. The higher the laser density, the higher the sampling resolution, but the photobleaching due to high depletion laser intensities is a problem that cannot be ignored [116].

Super-resolution microscope technology is advanced in observing small-scale intracellular components (such as protein [117,118], lysosomal [119,120], bacteria [121,122]). However, there are few attempts to use SIM and STEDM for micromanipulation because the current robot-aided manipulation tools can only handle objects at as low as sub-micrometers. With the rapid development of microfabrication and information technology, super-resolution microscope will play a more important role in the future.

2.5. Insights

Optical sampling methods should be selected in terms of micromanipulation tasks.

Table 1. Comparison of image sampling methods based on different principles.

Methods	Lateral Resolution Level	Axial Resolution Level	Advantages	Limitations	References
CFM	~200 nm	~500 nm	out-of-focus blurs rejection; optical section	high light intensity; slow sampling speed	[42,43,44,45,46,47,48,49,50,51,52,53,54,55]
TPFM	~500 nm	~1 µm	out-of-focus blurs rejection; high penetration depth; long-term measurement	low sampling rate; relatively low axial resolution; high purchase expenses	[56,57,58,59,60,61,62,63,64,65,66,67,68,69]
LSFM	~300 nm	>1 µm	out-of-focus blurs rejection; long-term measurement	low axial resolution; confined space	[70,71,72,73,74,75,76,77,78,79,80,81,82,83]
PAM	~300 nm	~2 µm	out-of-focus blurs rejection; high penetration depth; label-free measurement	slow sampling speed; complicated configuration; unavailable commercially	[84,85,86,87,88,89,90,91,92,93,94,95]
WFFM	~400 nm	>800 nm	low purchase cost; great compatibility;	out-of-focus blurs; low resolution	[96,97,98,99,100,101,102,103,104,105,106,107]
SIM	<100 nm	<300 nm	high resolution; out-of-focus blurs rejection	specimen fixation; high purchase costs	[108,109,110,111, 118,119,121]
STEDM	<100 nm	<100 nm	high resolution; out-of-focus blurs rejection	high light intensity; high purchase costs	[107,112,113,114,115,116, 120,122]

summarizes the characters of each method. Generally, resolution is key to obtaining high-quality information. For intracellular organelle micromanipulation, imaging resolution should reach at least the micrometer level, which is achievable with most methods. Micromanipulation is a multistep task through which sampling is a single operation procedure, so the adopted methods should also be evaluated in terms of integration with robot-aided manipulation tools and automation control and programming. Furthermore, the possible photobleaching and phototoxic effects should be considered, and correct instruments with appropriate measurement parameters should be chosen. Accessibility and purchase costs should also be taken into consideration.

3. Intracellular Micromanipulation with Computational Image Information

Intracellular micromanipulation is difficult to perform and has a low operation success rate because of the small sizes, irregular shapes, and irregular distribution of targeted specimens and the complexity of intracellular environments. However, specimen morphology, position, orientation, manipulation speed, force, and other operation details are hidden in sampled images. Advanced computational image processing methods can extract such details from high-quality sampled images and facilitate complicated intracellular manipulation tasks.

The following section summarizes different types of computational processing methods, such as planar localization, depth acquisition, and 3D analysis. It also discusses their suitability and limitations.

3.1. Planar Localization

3.1.1. Segmentation

Segmentation is a preliminary process to obtain clear planar contours of targets or detect small particles inside cells. Normally, cells or other biological targets are covered by other devices and similar objects. The localization of such targets requires specific segmentation algorithms. Thresholding is a simple approach for eliminating unnecessary information. Sampled images can be binarized into two categories by setting a constant or adaptive threshold value: a bright target and an unnecessary background. A simple process using the Otsu adaptive thresholding was proposed for detecting the position of an oocyte. The area and roundness of the segmented object were calculated as an identification factor to remove unnecessary objects [123]. In many intracellular micromanipulation tasks, microfluidic chips are utilized to immobilize floating cells. Trapped cell contours can be extracted with a Canny edge detector [124] and a Sobel edge detector [125] and then fitted with a Hough circle transform, as shown in Figure 5a. With the use of fluorescent dye, stained organelles exhibit a higher pixel density than that of the others. Figure 5b illustrates the intracellular localization of JC-1 stained mitochondria. The mitochondrion contour was extracted by setting a threshold defined as Hue, Saturation and Value (HSV) range value [126]. Although Gauss [124], and Median filters [126] or morphological operations [123] was adopted to smoothen the sampled images in the aforementioned methods, most out-of-focus blurs and imaging noises were difficult to eliminate. Direct segmentation with a threshold value or edge detectors on original sampled images is only suitable for the manipulations of large-sized objects, because it may erase useful information near real object boundaries and produce inaccurate contours and localization of targeted specimens. WFFM-sampled images should be subjected to a deconvolution process to de-blur and de-noise, the deconvoluted images can then be processed through segmentation. Other sampling methods with out-of-focus blurs rejection capability should be considered. A nucleus laser ablation system was proposed based on LSFM. The region of interaction between laser and the targeted nucleus was segmented in different shapes on the acquired high-contrast images for subsequent ablation [127], as shown in Figure 5c. Compared with WFFM and CFM which can observe the fluorescence structure after laser manipulation, LSFM can record the target quickly and persistently before and after operations.

3.1.2. Feature Reference Points

Feature points are essential for many intracellular micromanipulation tasks as a position reference. For example, the nucleus of adherent cells is visibly ellipsoidal, and the nucleolus has explicit light intensity differences with the surroundings, so they can be localized as operation references for the manipulation of attached cells. A localization method based on K-means cluster and growing circles has been proposed [128]. In this method, circles around the nuclear region were interactively grown on the basis of the low-level information from a Canny operator. The nucleus and cytoplasm positions were identified and used as a reference for precise single cell electroporation [129], as shown in Figure 6a. In intracytoplasmic sperm injection, oocytes must be oriented properly into a desired plane to preserve the tested cells. Polar body is usually selected as the reference point to identify oocyte position. A micropipette orientation approach based on the WFFM-sampled images of a polar body has also been developed [130]. The sampled image was processed with morphological close and open operations, and a potential poly-body edge was obtained. As polar body usually looks like a bump on the cytoplasm, so the position of polar body can be recognized by calculating the average distance of potential points to the fitted center of the tested oocyte. A sheep oocyte oritentation was proposed based on a holding pipette and a manipulation pipette. The polar body was out of the plane initially. The oocyte was rotated by slightly rubbing the micropipette until the polar body reached the desired position. For the period that the polar body was invisible in the current focal plane during the out-of-plane rotation, recognizing the real-time orientation from such reference points becomes relatively difficult [131]. Besides, polar body is a special intracellular organelle of oocytes, and this method is not universal for all cell types. Therefore, polybeads were attached to the cells without polar bodies for orientation reference [132], and the three-dimensional rotation of such cells was achieved with optical tweezers. The localization of the polybead was acquired by a specific threshold pixel value, as shown in Figure 6b. However, inevitable noises and similar small particles exist around the reference polybead, so the identification of the desired point may be erroneous, and its long-term accuracy is questionable. Besides, the aforementioned visualization problem remains. Some of the inherent feature positions of cells can also be selected as reference points for orientation control [133]. Given the uniqueness of each tested cell, these feature points have to be selected individually. Super-resolution microscopy can detect small biological features with high quality. Recently, a new approach combining a STEDM with an atomic force microscope (AFM) was proposed to study the density and function of the actin cluster in mutated cells [134]. Polymerizing actin can be imaged at super-resolution by STEDM and then stimulated by an AFM tip. However, synchronization and correlation between the two systems are challenges that cannot be ignored. An automatic AFM tip detection method was proposed to record the relative position of AFM and STEDM images [135]. The area of interest was selected and then filtered to remove unnecessary details. The tip reflection was assumed to be the brightest point and identified after the iterative filter process, as shown in Figure 6c.

3.1.3. Tracking

In intracellular micromanipulation, manipulation tools and targets are normally small and fragile, so their localization requires skilled operations and tedious time. The automated tracking of manipulation tools based on image information can update operation position, adjust the moving path simultaneously, and increase micromanipulation success rate. A tracking algorithm based on motion history image was proposed [136], as shown in Figure 7a. The subtraction of two neighboring frames records the motion history of tool tips. Once the pixel value was larger than a threshold, the algorithm initiated an active contour model and detected boundaries of the manipulation tools. However, this tracking method may probably fail when the targeted specimens suddenly appear in the region of interest (ROI), or manipulation tools come in contact with them. Thus, a reinitialization and recovery mechanism named as Delock and ReloCk has been proposed [137]. When a targeted cell occluded the ROI, the reinitialization process was triggered, and an instantaneous template was recorded for accurate tracking. When the micropipette touched and deformed the cell, the pipette tip was localized by a template updated from the last two frames, as shown in Figure 7b. Besides, a confidence-based hybrid method has been developed to enhance tracking robustness; in this method, a confidence vector was weighted on estimations from motion-cue feature detection and similarity score-based matching methods [138]. Another tracking demand in micromanipulations lies in investigations on targeted specimens. A modified joint probabilistic data association filter method has been applied to track sperms. Sperm shape information was derived from differential interference contrast images obtained with a WFFM [139]. Figure 7c presents a robust multiple sperm tracking and sperm immobilization for intracytoplasmic sperm injection [140]. In cell aspiration measurements, a cell is deformed under negative pressure, and the aspirated length should be tracked to characterize cellular mechanics. Gradient subtraction method [141], Sobel filtration method [142], or Shi–Tomasi corner detection method [143] was used to enhance cell edges and eliminate adjacent interferences. The light intensity of cell edges is higher than the surrounding area. Therefore, the edge profile should remain bright, and the adjacent interferences will disappear after the filtering operation. The contact position between the cell and the aspiration micropipette can be initially identified. Then, the sampled images were separated into two ROIs containing the aspirated part and the remaining part outside the micropipette. The aspiration length can be tracked in the ROI of the aspirated part, as depicted in Figure 7d. In comparison with the sampled images from WFFMs, fluorescence sampling from CFMs provides clearer localization, which does not require complicated image processing procedures. The aspirated length was easily tracked with CFM-sampled images [144], as displayed in Figure 7e. Therapies based on microrobots has shown great potential for disease treatment, in which microrobots loaded with therapeutic cells can navigate to desired locations in the body. PAM provides image guidance for transportation of microrobots in deep tissues and organs. Figure 7f illustrates the trajectory of the burr-like porous microrobot in the mouse inferior vena cava. The microrobot showed a clear light difference with the mouse vena cava at the imaging depth of 6 mm, and can be tracked with minor position errors [145]. Recently, some photon-activated molecules have been manipulated and tracked to study intracellular activity. Compared with the WFFM illumination that activates many cells at the same time, with the help of one photon excitation of TPFM, the manipulation of a single cell can be realized. Photosensitive optogenetic actuators were activated and tracked with TPFM to study the functional reactions of immune cells involved in calcium interaction [146]. Figure 7g shows the long-term tracking of calcium-mediated actuators inside mouse T-cells, where cell arrest was measured during the induction of calcium influx.

3.2. Depth Acquisition

Intracellular manipulation tasks are tedious and time consuming, so they are difficult to be accomplished with a high successful operation rate and low operation damage even when they are performed by skillful operators. In many micromanipulations, manipulation tools should be aligned with the height plane of the targeted specimens at the beginning of experiments. However, manual manipulation becomes unreliable, especially small cells or intracellular organelles with irregular forms. Some advanced image processing strategies that can acquire depth information and automatically adjust the height of manipulators or specimens have been proposed in the literature. Template matching and autofocusing are the two main methods, which will be discussed in the following sections.

3.2.1. Template Matching

Template matching is based on the sampled image in the focal plane, where the contour of adopted manipulation tools or the targeted biological specimen is set as a template at the beginning of manipulation. By comparing the template with the features in the sampled image during the operation task, the current imaging depth can be estimated. A template matching method has been proposed to adjust the height of a micropipette tip in microinjections [147]. In this method, a micropipette was moved along the normal direction of a focal plane, and real-time images were recorded during the micropipette movement. The micropipette was assumed to be positioned in the focal plane when the pixel error between a template and recorded images was minimized. However, the micropipette may go to a wrong focal plane when it is close to the focal plane, as shown in plane A in Figure 8a. An optimized template matching method has also been developed [148]. In this method, a template image was binarized to remove useless information. The gradient of differences between the template and the recorded images was compared to determine if the micropipette was above the focal plane. A template similarity score was established to calculate template differences [149,150]. When the score is maximized, manipulation tools was aligned to the height of the focal plane. In attached cell microinjections, a micropipette is lowered down to penetrate the cell without sliding on the dish substrate for the sake of breakage. In a three-frame template matching method [151], a current source image was compared with Frames 1 and 3 to check if the micropipette touched and scratched the substrate, as illustrated in Figure 8b. In addition, the centroid coordinates of the cells in the central template are also compared with the surrounding frame to identify the paired target [152]. However, template matching has some drawbacks. First of all, the template image should be manually selected before starting the operation, which requires a lot of labor and operation time. Second, when changing the target, the selected template is not universal, and a new template should be selected again. Third, the method is susceptible to external disturbances, noises, and imaging errors. When the imaging field moves during micromanipulation, the current focal plane probably differs largely from the original one, and the matched depth is far from the targeted plane. Therefore, template matching is suitable for processing biological objects whose size is much larger than the size of manipulation tool, because the estimation errors will not cause operation to failures.

3.2.2. Autofocusing

In comparison with the template matching method that involves the use of a predefined image to estimate the target height, the autofocusing method is utilized to analyze depth information based on imaging characters from sampled images. In imaging practices, the light intensity of a target edge experiences an abrupt change during the focal plane movement. Therefore, imaging depth can be obtained by searching sampled images from the focused plane. Based on this idea, many attempts called depth from focus (DFF) methods have been applied. An entropy computation approach was designed to focalize micropipette tips [153]. In this method, the rectangular region of the micropipette tip was first selected. When the tip was focused, the entropy value in this region reached its maximum clarity. A Prewitt operator was proposed to highlight cell edges of sampled images [154]. The processed pixel points exhibit higher intensity for sharper edges. In this way, the pixel intensity of the processed images was counted, and the focal plane was located in the plane with the largest amount, as shown in Figure 9a. However, out-of-focus blurs in sampled images were hardly removed with the Prewitt operator and even produced artifacts in highlighted edges. Currently, another micropipette autofocusing method called depth from border intensity variation has been created [155,156]. In this method, a stripe near the target object boundary was defined to compensate for the inaccurate boundaries and imaging noises. A focal plane can be selected by using a proposed defocus function from a Fourier spectrum, as presented in Figure 9b. DFF methods are time consuming because a large number of sampled images are required. For depth from defocus (DFD) methods, which only examine a limited number of images, the detection efficiency is greatly improved. In the proposed multiple DFD method [157,158], microscope imaging behavior was expressed as a simplified fixed Gaussian model. The depth of the current imaging plane can be calculated from the cell diameter and image depth in two other sampled images, as illustrated in Figure 9c. The authentic light spread function is space-variant because of the spherical aberrations caused by the mismatch of refractive indices between a lens immersion medium and a specimen and the discrepancy between the ideal and real parameters of the microscope system. Therefore, a fixed model is inaccurate for most specimens. Meanwhile, the cell diameter in this approach was derived from image segmentation by maximizing the posteriori estimation. The actual cell boundary was hardly extracted under the interferences of out-of-focus blurs and noises.

3.3. 3D Analysis

The aforementioned approaches either adjust manipulation tools or rotate targeted objects into a predefined height, so they seem applicable to large specimens, such as embryos and oocytes. For small organelles with irregular shapes and random spatial distribution, an operation plane should be selected through a deliberate 3D analysis of sampled images. In an active snake node method proposed to generate closed biomembrane contours, a 3D model of an egg cell was reconstructed using third-degree nonuniform rational B-splines to inspect the micropipette insertion [159] and measure the force [160], as shown in Figure 10a. An ellipsoid estimation method was applied to reconstruct attached bladder cancer cells [161]. Four intersection points were detected on the upper cell surface with motion history images. However, these fitted models were derived from the assumption that targeted cells have a regular shape. Specifically[159], it was assumed that the sampled image was assumed to lies in the symmetric central plane, leading to undoubted discrepancies with reality. Another 3D morphology reconstruction was achieved on the basis of a standard alpha-shape algorithm for cell rotation measurement [162]. The cell contours in each sampled image were extracted with a threshold value. Even though contour extractions were repeated for several rounds, they may contain background noises and out-of-focus blurs. Besides, this method is not feasible for organelles because they are undistinguished from other intracellular components in WFFM-sampled images. Fluorescence labeling can filter surrounding objects, but it cannot reject imaging blurs and noises. However, optical fluorescent sampling with a CFM can be applied to reject out-of-focus blurs and reflect object morphology. Figure 10b illustrates the CFM reconstruction of endothelial cells [163] and the nucleus of fibrosarcoma cells [164]. Manipulation tools lowered down and indented the cells with a visualized distance under 3D observation. Besides, a 3D reconstruction method integrated with a noise-regulated maximum likelihood estimation (NRMLE) deconvolution and filtered segmentation algorithm has been developed for WFFM [165]. In this method, the sampled fluorescent images were deconvoluted with NRMLE for deblurring and denoising. A filtered process was initiated before segmentation to compensate for the non-uniform concentration and distribution of fluorescence molecules. The reconstructed nucleus has high accuracy and a similar structure to that observed via CFM. An appropriate operation position can be obtained through the geometric analysis of the generated model, thereby improving the accuracy and efficiency of nuclear delivery, as shown in Figure 10c. Recently, a SIM-based accurate 3D spine neurons measurement and analysis method was proposed [166]. Figure 10d shows the 3D computational reconstruction process. First, the voxel data of SIM sampled images was converted into surface mesh data, then 3D features were extracted from the spine morphology dataset, and the 3D model was finally generated from dimension transformation and shape classification. With the super-resolution ability of SIM, this method can detect head curvature changes in small-sized spines (<0.18 µm³).

4. Conclusions

Intracellular micromanipulation has been widely adopted in genetic diagnosis, genome-editing, and other biomedical tasks. Currently, most clinical manipulations rely on the experience and proficiency of well-trained operators. Automating the process of micromanipulation will potentially contribute economical benefits and social significance to human health engineering. High-resolution imaging is highly demanded for visual guidance in successful operations. However, the precise and fast extraction of visual information from sampled images is still an obstacle in intracellular micromanipulation. Although many research efforts have been devoted to collecting targeted specimen features in sampled images, a tradeoff exists between computational accuracy and speed. In terms of different micromanipulation tasks, appropriate operation information can be quickly derived through model simplification, shape estimation, sampled image reduction, manual selection, and other methods, but their accuracy is insufficient. Specifically, these speed-emphasized methods are feasible for manipulating large specimens because few pixel-related mistakes in specimen boundaries slightly affect operational success. For small specimens whose total dimension is comparable with the imaging pixel size, accuracy should be highly prioritized. In some practices, imaging data are processed to generate 3D reconstruction models and conduct geometric analysis for precise organelle micromanipulation. Real-time reconstruction is relatively difficult to be realized and applied in high-throughout operations because of the computation complexity of 3D reconstruction. Nowadays, machine learning-based methods have emerged to facilitate biological imaging [167]. Detection robustness [168] and imaging resolution [169] can be largely improved with less computation complexity. As shown in

Table 2. Comparison of various image computational processing methods for intracellular micromanipulation.

Methods		Sampling Method		Advantages	Limitations
	Segmentation	Otsu thresholding [123]	WFFM	morphological smoothing	only for cell center localization; interferences of noises and blurs
		Canny edge detector [124]	WFFM	gauss noise filter; circle transformation	only for circular specimens; interferences of noises and blurs
		Sobel edge detector [125]	WFFM	morphological smoothing	only for cell existence determination; interferences of noises and blurs
		HSV value thresholding [126]	WFFM	more accurate thresholding value; median noise filter	interferences of noises and blurs
		Interaction segmentation [127]	LSFM	out-of-focus blurs rejection; full-time measurement	Only for indirect manipulation
Planar localization	Feature points	K-means cluster [128]	WFFM	accurate nucleus and cytoplasm detection	only for attached cells
		Potential edge [130]	WFFM	simple operation	only for poly-body; not for images without poly-body; susceptible to noises and blurs
		Polybead attachment [132]	WFFM	high compatibility	complicated attachment procedures; susceptible to noises and surroundings
		Inherent features [133]	WFFM	high compatibility	need individual selection; susceptible to noises and surroundings
		Bright point filter [135]	STEDM	simple operation	susceptible to interferences of surrounding bright points
		Motion history image [136]	WFFM	simple operation; high-speed	only for simple application with no interferences
		Reinitialization and recovery mechanism [137]	WFFM	specimen contact and appearance tracking	complicated procedures, slow speed
	Tracking	Confidence-based hybrid method [138]	WFFM	high tracking robustness	complicated procedures, slow speed
		Modified joint probabilistic data association filter [139]	WFFM	multiple sperm tracking; physical property measurement; high tracking robustness	only for sperm tracking; limited applications;
		Two ROI separation [141,142,143]	WFFM	real-time measurement	complicated procedures; susceptible to noises and blurs
		CFM measurement [144]	CFM	accurate measurement; simple image processing	high sampling light intensity; long sampling time;
		PAM navigation [145]	PAM	deep imaging depth; in vivo measurement	Only for measurement in large tissues and organs
		Actuator activiation and tracking [146]	TPFM	long term measurement; single cell manipulation	Only for photosensitive microrobots
		Micropipette tip template [147]	WFFM	simple operation	anterior–posterior ambiguity; new template requirement;
		Gradient template [148]	WFFM	ambiguity elimination; improved accuracy	susceptible to interferences;new template requirement
Depth acquisition	Template matching	Template similarity score differences [149,150]	WFFM	misalignment compensation; suitable for uncalibrated system	complicated procedures; slow speed; new template requirement
		Three-frames template [151]	WFFM	simple operation; fast speed	only for contact judgment
		Centroid template [152]	WFFM	simple operation	only for cell pairing; an aditional template requirement for a new cell
		Entropy computation [153]	WFFM	no manual selection of the template	susceptible to interferences; large number of sampled images for computation; slow speed
		Edge pixel intensity [154]	WFFM	highlighted edge	susceptible to noises and blurs; artifacts existing in edges
	Autofocusing	Depth from border intensity variation [155,156]	WFFM	inaccurate boundary compensation; improved accuracy	complicated procedures; slow speed
		Multiple depth from defocus [157,158]	WFFM	simple operation; fast speed; relative depth acquisition	inaccurate optical model; inaccurate cell diameter estimation
		Non-uniform rational B-splines [159]	WFFM	multiple biomembranes reconstruction; fast reconstruction	inaccurate shape assumption
3D analysis		Four intersections ellipsoid estimation [161]	WFFM	simple reconstruction	inaccurate shape assumption; complicated contact detection
		Alpha-shape algorithm [162]	WFFM	boundary extracted from multiple depths; close-to-real morphology	interferences of noises and blurs; artificial contours
		CFM reconstruction [163,164]	CFM	directed measurement; blurs and noises rejection; simple operation	long sampling time; phototoxicity; high purchase cost
		Noise-regulated maximum likelihood estimation (NRMLE) deconvolution and filtered segmentation algorithm [165]	WFFM	accurate image restoration; elimination of blurs and noises; fluorescence concentration compensation	relatively long computation time; only for immobilized specimens
		Computational 3D geometry analysis [166]	SIM	high resolution; Long-term measurement	only for structure analysis of immobilized specimens

, WFFM seems to be mainstream for most micromanipulation applications. Despite the limitation of a relatively low imaging resolution, WFFM has the advantages of great compatibility and low purchase costs. Before choosing sampling and processing methods, researchers are encouraged to take comprehensive consideration in terms of manipulation targets, tools, and tasks before choosing sampling and processing methods.