High-speed high dynamic range 3D shape measurement based on deep learning
Introduction
In recent years, with the development of technology, the three-dimensional (3D) information acquisition and processing is becoming more and more popular. Fringe projection profilometry (FPP), as a 3D measurement technique, has been extensively applied in many fields such as manufacturing, education, and entertainment industry [1], [2], [3]. A typical FPP system consists of one projection device and several cameras. The projection device is used to project coded fringe patterns to measured objects. Due to the limited dynamic range of camera, in actual measurement, image saturation inevitably occurs when the measured scene contains objects with high reflectivity. If directly reducing the projected light intensity (or exposure time), it will lead to a low signal-to-noise ratio (SNR) of objects with low reflectivity in the measured scene. To solve this problem, many high dynamic range (HDR) methods have been proposed. In general, they can be classified into two categories: the hardware-based methods and the algorithm-based methods.
As the name suggests, the hardware-based methods solve the HDR problem mainly by adjusting system hardware parameters. Among these methods, the multi-exposure method is the most widely used method. In photograph, images taken with long exposure times reveal details in dimly illuminated areas, while images taken with short exposure times provide details in highly illuminated areas. The multi-exposure method improves the dynamic range by merging these images with different details into a single HDR image [4]. Similarly, adjusting the projected light intensities has the same effect [5] and it can be considered as another form of multi-exposure. The multi-exposure method has advantages of broad measuring dynamic range and simple operation. However, its disadvantages are also obvious. For an arbitrary measured object, it is hard to rapidly find its proper exposure times, because of lacking its surface reflectance information. In order to obtain high-quality 3D measurement results, one has to use as many exposure times (or projected light intensities) as possible which causes the problem of low measurement efficiency. On the other hand, before collecting enough images with different exposures, the measured objects need to stay still, which means the multi-exposure method only applies to static measurement. In order to optimize the multi-exposure method, many improved methods have been proposed in recent years. These methods can avoid the experiment blindness, making the measurement process more efficient [6], [7], [8], [9], [10]. However, their measurement speed is still too slow for dynamic measurement. To increase the measurement speed, Suresh et al. proposed a optimized multi-exposure method based on digital-light-processing (DLP) technology [11]. This method has good real-time performance, but its improvement of dynamic range is limited. How to significantly improve the dynamic range with a satisfactory measurement speed is crucial in broadening the application of multi-exposure based HDR methods.
Different from the hardware-based methods, the algorithm-based methods do not adjust hardware parameters at all during measurement, and thus their measurement speed is much faster than that of most hardware-based methods. They aim to eliminate the measurement error caused by HDR in mathematical way, which can be considered as a kind of post-processing compensation method. In FPP, the sinusoidal fringe patterns are the most used patterns, due to their robustness to noise and ability to achieve high resolution [12]. For the sinusoidal fringe patterns, most algorithm-based methods are designed to eliminate the phase error caused by saturation. Their core idea is increasing the number of fringe patterns with different light intensities to ensure at least three unsaturated intensity values on the same pixel [13]. Some simple operations can achieve this effect, for example, increasing the number of the phase-shifting step [14] or inserting inverted fringe patterns [15]. It can be seen that their improvement of dynamic range is at the expense of increasing projected fringe patterns. Projecting too much patterns will bring motion error [16]. Therefore, it is also very difficult for the algorithm-based methods to solve the dilemma between improving the dynamic range and ensuring the real-time performance.
In this paper, we first attempted to introduce deep learning into the HDR 3D shape measurement to solve the dilemma between high-speed and HDR. Deep learning is a very powerful tool derived from machine learning [17], [18], [19]. It is developing very quickly nowadays and has showed its wide prospects of application in many fields. For deep learning, if providing enough amounts of training data, an optimized neural network structure can effectively eliminate the phase error caused by HDR. Based on this finding, we can improve the dynamic range of three-step phase-shifting by a factor of 4.8 without any additional fringe images or adjustment of hardware during measurement. To further improve the measurement speed, a stereo phase unwrapping (SPU) technique [20] is used to calculate the unwrapped phase (absolute phase). Experimental results will be presented to verified the success of the proposed method.
Section snippets
N-step phase-shifting method
A sinusoidal fringe pattern can be mathematically described as follows:where A is the average intensity, B is the intensity modulation, N is the phase-shifting step, k is the phase-shift index, and ϕ is the phase to be measured. The corresponding image captured by camera can be expressed aswhere α is the camera sensitivity, t is the camera exposure time, r is the surface reflectivity of measured objects, and In is the random noise. Assuming α and t
Experiments
We built a hardware system to evaluate the proposed method. The system consists of two parts: the data collection part and data training part. The data collection part contains a projector (model: TI LightCrafter4500) with a resolution of 912 × 1140 and two 8-bit cameras (model: Basler acA640-750um) with a resolution of 640 × 480. Each camera is outfitted with a 8 mm focal length lens (model: Computar M0814-MP2). The data training part contains a deep learning workstation with a CPU (model:
Conclusions and discussion
This paper provides a new concept for HDR 3D measurement, which is using the powerful calculating ability of deep learning to eliminate the phase error caused by HDR. In this paper, we have demonstrated that a deep learning network can perform high accuracy phase recovery in both the low SNR and saturation situations with only three fringe images. Compared with the conventional phase-shifting methods, our method can reduce the number of projected fringe patterns, thus effectively improving the
Funding
This research was funded by National Natural Science Fund of China (61722506, 61705105, 111574152), National Key R&D Program of China (2017YFF0106403), Final Assembly “13th Five-Year Plan Advanced Research Project of China (30102070102), Equipment Advanced Research Fund of China (61404150202), The Key Research and Development Program of Jiangsu Province, China (BE2017162), Outstanding Youth Foundation of Jiangsu Province of China (BK20170034), National Defense Science and Technology Foundation
CRediT authorship contribution statement
Liang Zhang: Methodology, Validation, Writing - original draft, Visualization, Software, Investigation. Qian Chen: Writing - review & editing, Supervision, Funding acquisition. Chao Zuo: Resources, Writing - review & editing, Project administration, Funding acquisition. Shijie Feng: Conceptualization, Writing - review & editing, Funding acquisition.
Declaration of Competing Interest
The authors declare no conflicts of interest.
References (36)
- et al.
A state of the art in structured light patterns for surface profilometry
Pattern Recognit
(2010) - et al.
General solution for high dynamic range three-dimensional shape measurement using the fringe projection technique
Opt Lasers Eng
(2014) - et al.
High-dynamic-range 3d shape measurement utilizing the transitioning state of digital micromirror device
Opt Lasers Eng
(2018) - et al.
Phase shifting algorithms for fringe projection profilometry: a review
Opt Lasers Eng
(2018) - et al.
Further study of the phase-recovering algorithm for saturated fringe patterns with a larger saturation coefficient in the projection grating phase-shifting profilometry
Opt-Int J Light Electron Opt
(2010) - et al.
Robust dynamic 3-d measurements with motion-compensated phase-shifting profilometry
Opt Lasers Eng
(2018) Deep learning in neural networks: an overview
Neural networks
(2015)- et al.
High-quality 3d shape measurement using saturated fringe patterns
Opt Lasers Eng
(2016) - et al.
Micro fourier transform profilometry (μftp): 3d shape measurement at 10,000 frames per second
Opt Lasers Eng
(2018) - et al.
Temporal phase unwrapping algorithms for fringe projection profilometry: a comparative review
Opt Lasers Eng
(2016)
Structured-light 3d surface imaging: a tutorial
Adv Opt Photon
Review of 20 years of range sensor development
Proc SPIE Int Soc Opt Eng
High dynamic range scanning technique
Opt Eng
Saturation avoidance by adaptive fringe projection in phase-shifting 3d surface-shape measurement
Optomechatronic technologies (ISOT), 2010 international symposium on
Adaptive fringe-pattern projection for image saturation avoidance in 3d surface-shape measurement
Opt Express
Adaptive pixel-to-pixel projection intensity adjustment for measuring a shiny surface using orthogonal color fringe pattern projection
Meas Sci Technol
A robust surface coding method for optically challenging objects using structured light
IEEE Trans Autom Sci Eng
Adaptive digital fringe projection technique for high dynamic range three-dimensional shape measurement
Opt Express
Cited by (63)
Deep learning in fringe projection: A review
2024, NeurocomputingA multi-task learning-based approach for single-projection HDR 3D reconstruction
2024, Measurement: Journal of the International Measurement ConfederationOverview of modulation techniques for spatially structured-light 3D imaging
2024, Optics and Laser TechnologyAccurate defocusing fringe projection profilometry in a large depth-of-field
2023, Optics and Laser TechnologyThree-dimensional reconstruction of moving HDR object based on PSP
2023, Optics and Lasers in EngineeringCitation Excerpt :Niu et al. [20] used frequency division multiplexing to separate sub-images with different exposure values from a single snapshot image, and then performed HDR image fusion to obtain the final fringe image. Combining the method of deep learning, Zhang et al. [21] solved the problem of phase information loss in HDR scenes through deep learning, which uses the convolutional neural network to process fringe images to obtain accurate phase information. Liu et al. [22] proposed to reconstruct the HDR object based on Fourier transform profilometry.