High-speed high dynamic range 3D shape measurement based on deep learning

https://doi.org/10.1016/j.optlaseng.2020.106245Get rights and content

Highlights

  • In this paper, we first attempted to introduce deep learning into the high dynamic range (HDR) 3D shape measurement to solve the dilemma between high-speed and HDR in the conventional N-step phase-shifting.

  • Our method can improve the dynamic range of three-step phase-shifting by a factor of 4.8 without any additional projected images or hardware adjustment during measurement.

  • The proposed method can realize HDR 3D shape measurement at 13.89 Hz within a depth precision of 0.1 mm

Abstract

For many three-dimensional (3D) measurement techniques based on fringe projection profilometry (FPP), measuring the objects with a large variation range of surface reflectivity is always a very tricky problem due to the limited dynamic range of camera. Many high dynamic range (HDR) 3D measurement methods are developed for static scenes, which are fragile for dynamic objects. In this paper, we address the problem of phase information loss in HDR scenes, in order to enable 3D reconstruction from saturated or dark images by deep learning. By using a specifically designed convolutional neural network (CNN), we can accurately extract phase information in both the low signal-to-noise ratio (SNR) and saturation situations after proper training. Experimental results demonstrate the success of our network in 3D reconstruction for both static and dynamic HDR objects. Our method can improve the dynamic range of three-step phase-shifting by a factor of 4.8 without any additional projected images or hardware adjustment during measurement. And the final 3D measurement speed of our method is about 13.89 Hz (off-line).

Introduction

In recent years, with the development of technology, the three-dimensional (3D) information acquisition and processing is becoming more and more popular. Fringe projection profilometry (FPP), as a 3D measurement technique, has been extensively applied in many fields such as manufacturing, education, and entertainment industry [1], [2], [3]. A typical FPP system consists of one projection device and several cameras. The projection device is used to project coded fringe patterns to measured objects. Due to the limited dynamic range of camera, in actual measurement, image saturation inevitably occurs when the measured scene contains objects with high reflectivity. If directly reducing the projected light intensity (or exposure time), it will lead to a low signal-to-noise ratio (SNR) of objects with low reflectivity in the measured scene. To solve this problem, many high dynamic range (HDR) methods have been proposed. In general, they can be classified into two categories: the hardware-based methods and the algorithm-based methods.

As the name suggests, the hardware-based methods solve the HDR problem mainly by adjusting system hardware parameters. Among these methods, the multi-exposure method is the most widely used method. In photograph, images taken with long exposure times reveal details in dimly illuminated areas, while images taken with short exposure times provide details in highly illuminated areas. The multi-exposure method improves the dynamic range by merging these images with different details into a single HDR image [4]. Similarly, adjusting the projected light intensities has the same effect [5] and it can be considered as another form of multi-exposure. The multi-exposure method has advantages of broad measuring dynamic range and simple operation. However, its disadvantages are also obvious. For an arbitrary measured object, it is hard to rapidly find its proper exposure times, because of lacking its surface reflectance information. In order to obtain high-quality 3D measurement results, one has to use as many exposure times (or projected light intensities) as possible which causes the problem of low measurement efficiency. On the other hand, before collecting enough images with different exposures, the measured objects need to stay still, which means the multi-exposure method only applies to static measurement. In order to optimize the multi-exposure method, many improved methods have been proposed in recent years. These methods can avoid the experiment blindness, making the measurement process more efficient [6], [7], [8], [9], [10]. However, their measurement speed is still too slow for dynamic measurement. To increase the measurement speed, Suresh et al. proposed a optimized multi-exposure method based on digital-light-processing (DLP) technology [11]. This method has good real-time performance, but its improvement of dynamic range is limited. How to significantly improve the dynamic range with a satisfactory measurement speed is crucial in broadening the application of multi-exposure based HDR methods.

Different from the hardware-based methods, the algorithm-based methods do not adjust hardware parameters at all during measurement, and thus their measurement speed is much faster than that of most hardware-based methods. They aim to eliminate the measurement error caused by HDR in mathematical way, which can be considered as a kind of post-processing compensation method. In FPP, the sinusoidal fringe patterns are the most used patterns, due to their robustness to noise and ability to achieve high resolution [12]. For the sinusoidal fringe patterns, most algorithm-based methods are designed to eliminate the phase error caused by saturation. Their core idea is increasing the number of fringe patterns with different light intensities to ensure at least three unsaturated intensity values on the same pixel [13]. Some simple operations can achieve this effect, for example, increasing the number of the phase-shifting step [14] or inserting inverted fringe patterns [15]. It can be seen that their improvement of dynamic range is at the expense of increasing projected fringe patterns. Projecting too much patterns will bring motion error [16]. Therefore, it is also very difficult for the algorithm-based methods to solve the dilemma between improving the dynamic range and ensuring the real-time performance.

In this paper, we first attempted to introduce deep learning into the HDR 3D shape measurement to solve the dilemma between high-speed and HDR. Deep learning is a very powerful tool derived from machine learning [17], [18], [19]. It is developing very quickly nowadays and has showed its wide prospects of application in many fields. For deep learning, if providing enough amounts of training data, an optimized neural network structure can effectively eliminate the phase error caused by HDR. Based on this finding, we can improve the dynamic range of three-step phase-shifting by a factor of 4.8 without any additional fringe images or adjustment of hardware during measurement. To further improve the measurement speed, a stereo phase unwrapping (SPU) technique [20] is used to calculate the unwrapped phase (absolute phase). Experimental results will be presented to verified the success of the proposed method.

Section snippets

N-step phase-shifting method

A sinusoidal fringe pattern can be mathematically described as follows:Ikp=A+Bcos(ϕ2kπN),k=0,1,,N1,where A is the average intensity, B is the intensity modulation, N is the phase-shifting step, k is the phase-shift index, and ϕ is the phase to be measured. The corresponding image captured by camera can be expressed asIkc=αtrIkp+In,where α is the camera sensitivity, t is the camera exposure time, r is the surface reflectivity of measured objects, and In is the random noise. Assuming α and t

Experiments

We built a hardware system to evaluate the proposed method. The system consists of two parts: the data collection part and data training part. The data collection part contains a projector (model: TI LightCrafter4500) with a resolution of 912 × 1140 and two 8-bit cameras (model: Basler acA640-750um) with a resolution of 640 × 480. Each camera is outfitted with a 8 mm focal length lens (model: Computar M0814-MP2). The data training part contains a deep learning workstation with a CPU (model:

Conclusions and discussion

This paper provides a new concept for HDR 3D measurement, which is using the powerful calculating ability of deep learning to eliminate the phase error caused by HDR. In this paper, we have demonstrated that a deep learning network can perform high accuracy phase recovery in both the low SNR and saturation situations with only three fringe images. Compared with the conventional phase-shifting methods, our method can reduce the number of projected fringe patterns, thus effectively improving the

Funding

This research was funded by National Natural Science Fund of China (61722506, 61705105, 111574152), National Key R&D Program of China (2017YFF0106403), Final Assembly “13th Five-Year Plan Advanced Research Project of China (30102070102), Equipment Advanced Research Fund of China (61404150202), The Key Research and Development Program of Jiangsu Province, China (BE2017162), Outstanding Youth Foundation of Jiangsu Province of China (BK20170034), National Defense Science and Technology Foundation

CRediT authorship contribution statement

Liang Zhang: Methodology, Validation, Writing - original draft, Visualization, Software, Investigation. Qian Chen: Writing - review & editing, Supervision, Funding acquisition. Chao Zuo: Resources, Writing - review & editing, Project administration, Funding acquisition. Shijie Feng: Conceptualization, Writing - review & editing, Funding acquisition.

Declaration of Competing Interest

The authors declare no conflicts of interest.

References (36)

  • J. Geng

    Structured-light 3d surface imaging: a tutorial

    Adv Opt Photon

    (2011)
  • F. Blais

    Review of 20 years of range sensor development

    Proc SPIE Int Soc Opt Eng

    (2004)
  • S. Zhang et al.

    High dynamic range scanning technique

    Opt Eng

    (2009)
  • C. Waddington et al.

    Saturation avoidance by adaptive fringe projection in phase-shifting 3d surface-shape measurement

    Optomechatronic technologies (ISOT), 2010 international symposium on

    (2010)
  • D. Li et al.

    Adaptive fringe-pattern projection for image saturation avoidance in 3d surface-shape measurement

    Opt Express

    (2014)
  • C. Chen et al.

    Adaptive pixel-to-pixel projection intensity adjustment for measuring a shiny surface using orthogonal color fringe pattern projection

    Meas Sci Technol

    (2018)
  • C. Zhang et al.

    A robust surface coding method for optically challenging objects using structured light

    IEEE Trans Autom Sci Eng

    (2014)
  • H. Lin et al.

    Adaptive digital fringe projection technique for high dynamic range three-dimensional shape measurement

    Opt Express

    (2016)
  • Cited by (63)

    • A multi-task learning-based approach for single-projection HDR 3D reconstruction

      2024, Measurement: Journal of the International Measurement Confederation
    • Three-dimensional reconstruction of moving HDR object based on PSP

      2023, Optics and Lasers in Engineering
      Citation Excerpt :

      Niu et al. [20] used frequency division multiplexing to separate sub-images with different exposure values from a single snapshot image, and then performed HDR image fusion to obtain the final fringe image. Combining the method of deep learning, Zhang et al. [21] solved the problem of phase information loss in HDR scenes through deep learning, which uses the convolutional neural network to process fringe images to obtain accurate phase information. Liu et al. [22] proposed to reconstruct the HDR object based on Fourier transform profilometry.

    View all citing articles on Scopus
    View full text