Rain Rendering for Evaluating and Improving Robustness to Bad Weather

Abstract

Rain fills the atmosphere with water particles, which breaks the common assumption that light travels unaltered from the scene to the camera. While it is well-known that rain affects computer vision algorithms, quantifying its impact is difficult. In this context, we present a rain rendering pipeline that enables the systematic evaluation of common computer vision algorithms to controlled amounts of rain. We present three different ways to add synthetic rain to existing images datasets: completely physic-based; completely data-driven; and a combination of both. The physic-based rain augmentation combines a physical particle simulator and accurate rain photometric modeling. We validate our rendering methods with a user study, demonstrating our rain is judged as much as 73% more realistic than the state-of-the-art. Using our generated rain-augmented KITTI, Cityscapes, and nuScenes datasets, we conduct a thorough evaluation of object detection, semantic segmentation, and depth estimation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection, 60% for semantic segmentation, and 6-fold increase in depth estimation error. Finetuning on our augmented synthetic data results in improvements of 21% on object detection, 37% on semantic segmentation, and 8% on depth estimation.

Appendices

A Field of View of a Drop in a Sphere

Fig. 19

Geometrical construction to compute a drop FOV. Considering $\mathbf {X}_\mathbf{0}$ and $\mathbf {X}_\mathbf{1}$ the drop position at shutter opening and closing, respectively. We assume a constant drop position $\mathbf {X} = \frac{\mathbf {X}_\mathbf{0}+\mathbf {X}_\mathbf{1}}{2}$ (during the exposure time, a few milliseconds). Note that we drew only a slice of the drop FOV for simplicity but a full 3D visualization would show a full 3D cone. The drop FOV in the environment map is the projection of the 3D drop FOV on the scene sphere of constant distance (refer to text for details)

We estimate the field of view (FOV) of a drop when projected on a sphere to compute the radiance and chromaticity of each streak, as detailed in Sect. 3.1.2 of the main paper. Despite its motion, we make the assumption of a constant field of view within a given exposure time. This is acceptable due to the short exposure time used here (i.e. 2 ms for KITTI, 5 ms for Cityscapes). For each drop, the simulator outputs the start position (i.e. shutter opening) and end position (i.e. shutter closing) in both the 3D camera-centered and the 2D image coordinate frames.

We refer to the Fig. 19 for a geometrical illustration of the following. Let us consider an imaged drop D, having 3D start position $\mathbf {X}_\mathbf{0}$ and end position $\mathbf {X}_\mathbf{1}$. We first compute $\mathbf {X} = \frac{\mathbf {X}_\mathbf{0}+\mathbf {X}_\mathbf{1}}{2}$ the assumed constant position for which we will estimate the corresponding FOV. The position being camera-centered, the drop viewing direction is therefore $\mathbf {d}=\frac{\mathbf {X}}{||\mathbf {X}||}$.

We compute the equation of the plane P going through $\mathbf {X}$ and orthogonal to the viewing direction $\mathbf {d}$:

$$\begin{aligned} P = \mathbf {d}_x+\mathbf {d}_y+\mathbf {d}_z-\mathbf {d}\cdot \mathbf {X} = 0, \end{aligned}$$

(6)

where $\cdot$ is the dot product and select a random vector $\mathbf {u}$ (with $||\mathbf {u}|| = 1$) lying on P. Accounting for the field of view of the drop $\theta \approx 165^\circ$ (according to Garg and Nayar 2007), we compute an arbitrary vector $\mathbf {v}$ on the viewing cone through the drop

$$\begin{aligned} \mathbf {v} = \mathbf {d}\cdot \mathbf {R}_{\mathbf {u}}(\theta /2), \end{aligned}$$

(7)

with $\mathbf {R}_{\mathbf {u}}(\theta /2)$ the 3x3 general rotation matrix of angle $\theta /2$ about vector $\mathbf {u}$. We use $\theta /2$ because the cone being symmetric along the viewing direction, the complete cone field of view obtain is $\theta$. The set $V'$ of vectors forming the viewing cone through the drop is obtained by the rotation of $\mathbf {v}$ all around the viewing direction. Formally,

$$\begin{aligned} V' = \{\mathbf {v}\cdot \mathbf {R}_{\mathbf {d}}(\alpha ) ~| ~\forall ~\alpha \in [0, 2\pi [\}, \end{aligned}$$

(8)

with $\mathbf {R}_{\mathbf {d}}(\alpha )$ the rotation matrix of $\alpha$ around vector $\mathbf {d}$. In practice, $V'$ is a finite set of radially equidistant vectors (for computational reason we use $|V'| = 20$).

To compute the coordinates of the drop FOV in the environment, we assume a projection sphere S of radius 10m. Hence, we compute the set $Q = \{\phi (S, \mathbf {v'}) ~| ~\forall ~\mathbf {v'} \in V'\}$ of points where vectors intersect the environment sphere, considering only the positive viewing direction axis. Given that the sphere is centered to the camera position and all drops 3D positions are expressed in the camera referential, the intersection $\phi (S, \mathbf {v'})$ of a vector $\mathbf {v'}$ and sphere S of radius $S_{\rho }$ is straight-forward with

$$\begin{aligned} \begin{aligned} \phi (S, \mathbf {v'})&= \mathbf {v'} + t\mathbf {d}\,\text {with},\\ t&= \frac{-b + \sqrt{b^2 - 4ac}}{2a},\\ a&= \mathbf {d}_x^2 + \mathbf {d}_y^2 + \mathbf {d}_z^2,\\ b&= 2(\mathbf {d}_x\mathbf {v'}_x + \mathbf {d}_y\mathbf {v'}_y + \mathbf {d}_z\mathbf {v'}_z),\\ c&= \mathbf {v'}_x^2 + \mathbf {v'}_y^2 + \mathbf {v'}_z^2 - S_{\rho }^2. \end{aligned} \end{aligned}$$

(9)

Having computed Q, the finite set of 3D positions intersecting our environment sphere S, the set $Q'$ of spherical coordinates (azimuth, altitude) are obtained from simple Cartesian to spherical mapping, and directly translated to the environment map. Thus, $Q'$ is the projection of the drop FOV on the environment map.

Table 3 Data splits for all experiments

Accounting for implementation details, one may note that $Q'$ is a discrete representation of the drop field of view contours. In practice, a polygon filling algorithm is used to obtain the drop FOV F, which we use for computing the photometry of a rainstreak (cf. Sect. 3.1.4 of the main paper).

B Compositing a Rain Streak with Different Exposure Time

In their seminal work, Garg and Nayar (2005) demonstrated that the streak appearance is closely related to the amount of time $\tau$ a drop stays on a pixel. It is thus important to account for the difference of exposure time in the streak appearance database (Garg and Nayar 2006) when adding rain to existing images. Given that the appearance database does not provide enough calibration data to recompute the exact original $\tau _0$, we estimate it using observations made in appendix 10.3 of Garg and Nayar (2007). The latter states that for a constant exposure time $\tau$ can be safely approximated with $\sqrt{a}/50$ (a the drop diameter, in meters), which we use to compute $\tau _0$ according to simulation settings in Garg and Nayar (2006).

Using the notation defined in Eq. (5) from the main paper, the radiance of streak $S'$ is corrected with

$$\begin{aligned} S'\frac{\tau _1}{\tau _0}, \end{aligned}$$

(10)

where $\tau _1$ is the time the current drop stays on a pixel, as obtained in a streak-wise fashion by the physical simulator. Noteworthy, Garg and Nayar (2007) also emphasizes that for a given streak the changes of $\tau$ across pixels are negligible, so $\tau$ can safely be assumed constant.

Finally, after normalization, the alpha of each streak is scaled according to $\tau _1$ and the targeted exposure time T. According to Garg and Nayar equations (cf. Eq. (18) from Garg and Nayar 2007), the composite rainy image is an alpha blending of the background image $I_{bg}$ and the rain layer $I_r$. For pixel $\mathbf {x}$ corresponding to $\mathbf {x'}$ in the streak coordinates, it leads to:

$$\begin{aligned} \begin{aligned} I_{rainy}(\mathbf {x})&= \alpha {}I_{bg}(\mathbf {x}) + I_{r}(\mathbf {x'}),\\&= \frac{T-S'_{\alpha }(\mathbf {x'})\tau _1}{T}I(\mathbf {x}) + S'(\mathbf {x'})\frac{\tau _1}{\tau _0}.\\ \end{aligned} \end{aligned}$$

(11)

C Experiments Data Splits

Table 3 contains the minutiae of the data splits of the various experimental steps of this paper.