Learning color space adaptation from synthetic to real images of cirrus clouds

Abstract

Cloud segmentation plays a crucial role in image analysis for climate modeling. Manually labeling the training data for cloud segmentation is time-consuming and error-prone. We explore to train segmentation networks with synthetic data due to the natural acquisition of pixel-level labels. Nevertheless, the domain gap between synthetic and real images significantly degrades the performance of the trained model. We propose a color space adaptation method to bridge the gap, by training a color-sensitive generator and discriminator to adapt synthetic data to real images in color space. Instead of transforming images by general convolutional kernels, we adopt a set of closed-form operations to make color-space adjustments while preserving the labels. We also construct a synthetic-to-real cirrus cloud dataset SynCloud and demonstrate the adaptation efficacy on the semantic segmentation task of cirrus clouds. With our adapted synthetic data for training the semantic segmentation, we achieve an improvement of $6.59\%$ when applied to real images, superior to alternative methods.

Appendix

1.1 A: Color adjustment operations

Brightness The brightness adjustment operation is defined as

$$\begin{aligned} \textsf {op}_b(x; \alpha _b)= {\left\{ \begin{array}{ll} x\cdot (1-\alpha _b)+\alpha _b, \quad &{}\text {if}\ \alpha _b>=0\\ x+x\cdot \alpha _b, \quad &{}\text {otherwise} \end{array}\right. } \end{aligned}$$

(7)

Fig. 12

More color adaptation results

Fig. 13

Styles overview. Retouched images of artist A (left) and artist B (right) from www.pexels.com

where x is the input image, and $\alpha _b$ is a scalar parameter that controls the extent of the adjustment. We clip $\alpha _b$ into the range $[-1,1]$.

Saturation The saturation adjustment operation is defined as

$$\begin{aligned} \textsf {op}_s(x; \alpha _s)= {\left\{ \begin{array}{ll} x+(x-\textsf {L}(x))\cdot \textsf {s}(x,\alpha _s), &{}\text {if}\ s>0\\ \textsf {L}(x)+(x-\textsf {L}(x))\cdot (1+\textsf {s}(x,\alpha _s)), &{}\text {otherwise} \end{array}\right. }\nonumber \\ \end{aligned}$$

(8)

where x is the input image and $\alpha _s$ is a scalar parameter that controls the extent of the adjustment. We clip $\alpha _s$ into the range $[-1,1]$.

$\textsf {L}(x)$ is the per-pixel average of the three channels $\frac{1}{2}\cdot [\textsf {rgb\_max}(x) + \textsf {rgb\_min}(x)]$, and $\textsf {s}(x,\alpha _s)$ is defined as

$$\begin{aligned} \textsf {s}(x; \alpha _s) = {\left\{ \begin{array}{ll} 1/\textsf {S}(x)-1, &{}\text {if}\ \alpha _s+\textsf {S}(x)>=1\\ 1/(1-\alpha _s)-1, &{}\text {otherwise} \end{array}\right. } \end{aligned}$$

$\textsf {S}(x)$ is defined as a per-pixel ratio

$$\begin{aligned} \textsf {S}(x) = {\left\{ \begin{array}{ll} \textsf {delta}(x)/(2\cdot \textsf {L}(x)), &{}\text {if}\ \textsf {L}<0.5\\ \textsf {delta}(x)/(2-2\cdot \textsf {L}(x)), &{}\text {otherwise} \end{array}\right. } \end{aligned}$$

where $\textsf {delta}(x) = \textsf {rgb\_max}(x) - \textsf {rgb\_min}(x)$.

Contrast The contrast adjustment operation is defined as

$$\begin{aligned} \textsf {op}_c(x; \alpha _c)= {\left\{ \begin{array}{ll} \bar{x}+(x-\bar{x})/(1-\alpha _c), &{}\text {if}\ \alpha _c>=0\\ \bar{x}+(x-\bar{x})\cdot (1+\alpha _c), &{}\text {otherwise} \end{array}\right. } \end{aligned}$$

(9)

where x is the input image, $\bar{x}$ is the average of all pixel values of x, and $\alpha _c$ is a scalar parameter that controls the extent of the adjustment. We clip $\alpha _c$ into the range $[-1,1]$ (Fig. 12).

1.2 B: Handcrafted approach for image augmentation

We execute an adaptation approach using handcrafted features in the experiment. First, we transfer the images to HSV space to extract features of saturation and brightness. Then, we fit a Gaussian distribution model to the feature points of the real images. Next, for each synthetic image, we shift its features toward a target point sampled from the Gaussian distribution model. Finally, we reconstruct augmented images from the shifted features. Compared with the handcrafted feature, our generator learns more powerful features in higher dimensions and leverages that to decide the best way to shift each synthetic image (Fig. 13).

Fig. 14

Adaptation on Pexels. Our method adapts the photos of artist A (top row) to the style of artist B (bottom row)

Fig. 15

Cross-dataset generalization. The model trained on the Pexels dataset is applied to raw images of the MIT-Adobe FiveK Dataset (top row) to obtain an artist style (bottom row)

1.3 C: Details of style transfer and image post-processing

All the training images are zero-centered and rescaled to $[-1, 1]$. We set the batch size to 8. We adopt the Adam optimizer with $lr=2\mathrm {e}{-4}$ and $\beta _1=0.5$ and train both the discriminator and the generator for 100 epochs. We show more color adaptation results on the Pexels dataset in Fig. 14. We also apply the model trained on the Pexels dataset to images in the MIT-FiveK dataset to show the ability of cross-dataset generalization (Fig. 15). While similar effects can be produced by [47], our method does not require reinforcement learning.