While many image colorization algorithms have recently shown the capability of producing plausible color versions from gray-scale photographs, they still suffer from limited semantic understanding. To address this shortcoming, we propose to exploit pixelated object semantics to guide image colorization. The rationale is that human beings perceive and distinguish colors based on the semantic categories of objects. Starting from an autoregressive model, we generate image color distributions, from which diverse colored results are sampled. We propose two ways to incorporate object semantics into the colorization model: through a pixelated semantic embedding and a pixelated semantic generator. Specifically, the proposed network includes two branches. One branch learns what the object is, while the other branch learns the object colors. The network jointly optimizes a color embedding loss, a semantic segmentation loss and a color generation loss, in an end-to-end fashion. Experiments on Pascal VOC2012 and COCO-stuff reveal that our network, when trained with semantic segmentation labels, produces more realistic and finer results compared to the colorization state-of-the-art.

中文翻译：

---

像素化语义着色

虽然最近许多图像着色算法已经显示出从灰度照片生成可信颜色版本的能力，但它们仍然受到语义理解有限的困扰。为了解决这个缺点，我们建议利用像素化对象语义来指导图像着色。其基本原理是人类根据对象的语义类别来感知和区分颜色。从自回归模型开始，我们生成图像颜色分布，从中采样不同的颜色结果。我们提出了两种将对象语义合并到着色模型中的方法：通过像素化语义嵌入和像素化语义生成器。具体来说，提议的网络包括两个分支。一个分支学习物体是什么，而另一个分支学习物体的颜色。该网络以端到端的方式联合优化颜色嵌入损失、语义分割损失和颜色生成损失。在 Pascal VOC2012 和 COCO-stuff 上的实验表明，与最先进的着色技术相比，我们的网络在使用语义分割标签进行训练时会产生更真实、更精细的结果。