MTRNet++: One-stage mask-based scene text eraser

Highlights

• The proposed MTRNet++ has a novel one-stage mask-based architecture.

• MTRNet++ achieves state-of-the-art results on the Oxford and SCUT datasets.

• MTRNet++ is end-to-end trainable. It converges on a large-scale dataset within an epoch.

• MTRNet++ demonstrates controllability and interpretability.

• We introduced some incremental modifications regarding training losses and strategy.

Abstract

A precise, controllable, interpretable and easily trainable text removal approach is necessary for both user-specific and large-scale text removal applications. To achieve this, we propose a one-stage mask-based text inpainting network, MTRNet++. It has a novel architecture that includes mask-refine, coarse-inpainting and fine-inpainting branches, and attention blocks. With this architecture, MTRNet++ can remove text either with or without an external mask. It achieves state-of-the-art results on both the Oxford and SCUT datasets without using external ground-truth masks. The results of ablation studies demonstrate that the proposed multi-branch architecture with attention blocks is effective and essential. It also demonstrates controllability and interpretability.

Introduction

Text removal is the task of inpainting text regions in scenes with semantically correct backgrounds. It is useful for privacy protection, image/video editing, and image retrieval (Tursun et al., 2019a). Recent studies with advanced deep learning models (Zhang et al., 2018, Tursun et al., 2019b, Nakamura et al., 2017) explore removing text from real world scenes. Previously, text removal with traditional methods has only been effective for text in fixed positions with a uniform background from digital-born content.

Text removal is a challenging task as it inherits the challenges of both text-detection and inpainting tasks. Zhang et al. (2018) claims that text removal requires stroke-level text localisation, which is a harder and less studied topic compared to bounding-box-level scene text detection (Gupta et al., 2016). On the other hand, realistic inpainting requires replacing/filling unwanted objects in scenes with perceptually plausible content. Deep learning approaches generate perceptually plausible content by learning the distribution of training data. Text can be placed in any region or on any object, and as such a text inpainting model is required to learn a wide range of distributions, exacerbating the challenge.

Recent text removal studies are based around two main paradigms: one-stage and two-stage methods. The one-stage approach uses an end-to-end model, which is free of auxiliary inputs; however, its generality, flexibility and interpretability are limited. EnsNet (Zhang et al., 2018) is the state-of-the-art one-stage text removal method. It removes all text that is written in scripts on which it was trained. Although it supports selective text removal with some extra steps (cropping or overlapping), this introduces new issues such as colour discontinuities or loss of context. Moreover, it lacks interpretability that makes fixing failure cases troublesome. Finally, yet importantly, we found that a one-stage approach without an explicit text localisation mechanism fails to converge on a large-scale dataset within a few epochs. For example, Pix2Pix (Isola et al., 2017) and EnsNet do not converge on a large-scale dataset with the same amount of training as their counterparts did according to the results reported in MTRNet (Tursun et al., 2019b) and this work.

The two-stage approach decomposes the text removal task into text detection and text inpainting sub-problems. Text detection can be manual or automatic. MTRNet (Tursun et al., 2019b), for example, is a representative inpainting stage of the two-stage approach with a deep neural network. The main advantage of a two-stage approach is that it has an awareness of the text regions that require inpainting. With explicit text regions, it gains generality and controllability. Two stage approaches can be effortlessly adapted to remove text in various scripts, and are able to remove or keep text based on selection. Such methods have strong interpretability as well; for example it is easier to understand if failure cases are caused by inaccurate detection or poor inpainting. Despite the promise of two-stage methods, they are limited in that they rely on a text localisation front-end. Compared to a one-stage approach, two stage approaches are inefficient and have a complex training process as at least two networks need to be trained.

In this work, we propose a one-stage text removal approach, MTRNet++. It can remove text either with a mask as per MTRNet, or without a mask as per EnsNet as shown in Fig. 1. It inherits the idea of MTRNet of using a text region mask as an auxiliary input. However, the network has a different architecture to MTRNet, and is composed of mask-refine, coarse-inpainting, and fine-inpainting branches. The mask-refine and coarse-inpainting branches generate intermediate results, while the fine-inpainting branch generates the final refined results. Moreover, compared to MTRNet, it can remove text under very coarse masks as shown in Figs. 1c and 1d.

The mask-refine branch is designed to refine a coarse mask into an accurate pixel-level mask. The mask-refining branch is introduced for the following reasons: (1) To remove text as EnsNet does without requiring third-party text-localisation information, but also to allow utilisation of third-party information as done by MTRNet. (2) To provide interpretability, flexibility and a generalisation ability. (3) To provide attention scores to the coarse-inpainting branch. (4) To speed up the convergence of the network on a large-scale dataset.

The coarse-inpainting branch is a parallel branch to the mask-refine branch, which performs a coarse inpainting using the coarse mask. The coarse-to-fine framework has been shown to be beneficial for realistic inpainting (Yu et al., 2018b, Ma et al., 2019), and the coarse-branch is guided by the attention scores generated by the attention blocks that map intermediate features of the mask-refine branch to weights.

The fine-inpainting branch is introduced for refining the results of the coarse-inpainting branch with precise masks from the mask-refine branch. The coarse results are blurry and lack details. The fine-inpainting branch increases inpainting quality. In this work, for efficiency, a light sub-network is used as the fine-inpainting branch.

In summary, our contributions are as follows:

• We propose a novel one-stage architecture for text removal. With this architecture, MTRNet++ is free from external text localisation methods, yet can also leverage external information.

• MTRNet++ achieves state-of-the-art quantitative and qualitative results both on Oxford (Gupta et al., 2016) and SCUT (Zhang et al., 2018) datasets without external masks. Ablation studies shows the proposed architecture and its components play important roles.

• MTRNet++ is a fully end-to-end trainable network and is easily trainable. It converges on large-scale datasets within an epoch. It also demonstrates controllability and interpretability.

We also introduce other incremental modifications regarding training losses, training strategy and the discriminator, which will be discussed in Section 3.

The rest of the paper is organised as follows. The next section presents related literature. Section 3 illustrates the proposed network architecture (generator and discriminator), training losses and training strategy. Section 4 presents experiments, ablation studies and analysis. Finally, we provide a brief summarisation of our work in Section 5.