Inferring spatial relations from textual descriptions of images

Highlights

• A novel dataset (REC-COCO) for spatial inference from text, where textual tokens are linked to bounding boxes in images.

• Experiments that prove the contextual information of textual captions helps inferring the spatial relation between objects.

• We present an experimental analysis of various scenarios to infer spatial relations from text.

• Qualitative results that suggest that neural network architectures can learn prototypical spatial relations between objects.

Abstract

Generating an image from its textual description requires both a certain level of language understanding and common sense knowledge about the spatial relations of the physical entities being described. In this work, we focus on inferring the spatial relation between entities, a key step in the process of composing scenes based on text. More specifically, given a caption containing a mention to a subject and the location and size of the bounding box of that subject, our goal is to predict the location and size of an object mentioned in the caption. Previous work did not use the caption text information, but a manually provided relation holding between the subject and the object. In fact, the used evaluation datasets contain manually annotated ontological triplets but no captions, making the exercise unrealistic: a manual step was required; and systems did not leverage the richer information in captions. Here we present a system that uses the full caption, and Relations in Captions (REC-COCO), a dataset derived from MS-COCO which allows to evaluate spatial relation inference from captions directly. Our experiments show that: (1) it is possible to infer the size and location of an object with respect to a given subject directly from the caption; (2) the use of full text allows to place the object better than using a manually annotated relation. Our work paves the way for systems that, given a caption, decide which entities need to be depicted and their respective location and sizes, in order to then generate the final image.

Introduction

The ability of automatically generating images from textual descriptions is a fundamental skill which can boost many relevant applications, such as art generation and computer-aided design. From a scientific point of view, it also drives research progress in multimodal learning and inference across vision and language, which is currently a very active research area [1]. In the case of scenes comprising several entities, it is necessary to infer which is an adequate scene layout, i.e., which entities to show, their location and size.

From the language understanding perspective, in order to generate realistic images from textual descriptions, it is necessary to infer visual features and relations between the entities mentioned in the text. For example, given the text “a black cat on a table”, an automatic system has to understand that the cat has a certain colour (black) and is situated on top of the table, among other details. In this paper, we focus on the spatial relations between the entities, since they are the key to suitably compose scenes described in texts. The spatial information is sometimes given explicitly, in form of prepositions (“cat on a table”), but more often implicitly, since the verb used to relate two entities contains information about the spatial arrangement of both. For example, from the text (“a woman riding a horse”) it is obvious for humans that the woman is on top of the horse. However, acquiring such spatial relations from text is far from trivial, as this kind of common sense spatial knowledge is rarely stated explicitly in natural language text [2]. That is precisely what text-to-image systems learn, relating both explicit and implicit spatial relations expressed in text with actual visual arrangements showed in images.

A large strand of research in text-to-image generation are evaluated according to the pixel-based quality of the generated images and the global fidelity to the textual descriptions, but do not evaluate whether the entities have been arranged according to the spatial relations mentioned in the text [3]. Closer to our goal, some researchers do focus on learning spatial relations between entities [4], [5], [6], [7], [8], [9]. For instance, in Gupta and Malik [6], Krishna et al. [8] the authors proposed to associate actions along with their semantic arguments (subject and object) with pixels in images (i.e., bounding boxes of entities) as a way towards understanding the images. V-COCO is a dataset which comprises images and manually created Subject, Relation, Object ($S,R,O$) ontological triplets, henceforth called concept triplets, where each $S$ and $O$ is associated with a bounding box in the image [6]. Note that the terms used to describe the triplet concepts are selected manually from among a small vocabulary¹ of an ontology, e.g. PERSON or BOOK, and are not linked to the words in the caption. Visual Genome is constructed similarly [8]. Typically, those datasets are created by showing images to human annotators, and asking them to locate the bounding boxes of the entities participating on predefined relations, and to select the terms for the relation and entities from a reduced vocabulary in a small ontology. Using such a dataset, Collell et al. [5] presents a system that uses concept triplets to infer the spatial relation between the subject $S$ and the object $O$. Given the bounding box of the subject, the system outputs the location and size of the bounding box of the object. Evaluation is done checking whether the predicted bounding box matches the actual bounding box in the image. The datasets and systems in the previous work require the use of manually extracted ontological triplets, and systems did not use the actual captions, posing two issues: a manual pre-processing step was required; and systems did not use the richer information in captions.

In this paper we propose to study the use of full captions instead of manually selected relations when inferring the spatial relations between two entities, where one of them is considered the subject and the other is the object of the action being described by the relation. The problem we address is depicted in Fig. 1. Given a textual description of an image and the location (bounding box) of the subject of the action in the description, we want the system to predict the bounding box of the target object. Note that we do not use the actual pixels for this task, but we include Fig. 2 for illustrative purposes. To the best of our knowledge, there is no previous work addressing the same problem, i.e. nobody studied before whether using full captions instead of concept triplets benefits spatial relation inference.

Our hypothesis is that the textual description² accompanying the image contains information that helps inferring the spatial relations of two entities. We argue that the information presented in manually created triplets alone is often insufficient to properly infer spatial relations. As a motivation, Fig. 2 shows pairs of examples (left and right) where the relation between the subject and the object (given by a verb) is not enough to correctly predict the spatial relation between them. In each row there are two examples for the same subject, relation and object (e.g. person, reading, book), but the spatial relation between subject and object is different, and depends on the interpretation of the rest of the caption. For instance, in the top-left caption the person is sitting while it is reading a book, so that the book is around the middle of the bounding box for the person, while in the top-right caption the person is laying in bed, and therefore the book is slightly above the person.

To validate the main hypothesis of our work, we created a new dataset called Relations in Captions (REC-COCO) that contains associations between caption tokens and bounding boxes in images. REC-COCO is based on the MS-COCO [10] and V-COCO datasets [6]. For each image in V-COCO, we collect their corresponding captions from MS-COCO and automatically align the concept triplet in V-COCO to the tokens in the caption. This requires finding the token for concepts such as PERSON. As a result, REC-COCO contains the captions and the tokens which correspond to each subject and object, as well as the bounding boxes for the subject and object (cf. Fig. 3).

In addition, we have adapted a well-known state-of-the-art architecture that worked on concept triplets [5] to work also with full captions, and performed experiments which show that: (1) It is possible to infer the size and location of an object with respect to a given subject directly from the caption; (2) The use of the full text of the caption allows to place the object better than using the manually extracted relation.

The main contributions of the work are the following:

• We show for the first time that the textual description includes information that is complementary to the relation between a subject and an object. From another perspective, our work shows that, given a caption, a reference subject and an object in the caption, our system can assign a location and a size to the object using the information in the caption, without any manually added relation.

• We introduce a new dataset created for this task. The dataset comprises pairs of images and captions, including, for each pair, the tokens in the caption that describe the subject and object, and the bounding boxes of subject and object. The dataset is publicly available under a free license.³