Comparative analysis on cross-modal information retrieval: A review

Highlights

• Summary on recent progress in the image–text cross-modal retrieval.

• Broad classification of various cross-modal techniques.

• Prominent benchmark datasets and evaluation metrics are introduced.

• Comparative analysis of diverse cross-modal methods.

• Challenges and open issues are presented in the area of multi-modal retrieval.

Abstract

Human beings experience life through a spectrum of modes such as vision, taste, hearing, smell, and touch. These multiple modes are integrated for information processing in our brain using a complex network of neuron connections. Likewise for artificial intelligence to mimic the human way of learning and evolve into the next generation, it should elucidate multi-modal information fusion efficiently. Modality is a channel that conveys information about an object or an event such as image, text, video, and audio. A research problem is said to be multi-modal when it incorporates information from more than a single modality. Multi-modal systems involve one mode of data to be inquired for any (same or varying) modality outcome whereas cross-modal system strictly retrieves the information from a dissimilar modality. As the input–output queries belong to diverse modal families, their coherent comparison is still an open challenge with their primitive forms and subjective definition of content similarity. Numerous techniques have been proposed by researchers to handle this issue and to reduce the semantic gap of information retrieval among different modalities. This paper focuses on a comparative analysis of various research works in the field of cross-modal information retrieval. Comparative analysis of several cross-modal representations and the results of the state-of-the-art methods when applied on benchmark datasets have also been discussed. In the end, open issues are presented to enable the researchers to a better understanding of the present scenario and to identify future research directions.

Introduction

When we fail to understand the contents of an image embedded in a text, figure captions, and referral text often help. Just by looking at a figure, a person might not be able to understand it exactly but with the help of collateral text, it can be understood efficiently. For instance, when we see a volleyball picture (Fig. 1), we may not be able to understand or know about the volleyball game. However, the picture can be completely understood with the help of collateral text (such as caption, figure reference, and related citation) describing the volleyball game. This implies that information from more than one source is beneficial in further understanding of things and also helpful in better information retrieval. This is where cross-modal data fusion and retrieval come into the picture.

Recently, cross-modal retrieval has gained a lot of attention due to the rapid increase in multi-modal data such as images, text, video, and audio. The term modality represents a specific form in which data exists and it is also associated with a sensory perception such as vision and hear modalities which are major sources of communication and responsiveness in human beings and animals. The data consisting of more than one modality is known as multi-modal data. It has the characteristic of high-level semantic homogeneity and low-level expressive heterogeneity such as the same thing having diverse representations. Different forms of representation help people better understand things as illustrated in the volleyball example above. While searching for something, people often want to get accurate results in different forms which create a need for an efficient multi-media information retrieval platform. Classic approaches to information retrieval are of uni-modal nature. Uni-modal means information derived just from one channel, such as only from images or only from the text (but not both). For example, only the text query is used for information search and retrieval from a text repository. This retrieval approach is of the least use these days when enormous multimedia data is being generated. Cross-modal and multi-modal systems, on the other hand, are able to link more than one modalities such as image, text, audio, and video. In cross-modal, input query mode and resultant mode are dissimilar. For example, query text for related images and query image for related text. However, the resultant mode can be similar to the query mode in a multi-modal system. For example, query text to retrieve related images and matched text. Cross-modal and multi-modal are explained using a simple example in Fig. 2 where ＋ represents both text and images can be retrieved using an image query and vice versa in multi-modal approach.

Therefore, the fundamental idea of cross-modal is to integrate numerous modes of information to derive better results than just one channel. For instance, an image–text cross-modal system integrates textual information along with an image which is known as image annotation. Vice-versa, it also queries text keywords to retrieve images, known as image retrieval. In simple words, image annotation is a process of explaining an image with appropriate linguistic cues. It is useful in knowledge transfer sessions for application areas such as medical science, military, business, education, and sports to name a few. For example, a CT scan is known to the radiologist but not to an intern or a patient. Therefore, the expert has to explain it using proper terminology by pointing out key areas on the given image. Image retrieval is a process of retrieving an appropriate image from the database as per the user query, for instance, with text keywords. With the evolution of the semantic web and huge data repositories, a major challenge comes into the picture which is effective indexation and retrieval of both still and moving images and the identification of key areas inside the images. An image cannot be expressed completely just by using visual features only as they under-constrain the information contained in it. Visual features of an image include color distribution, texture, shape, and edges. Typically, image retrieval systems make use of images and the corresponding text/keywords for indexing and retrieving images using both keywords and visual features of the image. Cross-modal image retrieval aims to use text for retrieving relevant images related to the text.

Cross-modal learning has become tremendously popular because of its effective information retrieval capability. Numerous cross-modal representation and retrieval methods have been proposed by researchers to resolve the issue of cross-modal retrieval considering several modalities. Various appealing surveys have been introduced which summarizes the work done in this field. Image and text are the highly utilized modalities and a number of articles on cross-modal retrieval have been published considering these. However, there is no proper survey mainly focusing on the image–text cross-modal retrieval techniques. The objective of this article is to conduct a comprehensive review of cross-modal retrieval which incorporates image and text modalities, the main concerns of which are different from previous surveys and reviews. So, the motivation behind this review article is:

• Lack of a full-fledged review article on image and text modalities.

• To present various challenges and open issues in the cross-modal retrieval field.

• Image and text modalities are the basic and highly utilized modalities, however, we are still away from achieving an ideal level in their cross-modal retrieval process.

Existing literature reviews related to cross-modal information retrieval have presented the topic quite well to the research community. [1] presented an overview of cross-modal retrieval in 2016, however, it does not comprise several significant works proposed in recent years. In [2], authors have presented numerous multi-modal techniques, but their focus is only on techniques based on machine learning. [3] is a contemporary survey, however, it presents a brief study of cross-media retrieval methods compared to the vastness of the topic. An overview of different cross-media retrieval techniques incorporating miscellaneous modalities has been provided in [4]. [5] article only explore various cross-media retrievals with joint graph regularization. The focus of [6] is on cross-media analysis and reasoning and the various analysis methods rather than cross-media retrieval. [7] has provided a survey on cross-media image and text information fusion where the main focus is on analyzing two methods of image and text associations.

Table 1 shows the comparison of the current survey with the existing reviews related to cross-modal learning sorted year wise. Comparison is performed on the basis of the domain, different modalities incorporated in the paper, comparative analysis, challenges, open issues, benchmark datasets, and evaluation metrics. It can be seen in the table that only one survey is focusing on image and text modalities but their main concern is an image–text association and not cross-modal retrieval. A blank cell in the table implies that the information is missing for that particular column and ✓means that it is present in the article. Domain column specifies the main focus of the article and all value under Mode column means that the article is not particularly focusing on any two or three modalities rather it is talking about the whole multi-media. Comparative analysis depicts whether the comparison among techniques has been performed quantitatively or qualitatively.

The significant contributions of this paper are as follows:

• This review focuses to present a summary of recent progress in cross-modal retrieval considering text and image (image-to-text and text-to-image). It comprises several novel works and references which are absent in previous surveys. It will act as a valuable resource for beginners to get acquainted with the topic.

• A broad classification of various cross-modal approaches has been presented and difference among them is also discussed.

• It provides information regarding various prominent benchmark datasets and evaluation metrics utilized for cross-modal method performance estimation.

• It presents a comparative analysis of diverse cross-modal representation techniques when applied on benchmark datasets. This analysis will be highly useful for future research.

• The article summarizes various challenges in the field of cross-modal retrieval and open issues to work upon by future researchers.

This article starts with an introduction of cross-modal retrieval in Section 1 which includes motivation for the survey, contributions, comparison with existing surveys, article road map, and organization. An appropriate review methodology (Section 2) has been shadowed in writing the proposed survey which incorporates five subtopics: research questions, sources of information, search criteria, data extraction, and publication metrics. The inception of cross-modal retrieval, its general architecture, applications, observed challenges in the process, and the initial related articles are presented in concert under the background section (Section 3). Section 4 discusses about the diverse cross-modal representation and retrieval techniques which are broadly classified into real-valued and binary techniques. The literature related to these techniques has also been included in this section. The famous image–text benchmark datasets which have been widely used by the researchers in the cross-modal field have been presented in Section 5. Section 6 is of comparative analysis which introduces different performance evaluation metrics along with a comparison of various cross-modal retrieval methods. A summary of several state-of-the-art cross-modal retrieval works has been demonstrated with the use of tables in Section 7. The miscellaneous open issues in cross-modal retrieval domain have been discussed in Section 8. Finally, Section 9 culminates the survey with the conclusion. Fig. 3 depicts the road map of the article.