Using AI and Social Media Multimodal Content for Disaster Response and Management: Opportunities, Challenges, and Future Directions

Abstract

People increasingly use Social Media (SM) platforms such as Twitter and Facebook during disasters and emergencies to post situational updates including reports of injured or dead people, infrastructure damage, requests of urgent needs, and the like. Information on SM comes in many forms, such as textual messages, images, and videos. Several studies have shown the utility of SM information for disaster response and management, which encouraged humanitarian organizations to start incorporating SM data sources into their workflows. However, several challenges prevent these organizations from using SM data for response efforts. These challenges include near-real-time information processing, information overload, information extraction, summarization, and verification of both textual and visual content. We highlight various applications and opportunities of SM multimodal data, latest advancements, current challenges, and future directions for the crisis informatics and other related research fields.

Introduction

During disasters and emergencies, getting a quick understanding of the situation as it unfolds is a challenging task for formal response organizations. Members of the public, humanitarian organizations, and other concerned authorities search for pertinent information either to prevent the crisis or to help victims as early as possible. However, traditional approaches to gain situational awareness (i.e., understanding a bigger picture of the situation) are time-consuming and labor-intensive (Hiltz and Plotnick, 2013). Widespread adoption of non-traditional information sources such as Social Media (SM) platforms has created numerous opportunities to gather relevant information in a timely manner to improve disaster response. Research studies show that the general public uses SM platforms during disasters and reports critical information such as early warnings, cautions, damages to infrastructure such as roads, bridges, and buildings (Castillo, 2016; Imran, Castillo, Diaz, and Vieweg, 2015; Vieweg, Castillo, and Imran, 2014). Moreover, SM is considered as a potential source to perform urgent needs assessment of the affected population after a major disaster (Vieweg et al., 2014). Research has also shown that many real-world events, including crises, are first reported on social media compared to traditional media outlets such as TV and radio (Kalyanam, Quezada, Poblete, and Lanckriet, 2016).¹

Despite these benefits, processing SM data is not trivial. The various types of content available on SM platforms such as text messages, images, videos, are considered noisy and less formal compared to standard Web data, such as news articles or images published with those articles. Therefore, data analysis techniques trained and built for standard Web data often do not work well on SM data. For instance, text messages on Twitter or Facebook cannot exceed a fixed length and thus contain shortened words such as “2moro” instead of “tomorrow” in addition to other issues, including misspellings and use of slang. Studies revealed that techniques, which are specifically trained and built to process SM content, seem to work better than the ones trained for the processing of standard and well-structured data (Singh and Kumari, 2016). However, this highlights the need for in-domain labeled data, which is usually scarce.

Despite the lack of labeled data, the crisis informatics research community has continued to show advancements in different SM data processing tasks. Among others, these include data parsing, classification, extraction, summarizing, ranking, and recommendation. Several of these research lines have been struggling to perform until the recent breakthroughs in the field of Artificial Intelligence (AI), especially deep learning, have shown promising results. Deep learning techniques have recently been used by both Natural Language Processing (NLP) and Computer Vision (CV) communities to show substantial performance improvements over traditional machine learning approaches for several data processing tasks.

This special issue provided an opportunity to researchers working on crisis informatics topics using NLP, CV, ML, and AI methods to publish their latest and novel research works. We focused on the use of AI and SM multimodal content for disaster response topics. In total, we received 13 papers, which were rigorously peer-reviewed. Based on the expert reviews, four articles were finally accepted and summarized in Section 5. In the next section, we highlight the applications of SM textual content and techniques aiming at different processing tasks. Section 3 highlights applications of SM imagery content and how the CV community is addressing various challenges for crisis response, followed by a discussion on multimodal learning as a potential future research direction in Section 4. We conclude the editorial in Section 6.