During disaster response and recovery stages, stakeholders including governmental agencies collect disaster's impact information to inform disaster relief, resource allocation, and infrastructure reconstruction. The damage data collected using field surveys and satellite imagery are often not available immediately after a disaster while rapid information is crucial for time-sensitive decision makings. Some researchers turned to social media for real-time situational information of disaster damage. However, existing damage assessment research mostly focused on single data modality (i.e. text or image) and made coarse-grained predictions, which limited their practical applications in assisting city-level operations. The difficulties of retrieving useful information from vast noisy social media data have been outlined by many studies. Thus, we propose a data-driven method to locate and assess disaster damage with massive multimodal social media data. The method splits and processes two data modalities, i.e. texts and images, using two modules. The image analysis module uses five machine learning classifiers that are organized in a hierarchical structure. The text analysis module uses a keyword search-based method. They together mine various damage information including hazard types (e.g. wind and flood), hazard severities, damage types (e.g. infrastructure destruction and housing damage). The method is applied and evaluated with two recent hurricane events. In practice, the method acquires damage information throughout extreme events and supplements conventional damage assessment methods. It enables the rapid damage information access and disaster response for both first responders and the general public. The research effort contributes to achieving more transparent and effective disaster relief activities.

在灾难响应和恢复阶段，包括政府机构在内的利益相关者会收集灾难影响信息，以为救灾，资源分配和基础设施重建提供信息。在灾难发生后，通常无法立即获得使用现场调查和卫星图像收集的破坏数据，而快速信息对于时间敏感的决策至关重要。一些研究人员转向社交媒体获取灾害破坏的实时情况信息。但是，现有的损害评估研究主要集中在单一数据模式（即文本或图像）上，并做出了粗粒度的预测，这限制了它们在协助城市级运营中的实际应用。许多研究都概述了从大量嘈杂的社交媒体数据中检索有用信息的困难。从而，我们提出了一种数据驱动的方法，可以使用大量的多模式社交媒体数据来定位和评估灾难破坏。该方法使用两个模块拆分和处理两个数据模式，即文本和图像。图像分析模块使用五个以分层结构组织的机器学习分类器。文本分析模块使用基于关键字搜索的方法。他们一起挖掘各种破坏信息，包括危害类型（例如风和洪水），危害严重性，破坏类型（例如基础设施破坏和房屋破坏）。应用该方法并评估了最近的两次飓风事件。在实践中，该方法可在整个极端事件中获取损坏信息，并补充常规的损坏评估方法。它使急救人员和公众都可以快速访问损坏信息并进行灾难响应。研究工作有助于实现更加透明和有效的救灾活动。