Damage assessment plays an important role in disaster management for the enhanced situational awareness, but the data used in the conventional approaches is insufficient for instant damage assessment as it is not timely updated. The social media can serve as an alternative data source to remedy the defects of conventional methods and provide rapid damage assessment. Most studies attempting to interpret social media texts in respect of damage assessment depend on keyword extraction or topic modelling, which may cause high reliance on a domain-specific and exhaustive dictionary, as well as neglection of the abundant semantic information. Therefore, we propose a model to focus on the in-depth understanding of social media texts while involving minimum manual efforts in a semi-supervised manner. This paper introduces a novel method to evaluate the damage extent indicated by social media texts. The damage reports are recognized among the social media messages as damage relation categories by a multi-instance multi-class classifier, and the damage extent implied by the words in a report is quantified by measuring the word similarity with the known damage-related words. The proposed method is validated on tweets collected from Hong Kong and Macau during the two typhoons occurred in 2017. The damage extent map generally agrees with the actual damage reported by authorities and therefore it is shown that the proposed method is able to assess the typhoon damage reliably with little manual work involved.