A transformer-based deep learning model for recognizing communication-oriented entities from patents of ICT in construction

Highlights

• Communication-oriented entities of ICT in construction were introduced.

• A deep learning model to recognize the entities from the patents was proposed.

• The model was structured based on the Transformer rather than RNN.

• The Transformer enables the model to efficiently address the contextual information.

• The model provides an effective way to perceive the communication functionalities.

Abstract

The patents of information and communication technology (ICT) in construction are valuable sources of technological solutions to communication problems in the construction practice. However, it is often difficult for practitioners and stakeholders to identify the key communication functionalities from complicated expressions in the patent documents. Addressing such challenges, this study develops a deep learning model to enable automatic recognition of communication-oriented entities (CEs) from patent documents. The proposed model is structured based on the Transformer, consisting of feed-forward and self-attention neural networks to better recognize ambiguous and unknown entities by utilizing contextual information. The validation results showed that the proposed model has superior performance in CE recognition than traditional recurrent neural networks (RNN)-based models, especially in recognizing ambiguous and unknown entities. Moreover, experimental results on some research literature and a real-life project report showed satisfactory performance of the model in CE recognition across different document types.

Introduction

Information and communication technology (ICT) is an extensional concept, incorporating a wide range of technical approaches that mainly concentrate on communication functionalities [1]. The core benefit of ICT application in the construction industry is to enable and enhance communication, improving the coordination of data in the whole life cycle of construction projects [2,3]. Successful adoption of ICT relies on appropriate choices of technologies to enable desired communication functionalities according to specific objectives in construction practice [4,5]. In order to choose the right technologies for the confronting problems, practitioners and stakeholders need to fully comprehend the communication functionalities embedded in ICTs [2]. Patents are a common source for up-to-date technologies, from which 95% inventions can be found. The information of communication functionalities of ICT was archived as raw texts in patent documents [6,7]. Analyzing patent documents effectively is important to acquire technological knowledge, link potential solutions to problems and inspire innovation in the industry [8]. Therefore, exploiting information underlying patent documents has gained increasing interests by researchers, patent analysts, and practitioners [9].

In patent documents of ICT in construction, the hints of communication functionalities are hidden in complicated expressions like how construction data was transmitted through virtual or physical models and how it was coordinated among sites, users or stakeholders [5]. Examples of such expressions include “installation information was transferred from a radio frequency identification (RFID) tag to a construction item” in an RFID patent [10], and “the technology conveys geographic data to display devices that users could manipulate” in a geographic information system (GIS) patent [11]. To make this embedded information more accessible, this study seeks to develop a computer-aided system to automatically identify the communication-oriented entities (CEs) and categorize them into pre-defined types. The task is named as entity recognition in natural language processing (NLP) [12].

Although some patent analysis tools (e.g., TRIZ¹) have been developed to process patent documents, these approaches aim for general purposes and are limited in specific problem solving [13]. Entity recognition offers a way to analyze patents based on customized problems or interests. An entity is a category of phrases that have similar properties, including rigid designators or members of a semantic class [14]. Mostly, the entities are “names” (e.g., drug names, disease names, chemical names) [14]. They usually have highly distinguishable spellings (e.g., chemistry entity “Deuterium” can be easily recognized due to its unique combination of characters and the capitalized initial letter [15]). However, recognizing CEs from the patent documents of ICT in construction is a more complicated task. There are two main technical challenges. One is the ambiguity of CEs. An entity is ambiguous if its spellings appear as an entity at one position, and appear as a different entity type at another [16]. Communication functions in the patents are expressed by not only mixtures of unique technical terms that appear with distinguishable spellings, but also words that are typically normal terms [17]. Thus, for recognizing ambiguous entities, it is important to incorporate the contextual information surrounding the candidate entities to discern their relevancy. Another challenge is the unknown of entities (entities that appear in testing set but not in training set). The previous studies attempted to address these problems by using additional linguistic materials, such as lexicons, dictionaries, gazetteers, ontologies, knowledge graphs [[18], [19], [20], [21]]. However, due to the unavoidable limitations in the coverage of lexical databases, these problems remain critical [22].

This study resorts to deep learning techniques to utilize the contextual information for recognizing the ambiguous and unknown CEs from the patents of ICT in construction. Rather than focusing on word-level information, a deep learning method can enhance the understanding of entities by incorporating surrounding texts. As recognized deep learning approaches, the recurrent neural networks (RNN)-based models, such as long short-term memory (LSTM) and gated recurrent unit (GRU), have been widely adopted in many NLP tasks, including entity recognition, text classification, sentiment analysis, and machine translation [23,24]. In these models, bi-directional structures and convolutional neural networks (CNN) were adopted to achieve improved performance [25]. However, despite the elaborate architectures, the RNN-based models have limitations in addressing long-term dependencies. A deep learning model of the Transformer-based neural networks (TBNN) was adopted instead in this study to remedy this deficiency. Proposed in 2017 by Google AI team [26], the Transformer can enable the so-called “self-attention” mechanism that computes the contextual representations in parallel rather than in sequence [27], enabling a more effective approach to memorize both long and short term dependencies compared with the RNN-based models. Previous methods used for recognizing communication functionalities from ICT patents were mostly manual searching, which are labor-intensive and time-consuming [28,29]. The TBNN model developed in this study provides an efficient alternative. Also, It has its merits in utilizing contextual information, which is an important advancement for computer-aided systems to achieve intelligence in NLP tasks [16].

The research procedure is shown in Fig. 1.. First, based on the literature review, the main technical challenges were identified and the classes of CEs for recognition were illustrated. Second, the architecture of the proposed TBNN was illustrated in detail. Third, the validation of the model was conducted using the training and testing instances. Finally, the results and findings were discussed to report the performance of the proposed model compared with the baseline model.