T-GAN: A deep learning framework for prediction of temporal complex networks with adaptive graph convolution and attention mechanism

doi:10.1016/j.displa.2021.102023

Displays

Volume 68, July 2021, 102023

https://doi.org/10.1016/j.displa.2021.102023 Get rights and content

Highlights

•
We proposed a novel model T-GAN for prediction of properties and events of complex networks.
•
We proposed a new adaptive graph convolution network integrated with LSTM cells.
•
T-GAN has prominent ability in extracting features of temporal complex network.

Abstract

Complex network is graph network with non-trivial topological features often occurring in real systems, such as video monitoring networks, social networks and sensor networks. While there is growing research study on complex networks, the main focus has been on the analysis and modeling of large networks with static topology. Predicting and control of temporal complex networks with evolving patterns are urgently needed but have been rarely studied. In view of the research gaps we are motivated to propose a novel end-to-end deep learning based network model, which is called temporal graph convolution and attention (T-GAN) for prediction of temporal complex networks. To joint extract both spatial and temporal features of complex networks, we design new adaptive graph convolution and integrate it with Long Short-Term Memory (LSTM) cells. An encoder-decoder framework is applied to achieve the objectives of predicting properties and trends of complex networks. And we proposed a dual attention block to improve the sensitivity of the model to different time slices. Our proposed T-GAN architecture is general and scalable, which can be used for a wide range of real applications. We demonstrate the applications of T-GAN to three prediction tasks for evolving complex networks, namely, node classification, feature forecasting and topology prediction over 6 open datasets. Our T-GAN based approach significantly outperforms the existing models, achieving improvement of more than 4.7% in recall and 25.1% in precision. Additional experiments are also conducted to show the generalization of the proposed model on learning the characteristic of time-series images. Extensive experiments demonstrate the effectiveness of T-GAN in learning spatial and temporal feature and predicting properties for complex networks.

Introduction

Complex network is a graph network with non-trivial topological features, which often occurs in real systems, such as Internet, social networks, sensor networks, transportation networks and utility networks. Study of complex networks is still a relatively new but active research area. In the early research on complex networks, the main focus is on the analysis and modeling the features of complex networks. More specifically, the small world [1] and scale-free [2] characteristics of complex networks have been deeply explored and proved theoretically. In addition, the research problems of community detection [3] and link prediction [4] are widely investigated for complex networks. With more extreme weathers and advanced large scale cyber-security attacks, the vulnerability of real complex networks has never been exposed as now and the cost of complex network breaking down could be very high. There is increasing demand on the prediction of complex network states and events, which can provide important support for intelligent decision making on the control and management of complex networks. However, prediction of complex networks is very challenging due to the large scale and complex interaction among the network nodes. The research problem of prediction of complex networks has been rarely studied in the literature.

Many real applications on complex network can be modeled as a dual problem of network topology prediction or network node feature forecasting. For example, as shown in Fig. 1, e-commerce recommendation can be modeled as a topology prediction problem in a user-commodity bipartite graph [5], in which the purchasing behaviors of consumers will be recorded at different time step ( $t_{1}$ and $t_{2}$ ) that constitute the time sequence of the bipartite graph, and taking advantage of the historical information from the time sequence to make prediction for the future orders (connections between consumers and items at time step $t_{3}$ ) is the core objective in recommendation systems. Malicious node detection in wireless sensor networks (WSN) [6] can be applied to the network lifetime assessment for sensor networks [7], and risk forecasting for finance companies [8] can be considered as the node classification task in complex networks, as is shown in Fig. 2. In Fig. 2, there was only one malicious node in the network at time step $t_{1}$ , with another 2 nodes infected soon at $t_{2}$ . Hence, there is an urgent need to use the time series to study the evolution pattern of the infection; making prediction for the next time step $t_{3}$ is of vital importance.

Topology evolution in social networks can be formulated as the link prediction problem for undirected or directed weighted graphs, as shown in Fig. 3, where the original snapshot of the dynamic network is presented as time step $t_{1}$ , the blue dotted line in the graph at $t_{2}$ indicates that the link between node 1 and node 2 is broken, and the solid black line denotes that a new link presents between node 2 and node 5. The topology prediction task for this temporal network shown in Fig. 3 includes predictions for weights or links of each node pair of the graph at time step $t_{3}$

For the real complex networks, along with complex network topology, another prominent feature of temporal evolution makes the network prediction and control problems more challenging. The network topology and the properties of the networks nodes and links could all change significantly with time. In most of the existing research works on complex network, static complex networks have been studied, while temporally evolving complex networks received much less study.

In view of the above research gaps and challenges on the study of prediction of complex networks properties and events, we are motivated to propose a novel end-to-end deep learning based network model, which is called temporal graph convolution and attention (T-GAN) for predicting property and events of temporal complex networks. An encoder-decoder framework is applied to achieve the objectives. As long short-term memory (LSTM) has demonstrated the ability in extracting temporal features and has achieved huge successes in deep learning based computer vision and natural language processing, we apply it to our proposed model for joint extraction of both spatial and temporal features of complex networks. As GCN adopts symmetric Laplacian matrix as the graph shift operator in frequency domain, it has been proved that GCN has the property of low-pass filtering [9], which directly affects the depth of GCN based deep learning model [10]. Therefore, we propose Adaptive Graph Convolution Network (AGCN) and integrated it with LSTM cells, which provides T-GAN stronger ability in extracting features of complex networks. Furthermore, a dual attention mechanism is introduced to improve the sensitivity of the model to each time step of temporal networks. After generating the embedded representation vectors for the nodes in dynamic complex networks, a neural network with single layer serves as the decoder, in which nonlinear affine transformation is used as the mapping function. Compared with the fully connected neural network, the number of the parameters of the decoder designed in T-GAN can be greatly reduced due to parameter sharing, which plays a significant role in reducing training time and avoiding over-fitting.

The main contributions of this paper are summarized as follows:

•
We proposed a novel deep learning based network model (T-GAN) with an encoder-decoder framework for the purpose of prediction of properties and events of complex networks. The model can be used for the control and management of complex networks, with many real applications.
•
For the encoder of the model T-GAN, we propose new adaptive graph convolution network and integrate it with LSTM cells, which provides T-GAN strong ability in extracting features of complex network and greatly improves the scalability of the model. And we applied attention blocks to help capture the importance and uniqueness of different time steps in complex networks and enhance the sensitivity of the model.
•
Our proposed T-GAN architecture is general and scalable, which can be used for a wide range of real applications. We demonstrate the applications of T-GAN to three prediction tasks for evolving complex networks, namely, node classification, feature forecasting and topology prediction over 6 open datasets. Our T-GAN based approach significantly outperforms the existing, achieving improvement of more than 4.7% in recall and 25.1% in precision.
•
We also conducted additional experiments to evaluate the generalization of the proposed model on learning the characteristic of time-series images. Extensive experiments demonstrate the effectiveness of T-GAN in learning spatial and temporal feature and predicting properties for complex networks.

The remainder of the paper is organized as follows. Section 2 presents the related works. Section 3 presents the problem definition and notations. Section 4 introduces the proposed architecture in detail. Extensive experimental results and analysis are shown in Section 5, and additional experiments are presented in Section 6. Finally, conclusions are drawn in Section 7.

Section snippets

Literature review

Complex network is a relatively new research area. The main focus of complex network in the early stage is on the analysis and modeling of complex networks. Graph embedding (graph representation learning) is an important research branch of deep learning applied in complex networks [11], which takes advantage of the powerful non-linear fitting capability of the deep neural network that maps high-dimensional sparse networks to dense vectors in a lower-dimensional space. It is convenient to

Problem definition and notations

Temporal complex network can be denoted by a time sequence $G_{T} =$ { $G_{1}, G_{2} \dots, G_{t}$ }, where $G_{t}$ is a sampling snapshot at time step t which is a triple comprised of vertices (nodes), edge set and node features, i.e., $G_{t} = (V, E_{t}, X_{t})$ . $V =$ { $v_{1}, v_{2}, \dots$ } denotes the set of vertices in the network. $E_{t}$ denotes the set of edges connecting the network nodes at time step t. $X_{t} \in R^{|V| \times F_{t}}$ is the node feature matrix at the sampling time t, where $|V|$ is the number of the nodes in the network and $F_{t}$ is the dimension of node features.

System framework of T-GAN

In order to capture the evolution pattern of topology and node features of temporal networks, we propose a deep learning network model T-GAN with adaptive graph convolution and attention mechanism. T-GAN is a deep neural network model consisting of an encoder and decoder, which integrates the topological structure of complex networks with the extensive feature information of vertices for learning and modeling the evolutionary property of temporal networks. Stacked-LSTM layers are used as the

Experiments and analysis

In order to verify the effectiveness of the proposed model, we conduct various experiments and analysis of the model for 3 applications over 6 open temporal complex network datasets. We also compare our approach with the current baseline methods. All experiments are run on the server equipped with two Nvida RTX 3060. The deep learning model is built by Keras 2.1.4. The version of TensorFlow on the server is TensorFlow-GPU 1.14.0, and the CUDA version is 10.1. The code of the T-GAN deep learning

Additional experiments beyond graph

In this section, we will visualize the beneficial effect that the attention module shows on the proposed model through additional experiments, and evaluate the feasibility of applying the revised T-GAN to the image reconstruction and prediction task.

Conclusion

Prediction of properties and events of complex networks is of increasing importance but has not been well studied. In this paper, we investigated the prediction of properties and events of complex networks with temporally changing features, and proposed a novel deep learning based model T-GAN for the challenging prediction task. In the T-GAN network model, adaptive graph convolution network was proposed and integrated with LSTM to extract the spatial characteristics of nodes and learn the

CRediT authorship contribution statement

Ru Huang: Conceptualization, Methodology. Lei Ma: Software, Data curation, Writing - original draft. Jianhua He: Visualization, Investigation. Xiaoli Chu: Supervision, Validation, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This work was supported by National Natural Science Foundation of China, grant number 61673178 and 61922063, Natural Science Foundation of Shanghai, No.20ZR1413800. Our work has also received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 824019 (project COSAFE) and the Marie Sklodowska-Curie grant agreement No 101022280 (project VESAFE). The authors would like to thank the anonymous reviewers for their

Ru Huang received his B.S. degree from Nanjing University, Nanjing, China, in 1999 and his Ph.D. degree in circuit and system from Shanghai Jiao Tong University, Shanghai, China, in 2008. He was a visiting scholar in the University of Wisconsin-Madison, WI, USA, from March 31, 2015 to March 30, 2016. He is currently an associate professor of electronics and communication engineering at East China University of Science and Technology, Shanghai, China. His current research interests include

References (56)

Y. Gandica et al.
Complex propagation on directed small world networks
Phys. A Stat. Mech. Its Appl.
(2010)
Z. Xu et al.
Interaction between epidemic spread and collective behavior in scale-free networks with community structure
J. Theor. Biol.
(2019)
H. Jiang et al.
Community detection in complex networks with an ambiguous structure using central node based link prediction
Knowledge-Based Syst.
(2020)
M. Gao et al.
Projection-based link prediction in a bipartite network
Inf. Sci. (Ny)
(2017)
Z. Zhu et al.
Multi-kernel Gaussian process latent variable regression model for high-dimensional sequential data modeling
Neurocomputing.
(2019)
J. Chen et al.
E-LSTM-D: A deep learning framework for dynamic network link prediction
IEEE Trans. Syst. Man Cybern. Syst.
(2019)
Z. Gao et al.
Extended geometric models for stereoscopic 3D with vertical screen disparity
Displays
(2020)
J. You et al.
Perceptual-based quality assessment for audio-visual services: A survey
Signal Process. Image Commun
(2010)
M. Li et al.
Evolutionary network embedding preserving both local proximity and community structure
IEEE Trans. Evol. Comput.
(2020)
A. Yessembayev et al.
Detection of good and bad sensor nodes in the presence of malicious attacks and its application to data aggregation
IEEE Trans. Signal Inf. Process. over Networks.
(2018)

I. Khalil et al.

Stealthy attacks in wireless ad hoc networks: detection and countermeasure

IEEE Trans. Mob. Comput.

(2011)

K. Pongaliur et al.

CENDA: Camouflage event based malicious node detection architecture

IEEE 6th Int Conf. Mob. Adhoc Sens. Syst.

(2009)

Q. Li, X.-M. Wu, H. Liu, X. Zhang, Z. Guan, Label efficient semi-supervised learning via graph filtering, in: Proc....

Q. Li, Z. Han, X.-M. Wu, Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning., in: Proc....

S. Pan et al.

Learning graph embedding with adversarial training methods

IEEE Trans. Cybern.

(2020)

B. Perozzi, R. Al-Rfou, S. Skiena, Deepwalk: Online learning of social representations, in: Proc. 20th ACM SIGKDD Int....

S. Ji et al.

Parallelizing Word2Vec in shared and distributed memory

IEEE Trans. Parallel Distrib. Syst.

(2019)

F. Fouss et al.

Random-walk computation of similarities between nodes of a graph with application to collaborative recommendation

IEEE Trans. Knowl. Data Eng.

(2007)

K.W. Church

Word2Vec

Nat. Lang. Eng.

(2017)

J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, Q. Mei, LINE: Large-scale Information Network Embedding. Proceedings of the...

A. Grover, J. Leskovec, node2vec: Scalable Feature Learning for Networks., in: Proc. 22nd ACM SIGKDD Int. Conf. Knowl....

J. Wu, Y. Sha, B. Jiang, J. Tan, DSINE: Deep Structural Influence Learning via Network Embedding., in: Thirty-Third...

X. Li, N. Du, H. Li, K. Li, J. Gao, A. Zhang, A Deep Learning Approach to Link Prediction in Dynamic Networks., in:...

Z. Ghahramani et al.

Variational learning for switching state-space models

Neural Comput.

(2000)

L. Zhou, Y. Yang, X. Ren, F. Wu, Y. Zhuang, Dynamic Network Embedding by Modeling Triadic Closure Process., in: Proc....

N.M.A. Ibrahim et al.

Link prediction in dynamic social networks by integrating different types of information

Appl. Intell.

(2015)

X. Li, Y. Li, Y. Shang, L. Tong, F. Fang, P. Yin, J. Cheng, J. Li, DDNE: Discriminative Distance Metric Learning for...

N. Kamra et al.

DynGEM: deep embedding method for dynamic graphs

IJCAI Int. Work. Represent. Learn. Graphs

(2017)

Cited by (14)

CFA-GAN: Cross fusion attention and frequency loss for image style transfer
2024, Displays
Image style transfer is a significant challenge in computer vision, where the goal is to transfer the style of a reference image to a source image while preserving its background and feature information. Despite previous progress in this area, many studies have overlooked the issues of local details and complexity, leading to unrealistic generated images and slow model inference speeds. This paper proposes an efficient image style transfer method based on cross-fusion attention (CFA), inspired by generative adversarial networks (GAN) and a new joint loss. Our novel CFA encodes local contextual information between low-level and high-level features, models remote dependencies of features, and embeds CFA into the encoder of our proposed model to increase the diversity of small features and focus on more prominent feature parts. Furthermore, we use frequency domain loss to improve image reconstruction quality and a joint loss to train the network. Our experiments on multiple datasets demonstrate that our proposed method offers significant advantages in terms of inference speed, background information preservation, and local details, making it a promising approach to image style transfer. The code for our method is available at https://github.com/berylxzhang/CFA-GAN.
Dual-graph hierarchical interaction network for referring image segmentation
2023, Displays
Referring Image Segmentation (RIS) aims to extract the object or stuff from an image according to the given natural language expression. As a representative multi-modal reasoning task, the main challenge of RIS lies in accurately understanding and aligning two types of heterogeneous data (i.e. image and text). Existing methods typically conduct this task via inexplicit cross-modal fusion toward the visual and linguistic features, which are separately extracted from different encoders and hard to capture accurate image–text alignment information due to their distinct latent representation structures. In this paper, we propose a Dual-Graph Hierarchical Interaction Network (DGHIN) to facilitate the explicit and comprehensive alignment between the image and text data. Firstly, two graphs are separately built for the initial visual and linguistic features extracted with different pre-trained encoders. By means of graph reasoning, we obtain a unified representation structure for different modalities to capture the intra-modal entities and their contexts, where each projected node incorporates the long-range dependencies into the latent representation. Then, the Hierarchical Interaction Module (HIM) is applied to the visual and linguistic graphs to extract comprehensive inter-modal correlations from the entity level and graph level, which not only capture the corresponding keywords and visual patches but also draws the whole sentence closer to the image region with the consistent context in the latent space. Extensive experiments on RefCOCO, RefCOCO+, G-Ref, and ReferIt demonstrate that the proposed DGHIN outperforms many state-of-the-art methods. Code is available at https://github.com/ZhaofengSHI/referring-DGHIN.
Exploring network reliability by predicting link status based on simplex neural network
2023, Displays
Complex networks are graph-based structures with non-trivial topological features that frequently occur in real systems. Link prediction plays an important role in various real-world networks application, such as recommendation systems, protein structure prediction, packet forwarding strategy optimization, etc. The existing link prediction approaches mainly focus on superficial heuristic features, while ignoring high-order structure information. In this paper, we propose a deep-learning based model, named Weisfeiler–Lehman Simplex Neural Network (WL-SNN), which can learn the high-order simplex information of the network. In particular, we design a third-order Laplace operator to extract the simplicial features and utilize the graph convolutional network to compensate for the possible deficiencies of the model resulting from the single-channel features. Furthermore, we use the Weisfeiler–Lehman algorithm to extract closed subgraphs of the target, which significantly enhances the adaptability of the model to large-scale networks. Experimental results on six real-world networks show that our approach achieves comparable performance in the link prediction task as well as in the stability analysis of the network.
LRB-Net: Improving VQA via division of labor strategy and multimodal classifiers
2022, Displays
Citation Excerpt :
The key idea is to communicate between image regions to build contextual representations. Ru Huang et al. [21] proposed a temporal graph convolution and attention network (T-GAN). They integrate the graph convolution network (GCN) with LSTM cells so that the complex network’s ability to extract features is enormously improved.
Visual question answering (VQA), along with multiple types of image and textual questions, makes it a challenging task to infer the correct answer. Consequently, traditional methods rely on relevant cross-modal objects and seldom leverage the cooperation of visual appearance and textual understanding. Here, we present LRBNet, a model-based approach to the problem by analyzing VQA as a division of labor strategy. Using a dictionary and pre-trained GloVe vectors, we embedded region captions and questions by the LSTM. Furthermore, we use graphs to model image region captions and features and then feed them into two GNN-based networks to capture the semantic and visual relations. Finally, we modulate the vertex features of multimodal graphs. Thus, the question embedding and vertex features are fed into the multi-level answer predictor to produce the results. We experimentally validate that LRBNet is an exciting framework for visual–textual understanding and a more challenging alternative to better VQA because it needs to understand the image to be successful. Our study provides complementary prediction through hierarchical representation within and beyond the interactive understanding of the textual sequence and the images, and the experimental results show that LRBNet outperforms other leading models in most cases.
Improving adversarial robustness of traffic sign image recognition networks
2022, Displays
The robustness of deep neural networks is an increasingly essential issue as they become more and more prevalent in several real-world applications like autonomous vehicles. If traffic signs turn to adversarial examples, an autonomous vehicle will probably be misled and cause fatal accidents. To improve adversarial robustness, a new cost function for training convolutional neural recognition networks is proposed in this paper. Recent works proved that by employing the classifier probabilities on the complement (incorrect) classes as well as the ground-truth class in Softmax Cross Entropy, the model achieves better performance on adversarial inputs. In this paper, we show that in addition to using the information from Softmax layer, the extracted features from convolutional layers also enhance the robustness. In our new cost function, Regularized Guided Complement Entropy (RGCE), by decreasing the output of convolutional layers’ activation functions alongside utilizing Softmax layer output in training phase, we reach better model performance on adversarial attacks. Our proposed algorithm is evaluated on CIFAR-10 and GTSRB datasets. The performances of different convolutional neural networks on clean and adversarial images are reported and compared with other methods.
TERQA: Question answering over knowledge graph considering precise dependencies of temporal information on vectors
2022, Displays
Citation Excerpt :
It is well known that the temporal constraints on natural language questions are closely linked to the relevant subcomponents of the questions themselves. Previous studies [7] usually spliced temporal information embeddings with other text embeddings in a simple way. In the case of complex questions, the dependency of temporal information on specific components is ignored, resulting in a wealth of uncertain associations, further constraining the accuracy of retrieving the answers.
Time questions involve explicit and implicit constraints as well as complex time interval interactions, making them critical criteria for measuring the effect of knowledge base question answering. Although attention to temporal questions has spurred the development of temporal knowledge graphs, existing studies have focused on the simple splicing and fusion of temporal information with question or knowledge base embeddings, losing sight of the hidden interaction features between temporal information and embedded vectors. In this paper, we proposed TERQA, a temporal knowledge base question-answering approach to explore precise spatial dependencies between temporal information and embedded vectors. The exploration of the deep dependency between time and embedded vectors was divided into two stages. In the first stage, the Transformer model of depth extraction was employed to extract richer features from questions and the representation was enhanced with temporal information; in the second stage, high-level capsules were adopted to extract the low-level vector features for detailed pose determination, allowing a more precise deep dependency of temporal facts on embedded vectors. We conducted an experiment using two temporal question answering datasets, TempQuestions and CronQuestions, and the results showed that accuracy for TERQA improved 11.3% from baseline on the dataset TempQuestions with higher annotated information. Additionally, the adapted TERQA also showed varying degrees of improvements over the baseline in the larger but simply annotated dataset CronQuestions.

View all citing articles on Scopus

Lei Ma received his B.S. degree in information engineering from the University of East China University of Science and Technology, in 2018.And he is now pursing his M.S. degree, majoring in Information and Communication Engineering at East China University of Science and Technology. And his Research directions comprises complex networks, deep learning, and data mining, specializing at building deep learning model for data mining tasks in complex networks. ([email protected])

Jianhua He received the Ph.D. degree from Nanyang Technological University, Singapore, in 2002. He is a Reader at University of Essex, U.K. His research interests include 5G and beyond technologies, ADAS, Internet of Things, machine learning and big data analytics. He has authored or co-authored over 100 technical papers in major international journals and conferences. Dr. He is Editor or Guest Editor of several international journals, including Wireless Communications and Mobile Computing, IEEE Access, International Journals of Communication Systems, International Journal of Distributed Sensor Networks, and KSII Transactions on Internet and Information Systems. He acted as TPC chair for international conference ICAIT’2010 and ICONI’2010. He is a TPC member of many international conferences including IEEE Globecom and IEEE ICC. ([email protected])

Xiaoli Chu received her B.Eng. degree in electronic and information engineering from Xi'an Jiao Tong University in 2001 and her Ph.D. degree in electrical and electronic engineering from Hong Kong University of Science and Technology in 2005. From September 2005 to April 2012, she was with the Centre for Telecommunications Research at King's College London. Her research interests include modelling, analysis, and algorithm design for improving the performance and efficiency of wireless communication systems. ([email protected])

^☆: This paper was recommended for publication by Prof G Guangtao Zhai.

View full text

T-GAN: A deep learning framework for prediction of temporal complex networks with adaptive graph convolution and attention mechanism☆