A novel temporal and spatial panorama stream processing engine on IoT applications

Abstract

Nowadays, more and more streaming data are generated with the development of Internet of Things. Although, streaming data show great application values in practical scenarios, raw streams from terminal sensors are quite massive, heterogeneous and complex, and those features make it difficult for applications to deal with them. In order to simplify streaming data processing for applications, a novel temporal and spatial panorama stream processing engine is proposed. This stream processing engine forms an effective link between bottom sensors and upper stream-based applications, and provides configurable, flexible, available and usable stream services for various upper applications. With support of Internet plus and domain data, raw streaming data are formatted thorough data fusion and are represented in temporal, spatial, logic and storage views. Fusion data are encapsulated with multiple strategies and methods of clustering models on every dimension. According to configurable strategies, encapsulated data are extracted, partitioned and distributed as the form of dynamical variable gratitude data blocks into stream channels. Our engine is applied to a practical application scenario. Case study of dynamic power distribution application based on people crowd streaming data proves the customized service capabilities of our engine. And the outstanding performance of the engine is shown further by experiments evaluation in this case.

Introduction

In recent years, with the rapid development of Internet of Things (IoT), IoT technologies are applied in more and more industrial and daily life scenarios, which leads to the boost of streaming data. Streaming data show great significance because of their real-time feature, and now become the basis of those applications like real-time analysis, continuous computation, and online machine learning.

However, streaming data also bring many challenges. In practical scenarios, IoT streaming data tend to be massive, heterogeneous and complex. Along with the expansion of smart cities, great quantity of sensing equipment is applied. Meanwhile, the quality and frequency of data collection are also improved. Billions of IoT devices have been put into use nowadays, and they have continuously generated zettabytes of data up to now. Such streaming data are not only massive but also heterogeneous. There are various kinds of sensors, which are different in operating principle, physical scenarios and transport protocols, generating streaming data diverse in dimensions, structures and formats. Besides, streaming data are quite complex as well. Different from traditional data formats, streaming data consist of both the data themselves and their contexts. Only when temporal, spatial and logical views of contexts are considered, the panoramic information of streaming data can be shown clearly.

Because of the features of streaming data, stream-based applications are complex and difficult to develop. IoT streaming data processing becomes an important and costly process for application development. It is impossible for all applications to deal with this process by themselves. Developers need to focus more on the business logic of an application. Therefore, it is worthwhile to realize an independent engine for IoT streaming data processing. The engine gathers origin data from IoT sensors and provides formatted stream service to upper applications. With the use of this engine, the stream-based applications can be much lighter in weight and more convenient to develop and maintain.

In this paper, a novel temporal and spatial panorama stream processing engine on IoT applications is proposed by us. The engine consists of a panorama data model and interfaces to IoT sensors, the Internet and upper applications. Kernel of our engine is this temporal and spatial panorama data model. In the model, streaming data are gathered, formatted, classified, extracted, partitioned and distributed.

Based on the fusion of streaming data generated from IoT scenarios and supporting data from the Internet and domain models, the information representation basis is constructed. Data are represented in unified format by dimensions from multiple categories. Fused data are encapsulated through our proposed parallel data classification framework. Multiple strategies and methods of data clustering models can operate in parallel mode. Though configurable extraction, partition and distribution, the encapsulated data are sent to applications in forms of variable granularity data blocks through stream channels. Without redundant and complex development work, flexible and customized stream services can be configured and realized easily using our engine.

Our proposed engine is also applied to a practical intelligent energy network scenario. In this application case, our engine is used to provide data service for dynamic power distribution application. The complex stream service needed by this application is configured and registered in convenience using our engine. The outstanding performance of our engine is also proved through several experiments and quantitative analysis.

The rest parts of this paper will be structured as follows. Related works of or study are introduced in Section 2. And in Section 3, the framework of our proposed temporal and spatial panorama stream processing engine is illustrated briefly. In Section 4, the methodologies and design details of the panorama data model are explained further. Section 5 presents the practical case of our engine. And the performance of the engine is evaluated by several experiments in Section 6. Finally, our contributions and future works are concluded in Section 7.