Large scale real-time water quality monitoring system usually produces vast amounts of high frequency data, and it is difficult for traditional water quality monitoring system to process such large and high frequency data generated by wireless sensor network. A real-time processing and early warning system framework is proposed to solve this problem, Apache Storm is used as the big data processing platform, and Kafka message queue is applied to classify the sample data into several data streams so as to reserve the time series data property of a sensor. In storm platform, Daubechies Wavelet is used to decompose the data series to obtain the trend of the series, then Long Short Term Memory Network (LSTM) model is used to model and predict the trend of the data. This paper provides a detailed description concerning the distribution mechanism of aggregated data in Storm, data storage format in HBase, the process of wavelet decomposition, model training and the application of mode for prediction. The application results in Xin’an River in Yantai City reveal that the prosed system framework has a very good ability to model big data with high prediction accuracy and robust processing capability.

中文翻译：

---

实时水质监测系统中大流量数据的分析与预测

大规模的实时水质监测系统通常会产生大量的高频数据，而传统的水质监测系统很难处理由无线传感器网络产生的如此大的高频数据。为了解决这个问题，提出了一个实时处理和预警系统框架，以Apache Storm作为大数据处理平台，并采用Kafka消息队列将样本数据分类为多个数据流，以保留时间序列。传感器的数据属性。在风暴平台中，使用Daubechies小波分解数据序列以获得序列趋势，然后使用长期短期记忆网络（LSTM）模型对数据趋势进行建模和预测。本文详细介绍了Storm中聚合数据的分布机制，HBase中的数据存储格式，小波分解过程，模型训练以及预测模式的应用。烟台市新安河的应用结果表明，该系统框架具有很好的建模能力，具有较高的预测精度和鲁棒的处理能力。