The amount of data generated by millions of connected IoT sensors and devices is growing exponentially. The need to extract relevant information from this data in modern and future generation computing system, necessitates efficient data handling and processing platforms that can migrate such big data from one location to other locations seamlessly and securely, and can provide a way to preprocess and analyze that data before migrating to the final destination. Various data pipeline architectures have been proposed allowing the data administrator/user to handle the data migration operation efficiently. However, the modern data pipeline architectures do not offer built-in functionalities for ensuring data veracity, which includes data accuracy, trustworthiness and security. Furthermore, allowing the intermediate data to be processed, especially in the serverless computing environment, is becoming a cumbersome task. In order to fill this research gap, this paper introduces an efficient and novel data pipeline architecture, named as CCoDaMiC (Coherent Coordination of Data Migration and Computation), which brings both the data migration operation and its computation together into one place. This also ensures that the data delivered to the next destination/pipeline block is accurate and secure. The proposed framework is implemented in private OpenStack environment and Apache Nifi.

数百万个已连接的物联网传感器和设备生成的数据量呈指数增长。在现代和下一代计算系统中需要从此数据中提取相关信息，因此需要高效的数据处理和处理平台，该平台可以将这样的大数据从一个位置无缝安全地迁移到其他位置，并可以提供一种预处理和分析方法数据，然后再迁移到最终目标。已经提出了各种数据管道架构，其允许数据管理员/用户有效地处理数据迁移操作。但是，现代数据管道体系结构不提供用于确保数据准确性的内置功能，其中包括数据准确性，可信赖性和安全性。此外，允许处理中间数据，特别是在无服务器计算环境中，这已成为一项繁重的任务。为了填补这一研究空白，本文介绍了一种高效且新颖的数据管道架构，称为CCoDaMiC（数据迁移和计算的一致性协调），它将数据迁移操作及其计算都整合到了一起。这也确保了传递到下一个目标/管道块的数据是准确和安全的。该框架在私有OpenStack环境和Apache Nifi中实现。这将数据迁移操作及其计算整合到了一起。这也确保了传递到下一个目标/管道块的数据是准确和安全的。该框架在私有OpenStack环境和Apache Nifi中实现。这将数据迁移操作及其计算整合到了一起。这也确保了传递到下一个目标/管道块的数据是准确和安全的。该框架在私有OpenStack环境和Apache Nifi中实现。