当前位置: X-MOL 学术Softw. Pract. Exp. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Plumb: Efficient stream processing of multi‐user pipelines
Software: Practice and Experience ( IF 3.5 ) Pub Date : 2020-10-11 , DOI: 10.1002/spe.2909
Abdul Qadeer 1 , John Heidemann 1
Affiliation  

Operational services run 24×7 and require analytics pipelines to evaluate performance. In mature services such as domain name system (DNS), these pipelines often grow to many stages developed by multiple, loosely coupled teams. Such pipelines pose two problems: first, computation and data storage may be duplicated across components developed by different groups, wasting resources. Second, processing can be skewed, with structural skew occurring when different pipeline stages need different amounts of resources, and computational skew occurring when a block of input data requires increased resources. Duplication and structural skew both decrease efficiency, increasing cost, latency, or both. Computational skew can cause pipeline failure or deadlock when resource consumption balloons; we have seen cases where pessimal traffic increases CPU requirements 6‐fold. Detecting duplication is challenging when components from multiple teams evolve independently and require fault isolation. Skew management is hard due to dynamic workloads coupled with the conflicting goals of both minimizing latency and maximizing utilization. We propose Plumb, a framework to abstract stream processing as large‐block streaming (LBS) for a multi‐stage, multi‐user workflow. Plumb users express analytics as a DAG of processing modules, allowing Plumb to integrate and optimize workflows from multiple users. Many real‐world applications map to the LBS abstraction. Plumb detects and eliminates duplicate computation and storage, and it detects and addresses both structural and computational skew by tracking computation across the pipeline. We exercise Plumb using the analytics pipeline for B‐Root DNS. We compare Plumb to a hand‐tuned system, cutting latency to one‐third the original, and requiring 39% fewer container hours, while supporting more flexible, multi‐user analytics and providing greater robustness to DDoS‐driven demands.

中文翻译:

Plumb:多用户管道的高效流处理

运营服务 24×7 全天候运行,需要分析管道来评估性能。在域名系统 (DNS) 等成熟服务中,这些管道通常会发展到由多个松散耦合团队开发的多个阶段。这种管道带来两个问题:首先,计算和数据存储可能会在不同团队开发的组件之间重复,浪费资源。其次,处理可能会出现偏差,当不同的流水线阶段需要不同数量的资源时会发生结构性偏差,而当输入数据块需要更多资源时会发生计算偏差。重复和结构偏差都会降低效率、增加成本、延迟,或两者兼而有之。当资源消耗膨胀时,计算偏差会导致管道故障或死锁;我们已经看到悲观流量将 CPU 需求增加 6 倍的情况。当来自多个团队的组件独立发展并需要故障隔离时,检测重复是具有挑战性的。由于动态工作负载以及最小化延迟和最大化利用率的相互冲突的目标,倾斜管理很困难。我们提出了 Plumb,一个将流处理抽象为大块流 (LBS) 的框架,用于多阶段、多用户工作流。Plumb 用户将分析表示为处理模块的 DAG,允许 Plumb 集成和优化来自多个用户的工作流。许多现实世界的应用程序映射到 LBS 抽象。Plumb 检测并消除重复的计算和存储,它通过跟踪整个管道的计算来检测和解决结构和计算偏差。我们使用 B-Root DNS 的分析管道来练习 Plumb。我们将 Plumb 与手动调整的系统进行比较,将延迟减少到原来的三分之一,容器时间减少 39%,同时支持更灵活的多用户分析,并为 DDoS 驱动的需求提供更大的稳健性。
更新日期:2020-10-11
down
wechat
bug