Distributed stream processing engines are designed with a focus on scalability to process big data volumes in a continuous manner. We present the Theodolite method for benchmarking the scalability of distributed stream processing engines. Core of this method is the definition of use cases that microservices implementing stream processing have to fulfill. For each use case, our method identifies relevant workload dimensions that might affect the scalability of a use case. We propose to design one benchmark per use case and relevant workload dimension.

We present a general benchmarking framework, which can be applied to execute the individual benchmarks for a given use case and workload dimension. Our framework executes an implementation of the use case's dataflow architecture for different workloads of the given dimension and various numbers of processing instances. This way, it identifies how resources demand evolves with increasing workloads.

Within the scope of this paper, we present 4 identified use cases, derived from processing Industrial Internet of Things data, and 7 corresponding workload dimensions. We provide implementations of 4 benchmarks with Kafka Streams and Apache Flink as well as an implementation of our benchmarking framework to execute scalability benchmarks in cloud environments. We use both for evaluating the Theodolite method and for benchmarking Kafka Streams' and Flink's scalability for different deployment options.

分布式流处理引擎的设计重点是可伸缩性，以连续方式处理大数据量。我们介绍了经纬仪方法，用于对分布式流处理引擎的可伸缩性进行基准测试。该方法的核心是定义实现流处理的微服务必须满足的用例。对于每个用例，我们的方法确定可能影响用例可伸缩性的相关工作量维度。我们建议为每个用例和相关的工作量维度设计一个基准。

我们提供了一个通用的基准测试框架，该框架可用于执行给定用例和工作负载维度的各个基准测试。我们的框架针对给定维度和不同数量的处理实例的不同工作负载执行用例数据流体系结构的实现。这样，它可以确定资源需求如何随着工作量的增加而变化。

在本文的范围内，我们提出了4个已确定的用例，这些用例来自于处理工业物联网数据，以及7个相应的工作量维度。我们提供了Kafka Streams和Apache Flink的4个基准测试的实现，以及基准测试框架的实现，以在云环境中执行可伸缩性基准测试。我们既可以评估经纬仪方法，也可以为不同的部署选项评估Kafka Streams和Flink的可扩展性。