We propose a benchmarking test suite to evaluate performance of cloud serverless platforms and an open source software tool to automate the test process. Additionally, we used this setup to compare the commercial offers of Amazon, Google, Microsoft, and IBM. The work builds on ideas and experiments reported in the literature that, nevertheless, did not offer a “standard” set of coherent and comprehensive tests, capable of analyzing diverse serverless platforms, across the same set of features. To that end, we defined seven tests that cover scalability (latency and throughput), the effect of allocated memory, the performance for CPU-bound cases, the impact of payload size, the influence of the programming language, resource management (e.g., reuse of containers), and overall platform overhead. We created a software tool to deploy the test code and collect metrics in a manner agnostic to the serverless platforms under study. At a time when serverless computing popularity is rising, this benchmarking suite and software test tool enable developers to take informed decisions on the suitability and performance of each provider’s serverless offer. Additionally, they also help to identify attributes in need for improvement in the existing platforms.

我们建议使用基准测试套件来评估无服务器云平台的性能，并提供一种开源软件工具来自动化测试过程。此外，我们使用此设置来比较Amazon，Google，Microsoft和IBM的商业报价。该工作建立在文献报道的思想和实验的基础上，尽管如此，这些思想和实验并未提供“标准”的连贯而全面的测试，无法对同一组功能分析各种无服务器平台。为此，我们定义了七个测试，这些测试涉及可伸缩性（延迟和吞吐量），分配的内存的影响，CPU约束情况下的性能，有效负载大小的影响，编程语言的影响，资源管理（例如重用）容器），以及整个平台的开销。我们创建了一个软件工具，用于部署测试代码并以与所研究的无服务器平台无关的方式收集指标。在无服务器计算日益普及的时代，该基准测试套件和软件测试工具使开发人员能够针对每个提供商的无服务器产品的适用性和性能做出明智的决策。此外，它们还有助于确定现有平台中需要改进的属性。