Skip to main content
SpringerLink
Log in

Benchmarking Serverless Computing Platforms

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

We propose a benchmarking test suite to evaluate performance of cloud serverless platforms and an open source software tool to automate the test process. Additionally, we used this setup to compare the commercial offers of Amazon, Google, Microsoft, and IBM. The work builds on ideas and experiments reported in the literature that, nevertheless, did not offer a “standard” set of coherent and comprehensive tests, capable of analyzing diverse serverless platforms, across the same set of features. To that end, we defined seven tests that cover scalability (latency and throughput), the effect of allocated memory, the performance for CPU-bound cases, the impact of payload size, the influence of the programming language, resource management (e.g., reuse of containers), and overall platform overhead. We created a software tool to deploy the test code and collect metrics in a manner agnostic to the serverless platforms under study. At a time when serverless computing popularity is rising, this benchmarking suite and software test tool enable developers to take informed decisions on the suitability and performance of each provider’s serverless offer. Additionally, they also help to identify attributes in need for improvement in the existing platforms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. AWS Lambda – Case Studies. Available online https://aws.amazon.com/pt/lambda/resources/customer-case-studies/ Retrieved on May 13th 2020

  2. Azure, M.: FUJIFILM boosts NPB Player recognition, reduces photo tagging costs with Microsoft Azure. Available online https://customers.microsoft.com/en-us/story/fujifilm-manufacturing-azure-ai-functions-japanhttps://customers.microsoft.com/en-us/story/fujifilm-manufacturing-azure-ai-functions-japan, 2018 Retrieved on May 13th 2020

  3. Butt, A.R., Johnson, T.A., Zheng, Y., Hu, Y.C.: Kosha: A Peer-to-Peer Enhancement for the Network File System. J. Grid Comput. 4(3), 323–341 (2006)

    Article  Google Scholar 

  4. Eivy, A., Weinman, J.: Be wary of the economics of “serverless” cloud computing. IEEE Cloud Comput. 4(2), 6–12 (2017)

    Article  Google Scholar 

  5. Leitner, P., Wittern, E., Spillner, J., Hummer, W.: A mixed-method empirical study of function-as-a-service software development in industrial practice. J. Syst. Softw. 149, 340–359 (2019)

    Article  Google Scholar 

  6. Amazon Lambda – AWS. Available online https://aws.amazon.com/lambda/ Retrieved on May 13th 2020

  7. Azure, M.: Functions, Microsoft Azure. Available online https://azure.microsoft.com/services/functions/,2018 Retrieved on May 13th (2020)

  8. Cloud, G.: Google Cloud Functions. Available online https://cloud.google.com/functions/, 2018 Retrieved on May 13th 2020

  9. IBM: IBM Cloud Functions. Available online https://cloud.ibm.com/openwhisk/, 2018 Retrieved on May 13th 2020

  10. Serverless Benchmark. Available online https://github.com/hjmart93/serverlessbenchmark, 2019 Retrieved on May 13th 2020

  11. Baldini, I., Castro, P., Chang, K., Cheng, P., Fink, S., Ishakian, V., Mitchell, N., Muthusamy, V., Rabbah, R., Slominski, A., et al.: Serverless Computing: Current Trends and Open Problems. In: Research Advances in Cloud Computing, pp. 1–20. Springer (2017)

  12. Serverless Architectures with AWS Lambda – Overview and Best Practices. Available online https://d1.awsstatic.com/whitepapers/serverless-architectures-with-aws-lambda.pdfhttps://d1.awsstatic.com/whitepapers/serverless-architectures-with-aws-lambda.pdf, 2017 Retrieved on May 13th 2020

  13. Cloud Functions serverless platform is generally available – Google Cloud Blog. Available online https://cloud.google.com/blog/products/gcp/cloud-functions-serverless-platform-is-generally-availablehttps://cloud.google.com/blog/products/gcp/cloud-functions-serverless-platform-is-generally-availablehttps://cloud.google.com/blog/products/gcp/cloud-functions-serverless-platform-is-generally-available, 2019 Retrieved on May 13th 2020

  14. How Cloud Functions works. Available online https://cloud.ibm.com/docs/openwhisk?topic=cloud-functions-abouthttps://cloud.ibm.com/docs/openwhisk?topic=cloud-functions-about, 2019 Retrieved on May 13th 2020

  15. The C4 model for software architecture. Available online https://c4model.com/ Retrieved on May 13th 2020

  16. Serverless, The way cloud should be.” Available online https://serverless.com/, 2018 Retrieved on May 13th 2020

  17. Apache, Apache JMeter. Available online https://jmeter.apache.org/, 2019 Retrieved on May 13th 2020

  18. McGrath, G.: Serverless Computing: Applications, Implementation, and Performance. PhD thesis, University of Notre Dame (2017)

  19. Malawski, M., Figiela, K., Gajek, A., Zima, A.: Benchmarking Heterogeneous Cloud Functions. In: European Conference on Parallel Processing, pp. 415–426. Springer (2017)

  20. Figiela, K., Gajek, A., Zima, A., Obrok, B., Malawski, M.: Performance evaluation of heterogeneous cloud functions. Concurr. Comput. Pract. Exper. 30(23), e4792

  21. AWS, AWS Lambda Limits. Available online https://docs.amazonaws.cn/en_us/lambda/latest/dg/limits.html,2018 Retrieved on May 13th 2020

  22. Ludovici, A., Moreno, P., Calveras, A.: TinycoAP: A Novel Constrained Application Protocol (coAP) Implementation for Embedding RESTful Web Services in Wireless Sensor Networks Based on tinyOS. J. Sens. Actuator Netw. 2, 288–315, 06 (2013)

    Article  Google Scholar 

  23. Mao, H., Qi, Z., Duan, J., Ge, X.: Cost-Performance Modeling with automated benchmarking on elastic computing clouds. J. Grid Comput. 15(4), 557–572 (2017)

    Article  Google Scholar 

  24. McGrath, G., Brenner, P.R.: Serverless Computing: Design, implementation, and performance. In: 2017 IEEE 37Th International Conference On Distributed Computing Systems Workshops (ICDCSW), pp. 405–410. IEEE (2017)

  25. Ribenzaft, R.: How to make Lambda faster: Memory performance benchmark. Available online https://epsagon.com/blog/how-to-make-lambda-faster-memory-performance-benchmark/https://epsagon.com/blog/how-to-make-lambda-faster-memory-performance-benchmark/https://epsagon.com/blog/how-to-make-lambda-faster-memory-performance-benchmark/, 2018 Retrieved on May 13th 2020

  26. Chapin, J.: The occasional chaos of AWS Lambda runtime performance. Available online https://blog.symphonia.io/the-occasional-chaos-of-aws-lambda-runtime-performance-880773620a7ehttps://blog.symphonia.io/the-occasional-chaos-of-aws-lambda-runtime-performance-880773620a7ehttps://blog.symphonia.io/the-occasional-chaos-of-aws-lambda-runtime-performance-880773620a7e, 2017 Retrieved on May 13th 2020

  27. Kaviani, N., Maximilien, M.: CF Serverless. Available Online http://schd.ws/hosted_files/cfsummiteu2016/87/CF%20Serverless.pdf 2016 Retrieved on May 13th (2020)

  28. Shillaker, S.: A provider-friendly serverless framework for latency-critical applications. 12th Eurosys Doctoral Workshop (2018)

  29. Casalboni, A.: Google Cloud Functions vs. AWS Lambda: Fight for serverless cloud domination begins. Available online https://cloudacademy.com/blog/google-cloud-functions-serverless/https://cloudacademy.com/blog/google-cloud-functions-serverless/, 2017 Retrieved on May 13th 2020

  30. Parenzan, M.: Microsoft Azure Functions vs. Google Cloud Functions vs. AWS Lambda: Fight for serverless cloud domination continues. Available online https://cloudacademy.com/blog/microsoft-azure-functions-vs-google-cloud-functions-fight-for-serverless-cloud-domination-continues/https://cloudacademy.com/blog/microsoft-azure-functions-vs-google-cloud-functions-fight-for-serverless-cloud-domination-continues/https://cloudacademy.com/blog/microsoft-azure-functions-vs-google-cloud-functions-fight-for-serverless-cloud-domination-continues/,2017 Retrieved on May 13th 2020

  31. Cui, Y.: Comparing AWS Lambda performance when using Node.js, Java, C# or Python. Available online https://read.acloud.guru/comparing-aws-lambda-performance-when-using-node-js-java-c-or-python-281bef2c740fhttps://read.acloud.guru/comparing-aws-lambda-performance-when-using-node-js-java-c-or-python-281bef2c740fhttps://read.acloud.guru/comparing-aws-lambda-performance-when-using-node-js-java-c-or-python-281bef2c740f, 2017 Retrieved on May 13th 2020

  32. Nolet, T.: AWS Lambda Go vs. Node.js performance benchmark: updated. Available online https://hackernoon.com/aws-lambda-go-vs-node-js-performance-benchmark-1c8898341982https://hackernoon.com/aws-lambda-go-vs-node-js-performance-benchmark-1c8898341982https://hackernoon.com/aws-lambda-go-vs-node-js-performance-benchmark-1c8898341982, 2018 Retrieved on May 13th 2020

  33. Billock, M.: The Serverless Performance Shootout. Available online https://dzone.com/articles/the-serverless-performance-shootouthttps://dzone.com/articles/the-serverless-performance-shootout, 2017 Retrieved on May 13th 2020

  34. Amazon data center size – Huan Liu’s Blog. Available online https://huanliu.wordpress.com/2012/03/13/amazon-data-center-size/https://huanliu.wordpress.com/2012/03/13/amazon-data-center-size/ Retrieved on May 13th (2020)

Download references

Acknowledgments

The work presented in this paper was partially carried out in the scope of the MobiWise project: From mobile sensing to mobility advising (P2020 SAICTPAC/0011/2015), co-financed by COMPETE 2020, Portugal 2020 - Operational Program for Competitiveness and Internationalization (POCI), European Union’s ERDF (European Regional Development Fund), and the Portuguese Foundation for Science and Technology (FCT).

Author information

Authors and Affiliations

  1. Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, University of Coimbra, Coimbra, Portugal

    Horácio Martins, Filipe Araujo & Paulo Rupino da Cunha

Authors
  1. Horácio Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Filipe Araujo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paulo Rupino da Cunha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paulo Rupino da Cunha.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Martins, H., Araujo, F. & da Cunha, P.R. Benchmarking Serverless Computing Platforms. J Grid Computing 18, 691–709 (2020). https://doi.org/10.1007/s10723-020-09523-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-020-09523-1

Keywords

Search

Navigation