Serverless performance battle

How dotnet can stand a chance against NodeJs and Python in the serverless world.

Icons made by https://www.flaticon.com/authors/webalys from https://www.flaticon.com/.

TL;DR => Check out the repository! All the code ready to run so you can go to the point.

https://github.com/EduardBargues/cicd-template

Background

Almost all my development life has been around dotnet. Almost 10+ years of development working on amazing projects in different sectors and always going with c#. I really like dotnet and it’s ecosystem. I hope it keeps getting better and amaze me with new features :) !

dotnet is considered very performant. In fact, asp.net core has been among the fastest web frameworks since I can remember but it wasn’t until recently that I encountered an issue (not really an issue, but my understanding of the framework was challenged).

3 years ago, I started working with serverless technologies such as Lambda in AWS. At that point, I was quite noob so my first iteration was simply repeating what I was used to. Our goal at that moment was to create a REST API based on serverless for one of our clients so my first solution resembled something like this: https://github.com/EduardBargues/cicd-template/tree/main/src/dotnet/WebApi

Simple webapi project with asp.net core and a single controller.cs.

Controller.cs with one endpoint. Logs request info, call internal service and returns response.

As you can see, nothing far from a basic dotnet webapi.

The problem (?)

I learned how to create lambda packages from dotnet webapis and I deployed that lambda together with an APIgateway using terraform.

• script to create (and upload) a dotnet lambda: https://github.com/EduardBargues/cicd-template/blob/main/scripts/artifacts/lambda/dotnet/create-dotnet-lambda-artifact.sh
• terraform file where a lambda is defined: https://github.com/EduardBargues/cicd-template/blob/main/terraform/lambda-dotnet-webapi.tf
• terraform file where the lambda is integrated with the APIgateway: https://github.com/EduardBargues/cicd-template/blob/main/terraform/api-gateway-integrations.tf

Script to create a dotnet lambda artifact and upload it into an S3-bucket.

integration between webapi lambda and apigateway using a terraform module.

I deployed the serverless stack (apigateway + lambda) and did some performance testing. Then, I realized that the response time for the first request was near 2000 ms :O ! How is that possible? We are just building a “hello-world” lambda (It just logs some info and response with an okay).

I was a little bit shocked. I’d been told that asp.net core was one of the fastest frameworks. What was I missing? At that moment I was having my first encounter with a well known “problem” in the serverless world: Cold-start.

Cold starts (and how to reduce them) are a great topic and maybe some day I’ll do an article about them. Very succinctly, cold starts occur when a Lambda is invoked for the first time. At that moment, AWS downloads your artifact, unzips it, compiles your dotnet code to the specific architecture defined in the Lambda’s context, and finally executes your code. In addition, for web api Lambdas, we need to add the startup time (usually, all the operations defined in the Startup.cs class). That last part, from my experience, can be the greatest contributor to your cold start time.

Bufff… No wonder it takes 2000 ms :D ! I did some research and the serverless community was suggesting to use interpreted languages like NodeJs or Python. The reason is those languages are not compiled so it is faster to spin up (and, if your functions are kept small, are also fast to execute).

The solution

I wanted to do my own independent analysis, so I coded 4 lambdas in different languages and approaches. All the lambdas do the same: log some information and return a OK-200.

• NodeJs: https://github.com/EduardBargues/cicd-template/blob/main/src/nodejs/lambda/main.js
• Python: https://github.com/EduardBargues/cicd-template/blob/main/src/python/main.py
• dotnet webapi: https://github.com/EduardBargues/cicd-template/tree/main/src/dotnet/WebApi
• dotnet serverless:https://github.com/EduardBargues/cicd-template/tree/main/src/dotnet/Function

NodeJs lambda.

Python lambda.

dotnet serverless lambda.

I deployed all of them behind an api gateway and created a k6 script (https://github.com/EduardBargues/cicd-template/blob/main/tests/performance/performance-tests.js) that calls thousands of times the api. The results surprised me a lot! (https://github.com/EduardBargues/cicd-template/runs/3663784153?check_suite_focus=true)

performance analysis results. Done with k6 (www.k6.io).

Comparison

Some interesting numbers to highlight:

• Both nodejs and python endpoints respond, in average, in 106 ms (_nodejs_function and _python_function rows).
• The endpoint supported by the dotnet webapi lambda (_dotnet_webapi) responds, in average, in about 137 ms.
• The endpoint supported by the dotnet serverless lambda responds, in average, in about 109 ms. Pretty much the same as the ones in NodeJs and Python!

Average response time is important but the response time for the first request is key in the serverless world. Let’s see those numbers (“max” value on the corresponding row):

• Python goes first with a max response time of 341 ms.
• NodeJs follows with 380 ms (Does NodeJs have worse coldstarts than Python? Not enough data to know at this point).
• dotnet serverless endpoint slowest response took about 765 ms. We are doubling NodeJs but it is, in my opinion, and acceptable first request response time.
• Last but not least, dotnet webapi endpoint took 1917 ms. 2.5 times the dotnet serverless endpoint.

Conclusions (and opinions)

Although the explanation took a couple of paragraph and photos, there is a lot going on here. I tried to provide a big picture explanation instead of a detailed one. If you are hungry for more all the code and analysis are public in my github repo :). Go crazy!

Now let’s write some highlights:

• NodeJs and Python are the same in terms of average response time.
• NodeJs and Python languages seem to beat dotnet. Both in average and cold start response times.
• Dotnet-serverless lambda beats dotnet-webapi lambda in average response time by 21%. Not bad!
• Dotnet-serverless and beats dotnet-webapi lambda in first request response time by 60%. Holy cow!
• Since both dotnet webapi and serverless lambdas are coded with c#, it is safe to assume that the different in coldstart and average response time is due to asp.net core loading time.

Here we go with MY opinions :D :

• I would chose NodeJs for serverless development. I find it very easy and a lot of libraries are already in place thanks to AWS and the community.
• Python is definitely a great option. I have less experience with it but I don’t see any problem. Definitely a go-to option for your serverless development.
• In case of coding your lambdas with dotnet seems that the reasonable approach is to go with the serverless approach. I see simplifications in the overall code and great performance increase. Also, asp.net core has proven to significantly reduce performance.
• Serverless development provides a bussiness-first approach to developers. That means that other concerns can be managed by other parts of your infrastructure: authentication, throttling, cors, api-endpoint versioning, … All those things, and many others, are provided by asp.net core and become redundant in a dotnet lambda.
• That doesn’t mean asp.net core is bad for serverless development. It’s just something to consider. In fact, moving from a legacy solution, webapi approach allows you to inmediately deploy your apis as functions without changing any code. Something to consider from a planning point of view.

Thanks for reading until the end. Hope you enjoy it! I’d love to hear your feedback. Did I make a mistake somewhere? Would you do something diferent in the analysis? Something more even? #HappyCoding!