Idempotency 102: serverless enters the scene

Leverage aws-lambda, dynamodb and dotnet with decorator pattern

TL;DR => Check out the code to jump into the action! :)

GitHub - EduardBargues/Idempotency-102

Idempotency series

Idempotency 101

Idempotency 102: serverless enters the scene

Background

Not long ago, I published an article about Idempotency where I gave my 5 cents about it and the reason behind it. Fintech and financial companies require their systems to be idempotent for obvious reasons (we don’t want to pay twice!) but this can be applied to any other system like an e-commerce website.

Idempotency can be seen as a sort of “protection”. Duplicated payments can be blocked and the impact of heavy request can be mitigated by returning the cache response instead.

In the previous article we developed a solution for idempotency that only worked in memory. That was great to get familiar with the idea but in production systems we need to deploy (usually) a service in our cloud provider. For this demo, I chose AWS. I won’t go into detail about AWS, just to say it is great and there are good reasons why it is number 1 cloud provider by far :) …

Let’s refresh the idea of how our idempotent system works with a diagram:

Use cases to manage in an idempotent system.

Basically, the idempotency layer will do the following:

• intercept the request,
• try to get ownership (link request and key) of the provided key,
• delegate the request to the actual api (if needed),
• and cache the response once the api answers.

For a single request, the idempotency service will try to link the request with the unique key. If nothing else is using the same key (case 1 in previous diagram), this will be owned and the request delegated to the api. If something else is using the key (case 2 in the previous diagram), the request will be aborted and a conflict-409 response will be send to the client. Note that for the requests that have already been processed, idempotency does not consider them “in use”, it simply retrieves the cache response and sends it back to the client.

Let’s say, for example, that we have a payments system that is exposed behind a Rest Api. There are few things to consider for this system to be successful.

• First, needs to be distributed: Payments can come from many places and all of them must be processed.
• Needs to (horizontally) scale: All payments must be processed within a reasonable amount of time. Let’s say 5 seconds. That’s what I’d be willing to wait in front of my mobile app when I’m doing a payment :P …
• Needs to be reliable: We don’t want people to pay double. Here is where the idempotency comes in.

For the api, I’ll be using dotnet, aws lambda and apigateway. For the storage part, I need something fast in terms of both writing and reading, so I’ll use DynamoDb. The usual serverless stack you could say …

DynamoDb

DynamoDb is an AWS NoSql database service. Provides fast writing and reading if the access patterns are supported (usually the case). Personally, I had good experiences with DynamoDb and I would definitely recommend it.

If you come from Sql, you might find it weird but I think that would be the case with any other NoSql database (MongoDb, for example). But, anyway! Let’s get to the point:

What are the access patterns in the storage layer?

Contracts for the storage layer.

From the previous diagram, we know the idempotency system will need to do 2 actions:

• Get the ownership of a key: Here we need to take into account that, in terms to own a key we need to know such key and who is asking (ownerId). Also, there’s an extra scenario to consider. Imaging that, after a service gets the key ownership (so, it can proceed with the request), for some reason fails unexpectedly. In that case, the idempotency storage needs to release the key so others (or the same service) can use it again. This scenario is covered with the input “timeToLive”, which specifies the time to wait before releasing the key in case the owner doesn’t update the stored item.
• Cache the response: Once a request has been processed, we need to cache the response. To do so, we need to know which key the request belongs to, the response to cache (in this case, statusCode and body), and the amount of time this response is going to be valid. After that period of time, the storage will invalidate the response and release the key again.

So, that gives us the 2 access patterns to care about:

• Save item: Fast operation based on the Partition Key. In our case, idempotencyKey.
• Update item: Update 2 attributes (body and statusCode) from an already existing item. Again, in this case, we only need the Partition Key.

So, only 1 access pattern! Access the item based on the Partition Key. Check out the actual implementation of the DynamoDb storage here:

content-Idempotency-102/DynamoDbStorage.cs at main · EduardBargues/content-Idempotency-102

How does the dotnet code look like?

There is still an important missing piece. How are we going to know when a conflict happen? And, how can we know that a response has already been cached?

To answer those questions, we need to check the code in the IdempotentFunction.cs file:

content-Idempotency-102/IdempotentFunction.cs at main · EduardBargues/content-Idempotency-102

IdempotentFunction.cs file. Responsible to decide what to do based on the storage response.

Here, we see a method that receives 5 inputs:

• APIGatewayProxyRequest: The actual request send by the client that comes from the ApiGateway.
• ILambdaContext: The Lambda execution context. Contains a logger, trace data and other things.
• IFunction: This is the actual api or, in this case, the function that holds the actual implementation. This function will be called in case the request has not yet been processed and the idempotency delegates it.
• IIdempotentStorage: The storage that we decided to use. In our case, DynamoDb (behind the interface previously described).
• IdempotencyConfiguration: Configuration about headers mainly.

As you can see, the main flow of this method is:

1. Check that the idempotency key is provided in the appropriate header. If not, return a BadRequest-400 response.
2. Try to get ownership of the key in the current request. Note that the ownerId is defined as the request id.
3. If the request has been previously processed, we just return the cache response (lines 17–18).
4. If the request is in progress by another owner, we return a Conflict-409 response (lines 19–20).
5. If we are the owner of the key, we delegate to the actual implementation the request execution.
6. Once the actual function responds, we cache the response and send it back to the client.

There is much more to cover in terms of code but I don’t want to bother you. If you are particularly interested about Idempotency you will dive deep into the code. It is open source! :)

Lambda

Lambda code to process transactions.

Let’s talk a little bit about the lambda that is supposed to hold the bussiness logic. In the previous picture I’m showing the class ProcessTransactionFunction.cs where we can see a single method called FunctionHandler. Also, there are 2 constructors: the empty one is the one AWS is going to execute once the Lambda gets called and the one in line 17 injects the lambda dependency for testing purposes. As a dependency injection container, I use the one build in Asp.Net core framework (IServiceCollection).

The main method can be divided in the following parts:

1. Initializing a DI scope for the Lambda execution.
2. Obtain some data that will delay the operation and make it succeed or fail (take into account that this lambda is for testing purposes.).
3. Resolve the dependency from the DI container and make it process the transaction.
4. Create the APIGatewayProxyResponse object and send it back to the APIGateway.

The decorator pattern

This Lambda works great (it doesn’t do much, thought :P …), but we want to make it idempotent without changing it’s internal implementation. To do so, we create a new ProcessTransactionIdempotentFunction.cs with the same signature that decorates it with the idempotency feature.

Idempotent Lambda function wrapping the actual IFunction implementation.

This second Lambda follows the same pattern as before:

• One empty constructor, in line 8, for production executions where the real dependencies are injected.
• A second constructor, line 14, where the dependencies are injected for testing purposes.
• A single main method called FunctionHandler that is going to be the one AWS will execute. Note this method executes the IdempotentHandler method inside the IdempotentFunction.cs class.

And with only this extra class, we are able to make our process transaction feature idempotent.

Infrastructure

Let’s talk about how is this system going to run in AWS. At the beggining of this article I listed the resources we were going to use: ApiGateway, Lambda, DynamoDb.

All those resources are managed using Terraform. My favourite infrastructure as code tool out there. I don’t want to get lost in the terraform details because, although it is worth it, don’t want to bore you. Here you have a piece of the terraform code with the main resources:

Part of the terraform code representing the apigateway, lambda, integration between both, and the dynamodb.

QA testing

We’ve built a complex system that can protect our payments system against duplicated requests and cache the responses to ensure our clients always get the same answer for the same request (literally, idempotency), but we didn’t test it much. Only some unit testing to make sure our code made at least sense.

For me, QA is the most important part of the whole DevOps cycle and I always try to develop my apis considering how is the client going to use it and how we are going to test it. Note that, for the sake of proper testing, I included extra headers that allow me to monitor several operations times (Not included here for the sake of simplicity but definitely look for them in the code ;) ).

In this project I included 3 types of tests:

• load testing: ensures the services behaves properly in terms of processing times under several stressful scenarios.

Unit tests

Here, I’m testing the service works properly with their dependencies mocked. They allow me to cover the main paths of my use cases and ensure they are covered before deploying. Here, I’m just including the simplest test but you can check them all out in the code.

One of the many unit tests written in c#.

E2E tests

Here, I want to answer the question “how is my service going to behave once deployed?”. Those tests ensure that the system will behave accordingly under defined scenarios in a production like environment.

For those tests I’m using Jest. It is a Javascript based testing framework. What can I say? I love Javascript :P … Jest allows me to start writting my tests and run them in a very intuitive and easy way. I love it!

Main e2e testing scenario written in Jest.

Here I’m only showing one scenario with several checks. I’m basically checking:

Some Jest execution logs.

Load tests

Finally, my favorite part of testing. Load testing! Where you really put your system under stress and see how it behaves. Definitely a place where everything can surprise you:

• Sudden failure of Lambda-DynamoDb connection.
• Soft or Hard limits in AWS.
• Unexpected behaviour due to high concurrency.
• … and many more! That’s why I call them surprises :P …

Anyway, since I’m a Javascript lover, I usually go with k6 as my load testing framework. It is a great framework and allows me to easily transition from coding to testing. Also, comes with a lot of features and it is open source!

GitHub - grafana/k6: A modern load testing tool, using Go and JavaScript - https://k6.io

Let me show you one of the load testing scenarios I coded in javascript using k6:

caching-scenario.js run using k6.

With it’s corresponding and beautiful results in the terminal:

Performance results in milliseconds.

There are some really interesting stats here:

• _idem-caching-time: time to cache a response in DynamoDb. Only 15 milliseconds!
• _idem-get-ownership-time: time to link a request with it’s idempotency key. Around 38 milliseconds. Not as fast as the caching time …
• _req0: response time in the first request without caching. 144 milliseconds.
• _req1: response time with response already cache. 134 milliseconds.

Conclusions

First of all I apologize. I underestimated the amount of data and topics I would have to throw into this article. I supposed something I learn for my next one (talking about splitting topics in smaller chunks).

Anyway, what have we done?

• We developed a serverless application that is able to process payments.
• Deployed such app using terraform to AWS.
• Developed an Idempotent package that allows our serverless app to become idempotent without modifying it’s internal implementation.
• Developed and run dozens of tests of several types: unit, e2e, and load testing.
• Idempotency is a must in certain industries and here I present you a possible solution based on dotnet.

All the code is available in my Github (check out the beginning of this article). I would love to hear from you and any feedback you might have. I’m sure I missed something. Happy coding :) !