Unit, Component, Integration, E2E, and Performance testing
What for, when, and how to use them
TL;DR => Busy day, am I right :P? Check out this repository to go directly to the code!
Background
As a DevOps engineer, I deeply care about the development, testing, and operations parts of a software product. I keep seeing this “battle” between developers, QAs, and operations people: devs just want to code, ops don’t care about the features and how to use them as a client, and QAs think devs break everything.
In my opinion, to properly create a software product the most important aspect is ownership. I find very difficult to manage a project without seeing the big picture and to do so there is no other way than to know a little bit of everything (maybe that’s why I became a DevOps).
Talking about big pictures, I usually divide the whole software project flow in the following parts:
- Development: Where the coding happens. Done mainly by devs.
- Integration: Where git happens :). Concepts like branch-strategy and pull-requests appear. Also, some teams manage to embrace continuous-integration once their setup is mature enough.
- Testing: There are many different ways to test such as unit, component, integration, end to end, performance, … and you can do it with a lot of tools: dotnet with xUnit, javascript with Jest, docker, docker-compose, k6, … It’s the never ending list! Also, this article will focus on testing and how the different types can complement each other.
- Delivery: This part takes care of building, packaging, and creating the artifacts (from .zip files to docker images) and exposing them in a repository to be pulled from later (s3 bucket, ECR, github packages, …).
- Deployment: Usually the team pulls the previously generated artifacts and make them available in a certain environment. Could be a development, staging, or production environment.
- Monitoring: Once the application is deployed we want to know how it is performing. Errors, Time to respond, internal communication latency, and many others! Here, tools like CloudWatch, DataDog, and Checkly appear.
- Scalability: During the lifetime of an application in production, mainly operations people take care to manage how resilient the application is. If an application is deployed in an EC2 instance we want it to scale horizontally in case the request number goes up and the same happens with ECS services or serverless set ups.
Since the big-picture is indeed BIG, let’s focus the discussion on Testing. In most of the teams I’ve been involved with has always been a topic of discussions between devs (coding), qas (testing), and ops (providing infrastructure for testing and running the app). In my opinion, the roles should be forgotten and embrace ownership as a team. That means, everybody knows a little bit of everything. Obviously, there will be leading people in certain aspects but we shouldn’t centralize the knowledge on individuals.
Let me show you how we managed the testing part of our web applications in the teams that were successful in the past. Hope you like it!
Unit testing (UT)
It is the thing that most developers think about when talking about testing. UT usually takes care of a specific class (if we specify our analysis to dotnet, of course). For example, taking as model the repository this article talks about, let’s see how a unit test would look like for the controller.cs class (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/src/api/WebApi/Controller.cs):
As you can see, the controller has 3 dependencies: ILogger, IUrlProvider, and IDependencyService. Using xUnit and Mock we can create a unit test like the one in the UnitTests.cs class (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/src/api/Tests/UnitTests.cs).
The unit test is divided in 3 blocks:
- ARRANGE: We mock the controller dependencies and setup their responses.
- ACT: We invoke the controller’s Post() method.
- ASSERT: We make sure we receive the expected response.
Component testing (CT)
UT is great, but has some drawbacks:
- Narrow scope: We are only testing a single class. What if our API has dozens?
- Implementation coupling: Since we are only testing classes, the tests are coupled with the internal implementation of our API. If we decide to change the application architecture, we better prepare to change a lot of tests.
CT aims to solve those 2 problems. The main idea is that we test our API as close as possible to how a client would. Thanks to Asp.net core, we can launch a test-server during a test execution that will locally expose the API and all it’s endpoints. To do so, we need to implement a class that I called CustomWebApplicationFactory.cs (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/src/api/Tests/CustomWebApplicationFactory.cs).
This class takes care of overriding your dependency injection container so you can mock external dependencies of your service. As you may see, I’m essentially using the same Startup.cs (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/src/api/WebApi/Startup.cs) and mocking only the services that my API depends on.
Then, we can do a test that makes much more sense from a client point of view (and also simplifies everything a lot! ). It goes like this (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/src/api/Tests/ComponentTests.cs):
The structure looks the same as before but there is a big difference. In the ACT section, we are actually making an Http call to the test server (created by the factory class on testing-time). This allows testing the API without relying on the internal implementation, which is a great deal! Also, they run like any other UT.
Integration testing (IT)
UT and CT basically ensure that your application works fine considering that the dependencies behave as expected (that’s why we mock them). So, the need to ensure that all the pieces of the system properly work together is the next challenge. We need to integrate them.
In legacy companies, the integration is taken care by ops and functional QAs. Usually, somebody spins up big machines and deploys everything there and passes it to a functional and/or automation QA that run some tests. I don’t need you to explain how far this is from continuous-integration oand continuous-deployment.
It is obvious that we cannot follow the legacy approach if we want to be agile. Also, QAs and devs can work together to own this part of the testing and there are great tools out there that help us achieve that: docker and docker-compose! Let’s see them in action.
First, some explanation about the service we are trying to test:
- It is composed of a main API that exposes 2 endpoints (see https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/src/api/WebApi/Controller.cs).
- Service discovery: In some environments, usually there is a central place to go to find the services (urls) that an application need to properly run. Since in a microservice approach deployments can happen at any point, companies centralize the service-discovery burden into a central API that abstracts it. In our case, this API is in the api-service-discovery folder (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/tree/main/src/api-service-discovery) and looks like:
- The main API has another dependency (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/tree/main/src/api-dependency) that basically acts as a sort of database. See the controller’s implementation:
So, the main API needs to communicate with the service-discovery and it’s dependency (that is why we mocked them in the UT and CT) but we don’t want to actually deploy them because it might take some time and/or costs us money.
Here, docker and docker-compose can help us and set up a local environment where all 3 applications will be deployed and communicate to each other. Then, we can run some tests against the main API without leaving our local environment.
Let’s first see the dockerfile (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/dockerfile.api) for the main API:
The dockerfiles for the other 2 APIs are similar so I’m not including them here.
Also, we need to dockerize the integration tests, located here (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/tree/main/tests/int)
As you can see, the ITs do:
- POST /api and check for Ok-response,
- and call GET /api to check the dummy entity is properly saved.
Having those 2 things in place we can use docker-compose (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/docker-compose.yml) to include all 3 APIs and the service that will perform the ITs.
Launching the run-int-tests.sh script (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/scripts/run-int-tests.sh) we obtain the following result:
End to end testing (E2E)
We’ve seen that IT allows QAs (and devs!) to test everything in a “real” environment thanks to docker and docker-compose.
Docker-compose brings a nice feature that allows to define networks and put containers(services) there to communicate with each other. If you pay close look, you’d see that:
- main network: contains api, api-dependency, and api-service-discovery.
- qa-network: contains api, api-dependency, and tests-int.
- client-network: contains api, and tests-e2e.
Doing a similar setup as with the IT, we can run the script run-e2e-tests.sh and obtain the following result:
Performance testing (PT)
Now my favorite, the PT :). Here I’m going to use a very nice framework called k6 (https://k6.io/) that allows creating a very simple JavaScript file (https://github.com/EduardBargues/content-unit-component-integration-e2e-and-performance-testing/blob/main/tests/perf/main.js) and launch it many times simulating many users. Really, check it out because I was impress by how easy and complete k6 is.
Executing the run-perf-tests.sh we obtain this really nice output from k6 container:
Comparison and Conclusions
In this article, we did a shallow introduction on how we can manage several kinds of testing when dealing with a web API in a micro-service setup. We’ve also seen that certain tools allow the team to take full ownership of both the development and testing of the application.
As final remarks, we can say:
- Unit testing: very easy to code + small impact (only a class) + difficult to maintain (they are coupled with the actual implementation).
- Component testing: easy to code + medium impact (they ensure the app works fine mocking it’s dependencies).
- Integration testing: moderately difficult to code + high impact (they ensure the app works fine in an environment together with it’s dependencies). For me, the difficulty to set up the integration tests (dockerfiles and docker-compose.yml) is compensated by the impact those tests bring.
- End to end testing: moderately difficult to code + high impact (they ensure the app works fine from a client’s point of view). E2E testing talks the same language as your clients.
- Performance testing: difficult to code + high impact (they ensure your app works fine under extreme load).
You might think that integration-testing and e2e-testing overlap but I think they serve different purposes. Note that, during integration testing, we have access to services that are not available to the client (the case of the api-dependency in our example). This is super useful for checking intermediate states inside our system. For example, between the api and the api-dependency we could put a Redis-like cache database and ONLY through doing integration testing we would be able to test it.
Special thanks
I want to dedicate one sentence to a person that has taught me a lot about testing. This person is Yan Cui. His appsync-master-class course is great! https://theburningmonk.thinkific.com/courses/appsync-masterclass-premium (yes, it sounds weird but it is a long course and in some lessons he goes into detail about testing strategies). Disclaimer: He is not paying me anything :P ! He doesn’t even know me :D !
