Dynamic automation of Contract Testing - Provisioning and Job
This issue is for discussing possible solutions to automate the now merged Contract tests that are for now only runnable in a fully manual mode, where the user needs to provide a merge request to run the tests against it.
What is currently possible
For now, our contract tests depend on an already running instance that is provisioned with a merge request that contains at list one comment. Upon running the test, you need to provide the script with the host as well as the merge request against which to run it. Such a command looks like this:
bundle exec bin/contract http://localhost --mr /path/to/the/mr
This of course is very much not how we want the finished product to look like. This is the reason, why we need to find a way to run it against a working API into which we can inject the e.g., merge request we need.
Update: Andrejs has provided an MR on how to make the run more integrated with our Rake tasks.
What is the problem
The main problem we are currently facing is provisioning. Contract tests are similar to E2E tests and just one step below them on the testing pyramid. This means that they need to be run against a production-like API which means that our tests need to run preferably against a docker instance of GitLab like we are currently using on our E2E tests. Now due to the nature of Contract tests, they need data that they can request to test the validity of the response against the contract. For example, we have a test which tests the endpoint /discussions.json
which is available on every MR. The tests currently test the common variables that every discussion on an MR needs to have. There are differentiating data points between discussions that are resolved, discussions that are on a line of code etc. We need to have different MRs for all of them if we want to test that.
The next problem is that these tests need to be fast. Once they run, they should be close to unit test speeds. Using Selenium for creating a Token is due to this not really an option. It would create a large overhead for a simple test like this.
The last one is mostly solved if we solve the first one, which is to run this against the current MR inside a job as we are currently doing with unit tests.
Edit: Contract tests lie between unit tests and integration tests. The consumer tests are similar to unit tests while the provider tests are similar to integration tests. This means that we don't need to run these tests in a live server. In fact, Pact runs against Rack::Test by default as it assumes it will run against the Rack server (ref: https://docs.pact.io/implementation_guides/ruby/verifying_pacts#using-rake-pactverify). With that in mind, we can use the same data generation library that the unit and integration tests use instead of relying on the API to generate the data.
Potential solutions
This will get updated with suggestions from the comments
1. Create an access token during creation
One possible and probably the most simple solution (though most uncertain) is to find a way to create a token during the creation of the docker environment. This would happen the same way we define that we want and admin user with a specific password. I do not think there is such a capability for now. So this solution would depend on the help of other teams and the implementation of such option, if it is even viable due to security concerns. When we have the token, we can use the public API to provision anything we need.
2. Do not use docker but a striped down version of the API
This is something where I am not sure if it would work, as I have not really got an answer from the back-end team for now. The idea would be to spin up just the API using not a docker image but rather directly the project. It has to be a running API, though, otherwise it would just be a standard unit test. We provision data by using a mocked object the API receives from us. This idea is very much still theoretical.
3. Integrate it into our E2E suite (not an option)
In this idea we would integrate the contract tests into our suite and just use selenium to create the token and store it in a variable as we are doing right now. We would then provision the data using our resource classes and run the contract tests. This would go against the fundamental principle of contract tests and in general these types of tests: fail fast. It would run in our package & qa
job which would drastically increase the time until the developer would see results.
4. Use FactoryBot gem to generate the test data and use Rack server
Update: This turned out to be the best option as the tests now run similarly to our lower level tests and can quickly be deployed in various pipelines.
This removes the dependency on a live server and the need for an API token as FactoryBot will handle the data generation directly to the database and the Rack server setup will allow our test to run without relying on a live server.
Links
- The mentioned MR also explains more about contract testing and how it is currently run