Make it easy to list all the tests in each scenario
As part of documenting the tests, we should make it possible to list all the tests in each scenario.
The easiest way to do this is to use RSpec's --dry-run
option. However, we'll need to pass it via gitlab-qa
if we want to find out the tests in each gitlab-qa
scenario, and that would also launch a GitLab docker container (and any other orchestrated containers), which we don't want to do just to get a list of tests.
Tasks:
-
Add a --dry-run
option togitlab-qa
that makes it skip running docker commands -
Add a --dry-run
option to the QA framework so that we can skip the strategy before hooks (which is where the framework checks if the website can load the login page, which we don't want to do just to get a list of tests)
Once that's done we can use it to automatically fetch lists of the tests in JSON format, and use that to create lists that we can post somewhere and keep updated relatively easily.