WIP: Add loop tag to some E2E specs to run them repeatedly
What does this MR do?
- Moves login test behavior into a shared_example, so that we can specify looping instructions using a '--loop' tag
- Make a ‘loop’ rspec tag in the login spec, to tell the runner to continuously run the test (actual ending condition TBD, for now just loops 10 times).
Why do we want to do this?
We want a way to run certain E2E tests repeatedly (e.g. for 40 minutes) as part of a CI job.
- We want to check that zero-downtime upgrades of GitLab are truly zero-downtime
- While upgrading a live (non-production) GitLab instance, we want to run several smoketests against it continuously and check if any tests fail during the upgrade process.
- Failures would show that the upgrade may not be zero-downtime.
Idea:
- Make a new pipeline project similar to qa-staging pipeline, that will use GitLab-QA to run tests against a live instance.
- Each test (e.g. login, create project, create issue) would run in their own jobs, would be triggered manually and would run continuously for the duration of the upgrade event
For each job the ci.yml file would then call gitlab-qa Test::Instance::Any EE $GITLAB_ADDRESS -- qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb --tag loop
WIP CI project: https://gitlab.com/jennielouie/jsl-test-ci-project