Create a script / Rake task to test the automated GitLab HA or Geo upgrade to ensure no downtime
For the Test no-downtime HA upgrades (including Geo) Epic, we're proposing to create a CI job to kick off testing the automatic HA, Geo upgrade. Said CI job will need to contain a series of steps with one being the automated testing of a GitLab HA or Geo update with no downtime:
-
Run the entire smoke test suite in a loop
-
Run specific smoke tests in a loop to ensure we cover critical functions, e.g.
- Sign in
- Sign up
- Creating Projects/repositories
- git push (push to secondary for Geo) / pull
- ...
We should aim to run 1. and the items in 2. in parallel to ensure we catch any potential hiccup.
We need to ensure we have tests to satisfy known issues that have occurred during an upgrade:
- https://gitlab.com/gitlab-org/gitlab-ce/issues/63510 - Signing in failed
- https://gitlab.com/gitlab-org/gitlab-ce/issues/64145#note_188616038 - Releases API failed
- https://gitlab.com/gitlab-org/gitlab-ee/issues/12458 - CI API failed
Edited by Ash McKenzie