Execute QA suites against release environments

Currently, the following process happens when a new release is tagged on a stable branch

  • The tag is created in the GitLab repository
  • The packages (both CNG and omnibus) are created from that tag, containing the code that was tagged
  • We deploy this new package to https://release.gitlab.net
  • Some level of QA is run against the https://release.gitlab.net environment to give us confidence the package is ready for release

As part of the ongoing effort to expand the maintenance policy, we are looking at expanding https://release.gitlab.net to include more "release environments" (epic here) and to deploy to them ahead of the release being tagged. This new process looks something like

  • Developer merges a backport to a stable branch
  • The package (currently just CNG) are created from the contents of the stable branch with the new merge/commit in question
  • We deploy this new package to new release environment depending on stable branch. e.g. branch 16-6-stable-ee is deployed to https://gitlab.16-6-stable.release.gke.gitlab.net
  • Some level of QA is run against the environment https://gitlab.16-6-stable.release.gke.gitlab.net to give the developer feedback and confidence their backport still passes deployment and testing, and to give release managers confidence that if a tag/release was done, tests would pass

We are looking to understand with help from quality what the last point would look like. How do we run QA against release environments with an arbitrary name, running a version of GitLab that may be older (depending on how big our maintenance policy extension is), and tests matching that version

Old description

Once we have images being built from merged commits onto stable branches in the gitlab-org/gitlab project (e.g. onto 15-7-stable-ee) and we have deploys to the appropriate release environment from these deploys (e.g. to environment 15-7-stable.release.gke.gitlab.net), we need to execute a QA suite against the release environment in question to give feedback and ensure the stable branch deploy is valid.

We need to identify and build a pipeline task or pipeline trigger to run a QA suite against the environment in question (e.g. 15-7-stable.release.gke.gitlab.net) and the test suite in question should be valid for the version running in the environment (e.g. the test suite should also be from the stable branch 15-7-stable-ee).

The trigger to these tests will happen in the gitlab-org/gitlab repository, on the pipeline that runs on merge to a stable branch.

Edited by Graeme Gillies