Pin version of test code used for live environment tests to the version that is running on the environment

Summary

During the incident production#20111 (closed), we found that test code from the master branch of gitlab-org/gitlab was being used to test the live environment gstg. gstg is always going to be behind the master branch though: So, some test changes will never pass: We will need to manually quarantine the test, wait for the required code change to reach gstg, and then manually un-quarantine the test. This is toil.

We have a short-term fix to understand and document what is going on right now: #21313 (comment 2604304263) Ideally, however, we would like to pin the version of the test code that is used to the code version that is running on the live environment.

This might require changes to release/tools to pass the version running in the live environment to the quality projects when we trigger pipelines there, such as this CI trigger job configuration:

https://gitlab.com/gitlab-org/release-tools/-/blob/12cb978e40a1a30d1cf6028f02d5b57969b6028c/.gitlab/ci/post-deploy-migrations-pipeline.gitlab-ci.yml#L229-242

Acceptance criteria

  • Figure out whether this is actually true: Are we using the master branch specs and running them against live environments?
  • Figure out how to pin the test code version and what changes are required in release/tools and quality CI pipelines
  • Implement the required changes!
Edited by Siddharth Kannan