Pin version of test code used for live environment tests to the version that is running on the environment
Summary
During the incident production#20111 (closed), we found that test code from the master branch of gitlab-org/gitlab was being used to test the live environment gstg. gstg is always going to be behind the master branch though: So, some test changes will never pass: We will need to manually quarantine the test, wait for the required code change to reach gstg, and then manually un-quarantine the test. This is toil.
We have a short-term fix to understand and document what is going on right now: #21313 (comment 2604304263) Ideally, however, we would like to pin the version of the test code that is used to the code version that is running on the live environment.
This might require changes to release/tools to pass the version running in the live environment to the quality projects when we trigger pipelines there, such as this CI trigger job configuration:
Acceptance criteria
-
Figure out whether this is actually true: Are we using the masterbranch specs and running them against live environments?- The expectation is that if we were doing this, then we would see a lot more test failures than we actually do.
-
Answer: Yes, we were using tests from the
masterbranch due to gitlab-org/quality/pipeline-common!523 (merged).
-
Figure out how to pin the test code version and what changes are required in release/tools and quality CI pipelines - Answer: This is not required. This was the behavior before the above MR which caused a regression. The bug was fixed in gitlab-org/quality/pipeline-common!532 (merged).
-
Implement the required changes!