Update release/docs with information about what version of the test code is used in live environment tests
Summary
During production#20111 (closed), we discovered that the version of the test code that is used when running End-to-End tests against a live environment is not from the same version of the GitLab code as the one that is deployed in that environment.
Link to Slack thread, internal-only, available until 2025-10-04
So, when an MR (such as gitlab-org/gitlab!195868 (merged)) which includes changes in the test code that will not pass against older versions of the application is merged into master, QA specs will start failing on all the live environment tests.
For deployment pipelines, we say in release/docs that the test code that runs is the same version as the one that is being deployed.
QA tests are run against the live environments. ... by retrying a job on an older pipeline, the test code being run will be from that pipeline, but the application code deployed to the environment may be from a newer pipeline
However, this is not true, we always run the test code from the master branch:
[Jul 03 2025 22:52:29 UTC (Gitlab QA)] INFO -- Shell command: `docker run -t --rm [snip] registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:master Test::Instance::All https://staging.gitlab.com --address https://staging.gitlab.com [snip]`Source: Job from Deployment QA pipeline - 18.2.202507032007-fd3e59b5144.e5e536b044d
This does not happen on MR pipelines because end-to-end specs run using bundle exec bin/qa and uses the locally checked out test code:
Running e2e specs via command: 'bundle exec bin/qa Test::Instance::All http://gitlab.172.18.0.2.nip.io -- --order random --force-color --format documentation'