RFH - Help with answering questions about test code version used to run tests against live environments

Context

The incident gitlab-com/gl-infra/production#20111 (closed) was caused because specs on the default branch of gitlab-org/gitlab were running against the live gstg environment (the main stage of the staging environment).

The failure was caused by a copy change MR: gitlab-org/gitlab!195868 (merged) The live environment was running the older version of gitlab-org/gitlab without this MR and thus with the copy Unresolved thread, while the test was expecting to see the copy Open thread.

I opened gitlab-com/gl-infra/delivery#21353 (closed) to discuss this further. I was redirected to open a RFH here, to get help from the #s_developer_experience team.

Investigation required

We have some questions that I have not been able to answer looking just at the logs of the test jobs. I have attached sample logs for each question.

  1. Is the version of specs used decided by the version of the image registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa that is used by the qa-smoke jobs? If this question is answered, I can look at the logs to confirm the answer for my other questions here.
    1. This job uses the tag :master for this image. Does that mean that the test code is also from the master branch? Or is the version of the gitlab-ee-qa image irrelevant?
  2. Do we run specs from the master branch of gitlab-org/gitlab against staging during a post-deploy migrations pipeline?
    1. qa:smoke:post_deploy_migrations:gstg
  3. Is this happening on smoke and smoke-main tests executed against staging canary during auto-deploy?
    1. qa:smoke:gstg-cny, qa:smoke-main:gstg-cny
  4. Does this also apply to the full quality set triggered after gstg-cny by the auto-deploy pipeline?
    1. qa:full:gstg-cny
  5. What about the smoke and smoke-main specs running on gprd-cny?
    1. qa:smoke:gprd-cny, qa:smoke-main:gprd-cny

Expectation

We expect that the test code that is used while running tests against live environments will be for the version of gitlab-org/gitlab that is running in gstg-cny. We deploy to the environments in this order:

flowchart LR
    gstg-cny --> gprd-cny --> gstg --> gprd

Users on GitLab.com will be served from a mixture of gprd and gprd-cny pods. So, tests for the version of the code on gprd-cny should always pass (mixed deployment testing).

During the incident response, we were talking about how doing this could potentially lead to a test failure whenever we have an MR that is not backwards compatible. This is probably OK and we should just ensure that every MR to gitlab-org/gitlab is backwards compatible?