review-qa-reliable job runs different tests on retry
We had a regression introduced with !85761 (merged) even though content_editor_spec.rb
reliable test failed multiple times there. The merge was successful because on one of the retries review-qa-reliable 9/10
job ran a different set of tests from the first run that failed and didn't include that test - see https://gitlab.com/gitlab-org/gitlab/-/jobs/2417964639 vs https://gitlab.com/gitlab-org/gitlab/-/jobs/2418307765. There might be a bug in retry logic that results in misleading results - we need to investigate this.
See gitlab-com/gl-infra/production#6991 (comment 938953171)
Fixes:
-
Update review-qa pipeline -
Update reliable and smoke for live envs -
Update full runs on live envs
Edited by Andrejs Cunskis