Handle E2E Tests During CustomersDot Staging Deployments
Problem
CustomersDot E2E tests currently experience intermittent 502 Bad Gateway
errors while there are ongoing deployments to CustomersDot Staging
See Intermittent 502 Bad Gateway Errors in E2E Tests in Staging for examples on how this is negatively impacting E2E tests and causing flaky failures
groupfulfillment platform does have plans to migrate to zero downtime deployments, but in the meantime, we should see if there is a solution in the interim to help increase the stability of our E2E tests
Proposal
Come up with a strategy and implementation plan for how to best handle E2E tests during deployments to CustomersDot Staging
One approach could be to explore pausing E2E test jobs until Staging deployments from any previous pipelines are finished (and/or also pausing Staging deployment jobs when there is an E2E job running). We may be able to leverage GitLab's CI/CD resource groups to control this.