Add details on what to do when master is broken: revert or quarantine
- Now that https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26920 is merged, we only run MySQL and PG 10 on
master
and on schedules which means that we may find problems specific to those only once it hitsmaster
.- It was suggested in https://gitlab.com/gitlab-org/quality/production/issues/4#note_157556285 that we should be more aggressive in reverting MRs that introduce such failures.
- With https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26934 we stopped automatically retrying jobs that fail at the
script
step, meaning that flaky tests can now lead to a broken pipeline, so I think we need to be more aggressive about putting specs in quarantine now since we won't retry those automatically at the job level, which means that we'll detect them more easily but also thatmaster
may fail more often because of that (but at least it won't be "hidden" by automatic retries).
This workflow update addresses those 2 things.
@meks @stanhu @marin @gitlab-org/maintainers/rails-backend WDYT?