Thursday 2024-03-28 13:00 UTC - `gitlab/gitlab-ee` broken `master` with build-gdk-image, cache-qa-gems-e2e-runners, update-gitaly-binaries-cache-fips, update-gitaly-binaries-cache, setup-test-env-fips, setup-test-env
gitlab/gitlab-ee pipeline #320980 failed
| Pipeline ID | Branch | Commit | Merge request | Source | Duration | Triggered by |
|---|---|---|---|---|---|---|
320980 |
master |
Merge branch '443662-save-data-in-spec' into 'master' | N/A | schedule |
356.47 minutes | Rémy Coutable |
Failed jobs (6):
-
build-gdk-image Job ID:
19819511(retry with@gitlab-bot retry_job 19819511) -
cache-qa-gems-e2e-runners Job ID:
19819510(retry with@gitlab-bot retry_job 19819510) -
update-gitaly-binaries-cache-fips Job ID:
19816652(retry with@gitlab-bot retry_job 19816652) -
update-gitaly-binaries-cache Job ID:
19816651(retry with@gitlab-bot retry_job 19816651) -
setup-test-env-fips Job ID:
19816650(retry with@gitlab-bot retry_job 19816650) -
setup-test-env Job ID:
19816649(retry with@gitlab-bot retry_job 19816649)
General guidelines
A broken master branch on dev prevents monthly releases from being built.
Fixing the pipeline is a priority to prevent any delays in releases.
The process in the Broken master handbook guide can be referenced since much of that process also applies here.
Investigation
Be sure to fill the Timeline for this incident.
- If the failure is new, and looks like a potential flaky failure, you can retry the failing job.
Make sure to mention the retry in the
Timelineand leave a link to the retried job. - Search for similar master-broken issues in https://gitlab.com/gitlab-org/quality/engineering-productivity/master-broken-incidents/-/issues
- If one exists, ask the DRI of the master-broken issue to cherry-pick any resulting merge requests into the stable branch
If the merge request author or maintainer is not available, this can be escalated using the dev-on-call process in the #dev-escalation slack channel.
Pre-resolution
If you believe that there's an easy resolution by either:
- Reverting a particular merge request.
- Making a quick fix (for example, one line or a few similar simple changes in a few lines).
You can create a merge request, assign to any available maintainer, and ping people that were involved/related to the introduction of the failure.
Additionally, a message can be posted in
#backend_maintainersor#frontend_maintainersto get a maintainer take a look at the fix ASAP. - Cherry picking a change that was used to fix a similar master-broken issue.
Resolution
Add a comment to this issue describing how this incident could have been prevented earlier in the Merge Request pipeline (rather than the merge commit pipeline).