Thursday 2024-01-25 11:25 UTC - `gitlab-org/gitlab` broken `16-7-stable-ee` with gitlab_git_test, graphql-verify, webpack-dev-server, feature-flags-usage, haml-lint ee, haml-lint, static-analysis 2/2, graphql-schema-dump, update-gitaly-binaries-cache, ...

gitlab-org/gitlab pipeline #1151061554 failed

Pipeline ID Branch Commit Merge request Source Duration Triggered by
1151061554 16-7-stable-ee Merge remote-tracking branch 'dev/16-7-stable-ee' into 16-7-stable-ee N/A push 45.8 minutes GitLab Release Tools Bot

Failed jobs (11):

  • gitlab_git_test Job ID: 6009364069 (retry with @gitlab-bot retry_job 6009364069)
  • graphql-verify Job ID: 6009363276 (retry with @gitlab-bot retry_job 6009363276)
  • webpack-dev-server Job ID: 6009363269 (retry with @gitlab-bot retry_job 6009363269)
  • feature-flags-usage Job ID: 6009363177 (retry with @gitlab-bot retry_job 6009363177)
  • haml-lint ee Job ID: 6009363168 (retry with @gitlab-bot retry_job 6009363168)
  • haml-lint Job ID: 6009363165 (retry with @gitlab-bot retry_job 6009363165)
  • static-analysis 2/2 Job ID: 6009363144 (retry with @gitlab-bot retry_job 6009363144)
  • graphql-schema-dump Job ID: 6009363102 (retry with @gitlab-bot retry_job 6009363102)
  • update-gitaly-binaries-cache Job ID: 6009363003 (retry with @gitlab-bot retry_job 6009363003)
  • setup-test-env Job ID: 6009363001 (retry with @gitlab-bot retry_job 6009363001)
  • no-jh-check Job ID: 6009362983 (retry with @gitlab-bot retry_job 6009362983)

General guidelines

A broken stable branch prevents patch releases from being built. Fixing the pipeline is a priority to prevent any delays in releases.

The process in the Broken master handbook guide can be referenced since much of that process also applies here.

Investigation

Be sure to fill the Timeline for this incident.

  1. If the failure is new, and looks like a potential flaky failure, you can retry the failing job. Make sure to mention the retry in the Timeline and leave a link to the retried job.
  2. Search for similar master-broken issues in https://gitlab.com/gitlab-org/quality/engineering-productivity/master-broken-incidents/-/issues
  3. If one exists, ask the DRI of the master-broken issue to cherry-pick any resulting merge requests into the stable branch

@gitlab-org/release/managers if the merge request author or maintainer is not available, this can be escalated using the dev-on-call process in the #dev-escalation slack channel.

Pre-resolution

If you believe that there's an easy resolution by either:

  • Reverting a particular merge request.
  • Making a quick fix (for example, one line or a few similar simple changes in a few lines). You can create a merge request, assign to any available maintainer, and ping people that were involved/related to the introduction of the failure. Additionally, a message can be posted in #backend_maintainers or #frontend_maintainers to get a maintainer take a look at the fix ASAP.
  • Cherry picking a change that was used to fix a similar master-broken issue.

Resolution

Add a comment to this issue describing how this incident could have been prevented earlier in the Merge Request pipeline (rather than the merge commit pipeline).