<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Collaborate/take over this issue](https://contributors.gitlab.com/manage-issue?action=work&projectId=278964&issueIid=627739) </details> <!--IssueSummary end--> ## Summary `clone-gitlab-repo` is the single point where `gitlab-org/gitlab` CI clones from Gitaly. When Gitaly load-sheds, that job exhausts its three `get_sources` attempts and fails with `exit code 128` — and because the runner reports the failure as `script_failure`, our `.default-retry` `when:` list does **not** cover it, so the job gets no job-level retry and the whole pipeline dies. Proposal: add a targeted `retry` to `clone-gitlab-repo` (and only that job) covering `script_failure`. ## Why `clone-gitlab-repo` is a choke point `.gitlab-ci.yml` sets `CI_FETCH_REPO_GIT_STRATEGY: "none"`, and `.repo-from-artifacts` wires every other job to take the repo from an artifact: ```yaml # .gitlab/ci/global.gitlab-ci.yml .repo-from-artifacts: variables: GIT_STRATEGY: "${CI_FETCH_REPO_GIT_STRATEGY}" needs: - clone-gitlab-repo ``` So one job clones and everything else depends on its artifact. A single `bad pack header` there fails the pipeline, and downstream jobs report `missing_dependency_failure` rather than a git error — which makes these harder to attribute than they should be. ## The gap `.gitlab/ci/global.gitlab-ci.yml`: ```yaml .default-retry: retry: max: 2 when: - api_failure - data_integrity_failure - runner_system_failure - scheduler_failure - stuck_or_timeout_failure - unknown_failure ``` `clone-gitlab-repo` inherits this via `.absolutely-predictive-job` → `.predictive-job` → `.default-retry`. But `script_failure` is absent from the list. Every one of the last 12 `git_issues_network_error` jobs on `gitlab-org/gitlab` in the analytics ClickHouse carries `failure_reason = script_failure`. Confirmed against three job traces from 2026-09-04 12:20–12:31 UTC (`16306740250`, `16306630037`, `16306908779`): ``` remote: error executing git hook remote: error resource exhausted, please try again later remote: aborting due to possible repository corruption on the remote side. fatal: protocol error: bad pack header ... ERROR: Job failed: exit code 128 ``` ## What the in-job retries already do `GET_SOURCES_ATTEMPTS: "3"` (`.gitlab-ci.yml`) is handled runner-side in `common/build.go` (`attemptExecuteStage`, wrapping only `BuildStageGetSources`). Exponential backoff between those attempts is already active — `FF_USE_EXPONENTIAL_BACKOFF_STAGE_RETRY` defaults to `true`, 5s→5min (gitlab-org/gitlab#422510, closed complete). The observed gaps were 5s then 10s. The problem is not backoff spacing. It's that all three attempts land inside a ~3 minute window while a load-shedding episode lasts far longer, and then there is no outer retry: ``` 12:27:49 get_sources → 12:28:50 fatal (61s) 12:28:55 Deleting tracked and untracked files... 12:28:55 get_sources → 12:29:56 fatal (61s) 12:30:06 get_sources → 12:31:07 fatal (61s) → exit 128, pipeline fails ``` A job-level retry re-queues the job minutes later, which is the timescale these episodes actually resolve on. ## Proposal ```yaml clone-gitlab-repo: extends: - .absolutely-predictive-job - .setup:rules:clone-gitlab-repo retry: max: 2 when: - api_failure - data_integrity_failure - runner_system_failure - scheduler_failure - stuck_or_timeout_failure - unknown_failure - script_failure # get_sources exhaustion surfaces as script_failure ``` Scoping `script_failure` to this one job matters: adding it to `.default-retry` globally would retry genuine test and lint failures, which we do not want. ### Trade-offs - `clone-gitlab-repo` has no `script:` of its own beyond a three-line git-config cleanup, so `script_failure` here is almost always `get_sources`. The blast radius of a false retry is small. - It does mean up to 9 clone attempts (3 job runs × 3 `GET_SOURCES_ATTEMPTS`) in the worst case. If that is judged too aggressive against an already-shedding Gitaly, the alternative is `retry: max: 1` plus lowering `GET_SOURCES_ATTEMPTS` for this job, keeping total attempts at 6 but spread over a longer window. - Every attempt is a **fresh full clone** (`GIT_STRATEGY: clone` + the worktree wipe between attempts), so each one costs a full ~61s and real Gitaly load. Spacing matters more than count. ## Context This is the `git_issues_network_error` bucket in the weekly master CI failure reports (gitlab-org/quality/analytics/team). It is bursty rather than steady — hourly counts on `gitlab-org/gitlab` for 2026-09-03: `09:00 → 53`, `12:00 → 27`, `14:00 → 52`, `16:00 → 31`, **`17:00 → 108`**, `19:00 → 12`. The underlying Gitaly-side cause is tracked separately (see gitlab-org/packhorse#78); this issue is only about our CI config not surviving those episodes. Note also that [gitlab-runner#39593](https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/39593) (the `rm` flood that used to blow the 4 MiB job-log cap and hide these failures under `logs_too_big_to_analyze`) was fixed in gitlab-runner v19.3.0, and the shared fleet is now on 19.4.0. These failures are visible in traces again.