## Summary Duo Agent Platform (DAP) sessions fail to start when Gitaly load-sheds during the `get_sources` / repository clone step. The failure surfaces as: ``` Session failed to start The job failed during the `get_sources` step when Git tried to fetch the repository, with the remote reporting a resource exhaustion error ("error resource exhausted, please try again later") that caused the pack transfer to abort with a `fatal: protocol error: bad pack header`. This indicates the Gitaly server-side git hook encountered an overload condition and rejected the clone, resulting in exit code 128. ``` This is the same failure mode tracked for CI pipelines in [gitlab-org/gitlab#627739](https://gitlab.com/gitlab-org/gitlab/-/work_items/627739) — Gitaly's pack-objects concurrency limiter rejects the request during an overload episode, truncating the pack stream and surfacing as `bad pack header` with exit code 128. DAP sessions perform a full repository clone at startup and are equally exposed to these load-shedding episodes. ## Example sessions Two DAP sessions from @jsoltani failed with this exact error: - [Agent session 7097957](https://gitlab.com/gitlab-org/gitlab/-/automate/agent-sessions/7097957) — first failure - [Agent session 7099069](https://gitlab.com/gitlab-org/gitlab/-/automate/agent-sessions/7099069) — second failure Both sessions reached `get_sources` and failed while fetching the repository during a Gitaly load-shedding episode. ## What is the current *bug* behavior? When Gitaly load-sheds during a DAP session startup, the `get_sources` step exhausts its retry attempts within a short window (all retries land inside the same load-shedding episode), the job exits with code 128, and the DAP session fails to start entirely — with no automatic recovery. The generated session job has **no job-level retry at all**: `Ci::Workloads::WorkloadDefinition#to_job_hash` (https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/ci/workloads/workload_definition.rb) emits no `retry:` key, so once the in-runner `get_sources` attempts are exhausted the failure is terminal. Exit code 128 surfaces to the runner as `script_failure`. ## What is the expected *correct* behavior? DAP sessions should be resilient to transient Gitaly load-shedding. Either: - The session should retry at the job level (re-queuing after a delay, giving the load-shedding episode time to resolve), or - The session should surface a user-friendly error with a "retry" option rather than a hard failure. ## Relationship to existing issues - **[gitlab-org/gitlab#627739](https://gitlab.com/gitlab-org/gitlab/-/work_items/627739)** — tracks the same problem for `clone-gitlab-repo` in CI pipelines, proposing a targeted `script_failure` retry on that job. The fix there does not cover DAP sessions. - **[gitlab-org/packhorse#78](https://gitlab.com/gitlab-org/packhorse/-/issues/78)** — ~~tracks the underlying Gitaly-side cause~~. **Correction:** this issue tracked a *different* failure mode — a packhorse fetch-cache key bug that produced `fatal: expected 'packfile', received 'acknowledgments'`. It was closed on 2026-08-13 via [packhorse!261](https://gitlab.com/gitlab-org/packhorse/-/merge_requests/261), and the `resource exhausted` failures continued afterwards because they are unrelated. - **Actual Gitaly-side cause:** the `resource exhausted, please try again later` response is Gitaly's concurrency limiter (pack-objects hook / adaptive limiter) *intentionally shedding load* during overload episodes. That is by-design protection against saturation, not a bug with a single fix — the episodes follow load bursts (see hourly counts in #627739). Ongoing Gitaly-side hardening is tracked in [gitlab-org/gitaly#7036](https://gitlab.com/gitlab-org/gitaly/-/work_items/7036) (adaptive-limiter load shedding). Because load-shedding is expected behavior during bursts, clients like DAP need to tolerate it — which is what this issue is about. ## Possible fixes 1. **Job-level retry for DAP session startup jobs** — add `retry:` support to `Ci::Workloads::WorkloadDefinition` (opt-in, since workloads are a shared abstraction) and set `retry: { max: 1, when: [script_failure] }` from `Ai::DuoWorkflows::StartWorkflowService`. A re-queued job lands minutes later, outside the load-shedding window. Notes: - `max: 1` rather than 2: unlike `clone-gitlab-repo`, the session job's `script:` runs the entire agent (up to a 2h timeout), so `script_failure` also covers genuine agent/setup crashes — the retry budget should stay small. Each retry is also a fresh clone against an already-shedding Gitaly. - Session state is safe under retry: `Ci::Workloads::UpdateWorkloadStatusEventWorker` marks the workload finished on `PipelineFinishedEvent`, so a retried job keeps the pipeline running and the session is not prematurely marked failed. 2. **Graceful retry UX** — surface the transient error to the user with a one-click retry rather than a terminal failure state. Worth doing regardless of fix 1. 3. ~~Shallow clone or incremental fetch~~ — mostly not applicable: sessions already clone with `--filter=blob:none` (partial clone) via `GIT_FETCH_EXTRA_FLAGS`, and a truly shallow clone conflicts with agents needing repository history/branches.