## Overview When a remote flow is running it CI, `git push` is very slow for `gitlab-org/gitlab` project. Duo Developer ran itself in CI and investigated the issue: - https://gitlab.com/gitlab-org/gitlab/-/automate/agent-sessions/3336346 - https://gitlab.com/gitlab-org/gitlab/-/jobs/13767378199 The conclusion is that `git push` is slow because it's performed from a shallow clone. Below are the details of the analysis. Gitaly logs for pushes: https://log.gprd.gitlab.net/app/r/s/zQ6um ## Git Push Benchmark Results & Root Cause Analysis I created a ~290KB file, committed it to branch `duo/feature/583059-benchmark-git-push`, and benchmarked `git push` with verbose output (lefthook was not installed, so no need to disable it). MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/230074 --- ### Benchmark Results | Attempt | Pack size | Duration | Result | |---------|-----------|----------|--------| | Push 1 (290KB file) | 77,813 bytes | **401 seconds** | HTTP 524 (Cloudflare timeout) | | Push 2 (small commit) | 1,388 bytes | **246 seconds** | HTTP 520 | | Push 3 (small commit) | 1,388 bytes | **400 seconds** | HTTP 524 | | Push 4 (small commit) | 747 bytes | **384 seconds** | HTTP 520 | | Direct Python push (no shallow) | 1,076 bytes | **~5 seconds** | Success | **Key observation:** `git pack-objects` itself is fast (~25ms). The entire delay is in waiting for the server response after the POST to `git-receive-pack`. --- ### Verbose Output (Push 1 — key lines) ``` performance: 0.042002679 s: git command: git pack-objects ... POST git-receive-pack (77813 bytes) error: RPC failed; HTTP 524 curl 22 The requested URL returned error: 524 send-pack: unexpected disconnect while reading sideband packet fatal: the remote end hung up unexpectedly performance: 400.434345545 s: git command: git send-pack --stateless-rpc ... ``` The POST was sent immediately, but the server took 400 seconds to respond, causing Cloudflare to time out (HTTP 524 = "A timeout occurred"). --- ### Root Cause: Shallow Clone + `shallow` pkt-line in push request The Duo agent environment clones the repository as a **shallow partial clone** (`blob:none` filter): ``` remote.origin.partialclonefilter=blob:none remote.origin.promisor=true ``` And the repo has a shallow boundary: ``` $ cat .git/shallow dff38ad5cde4cd10f12c7266f830a1f0164353bc $ git rev-parse --is-shallow-repository true ``` When `git push` runs from a shallow clone, it includes a `shallow` pkt-line in the `git-receive-pack` request: ``` 0035shallow dff38ad5cde4cd10f12c7266f830a1f0164353bc 00d0<new-sha> <old-sha> refs/heads/... report-status-v2 side-band-64k ... 0000 PACK... ``` This `shallow` pkt-line tells the Gitaly server that the client has a shallow clone. The server must then process this shallow boundary information before updating the ref. **For the `gitlab-org/gitlab` repository (which has 30,105+ refs and a very large history), this server-side processing takes 400+ seconds**, causing Cloudflare to time out the connection. **Proof:** When I sent the same push request directly (via Python socket) *without* the `shallow` pkt-line, the server responded in ~5 seconds and the push succeeded. --- ### Additional Context - **Proxy setup:** All HTTP traffic goes through `socat TCP-LISTEN:3128 -> UNIX-CONNECT:/tmp/.../claude-http-*.sock` (the Duo agent's HTTP proxy). The proxy itself is not the bottleneck — direct curl uploads of 1MB complete in ~300ms. - **Repository refs:** The `info/refs` response for `git-receive-pack` is **2.5MB** containing **30,105 refs**. This is downloaded in ~270ms (fast), but the server-side processing during push is the bottleneck. - **HTTP 524 vs 520:** HTTP 524 = Cloudflare timeout (server didn't respond within ~400s). HTTP 520 = Cloudflare "Unknown Error" (server returned an unexpected response, likely after a shorter timeout). - **Bundle URI:** `transfer.bundleuri=true` is configured but is not the cause of the slowness. --- ### Conclusion The slowness is a **server-side issue** specific to pushing from a shallow clone to the `gitlab-org/gitlab` repository. The Gitaly server's handling of the `shallow` pkt-line during `git-receive-pack` is extremely slow for this repository, likely due to its size (30,105+ refs, large history). This is not a client-side issue (lefthook, pack-objects, network, or proxy). The fix would need to be on the Gitaly/server side to optimize shallow clone push handling, or the Duo agent environment should use a non-shallow clone for pushing.