Make proxied Git pack request bodies replayable

What does this MR do and why?

Two fixes found during Phase 1 (canary) of the Cells SSH routing production rollout (gitlab-com/gl-infra/production#22944), both in the SSH-over-HTTP path.

1. Make proxied Git pack request bodies replayable

Go's HTTP transport retries a request on a new connection when the pooled one is lost before the server handled it (idle close, HTTP/2 GOAWAY, REFUSED_STREAM), but only if it can replay the body via Request.GetBody. The Cells and Geo pack requests (ssh-upload-pack, ssh-receive-pack, git-upload-pack, git-receive-pack) streamed SSH stdin through an io.Pipe, which cannot be replayed.

Under production load on gprd-cny (~50 SSH ops/s across 8 pods), each pod multiplexes its pack requests over a shared HTTP/2 connection to the edge. Whenever that connection was closed with requests in flight, those requests failed and the user's git fetch died:

Internal API unreachable: net/http: cannot rewind body after connection loss

About 2.3% of routed pulls failed this way (241 in 3.5 minutes), in bursts of up to 7 per second, each after ~40 ms. Zonal clusters without routing showed none. The transparent in-memory requests (/allowed, authorized_keys: 22k in the same window, all 200) were unaffected, because Go can retry those.

Staging never showed it: at ~0.07 ops/s each request got a fresh connection, so there was never a queued request to lose.

Fix: git.ReplayBody, a pipe that records consumed bytes (up to 1 MiB) and can hand out fresh readers from byte 0. Pack requests built from it set GetBody, so the transport resends them.

  • Streaming is preserved, so Git's multi-round negotiation still works; readers block for data, and the writer blocks when it runs 256 KiB ahead.
  • Reader Close only flags that reader but wakes a blocked Read. The transport closes the abandoned body while it may be waiting for stdin; without this the retry would deadlock.
  • Beyond 1 MiB, GetBody fails and the transport behaves exactly as before (no retry), so a large push never replays a truncated body.
  • Abort() is deferred by callers so the stdin-forwarding goroutine cannot block forever after the request ends.

Replaying is safe: Go only replays when the server did not process the request (or sent GOAWAY naming a lower last-stream-id). In the worst case for a push, receive-pack rejects the second attempt as non-fast-forward.

2. Treat Topology Service NotFound as the default-host answer

NotFound is the Topology Service's answer for every claim not assigned to a cell, which is most projects today. It was already not retried, but it was logged at WARN as "classify failed after retries, falling back to default host" (~600/min on canary) and counted under gitlab_shell_topology_requests_total{status="fail"}, which made the fail rate useless as a rollback signal.

Fix: log it at debug and count it under a new label, status="not_found". fail now only reflects real errors. Latency is still observed since the round trip completed.

Dashboards/alerts: anything using status!="ok" as an error rate should move to status="fail".

How to set up and validate locally

go test ./internal/gitlabnet/git ./internal/topology ./internal/command/githttp
make lint

Key tests:

  • TestSSHUploadPackRetriesAfterGoAway drives a real HTTP/2 server whose first connection sends GOAWAY after the request HEADERS and first DATA frame, reproducing the production failure deterministically. With a ReplayBody the request is resent on a new connection and handled once; with an io.Pipe it fails with cannot retry err [...] after Request.Body was written.
  • TestReplayBody* cover streaming, replay, the 1 MiB limit, trimming, backpressure, and the two unblocking cases (reader Close, Abort).
  • TestResolveNotFoundIsNotAFailure / TestIsClaimNotFound cover the NotFound handling and metric label.

Production validation plan

Build a custom image from this branch and run it on gprd-cny (same procedure as the staging validation in #874): Internal API unreachable with cannot rewind body should drop to ~0 while the pull rate stays the same, and topology_requests_total{status="fail"} should go to ~0 with not_found taking the volume.

Follow-ups (not in this MR)

  • HTTP/2 health checks (ReadIdleTimeout/PingTimeout) and a shorter IdleConnTimeout would reduce how often a dead pooled connection is picked in the first place. The replay fix handles the race regardless.

Related: gitlab-com/gl-infra/production#22944, #874, !1570 (merged)


🤖 Created with the help of OpenCode Duo

Merge request reports

Loading
Loading