Make proxied Git pack request bodies replayable
What does this MR do and why?
Two fixes found during Phase 1 (canary) of the Cells SSH routing production rollout (gitlab-com/gl-infra/production#22944), both in the SSH-over-HTTP path.
1. Make proxied Git pack request bodies replayable
Go's HTTP transport retries a request on a new connection when the pooled one is lost before the server handled it (idle close, HTTP/2 GOAWAY, REFUSED_STREAM), but only if it can replay the body via Request.GetBody. The Cells and Geo pack requests (ssh-upload-pack, ssh-receive-pack, git-upload-pack, git-receive-pack) streamed SSH stdin through an io.Pipe, which cannot be replayed.
Under production load on gprd-cny (~50 SSH ops/s across 8 pods), each pod multiplexes its pack requests over a shared HTTP/2 connection to the edge. Whenever that connection was closed with requests in flight, those requests failed and the user's git fetch died:
Internal API unreachable: net/http: cannot rewind body after connection lossAbout 2.3% of routed pulls failed this way (241 in 3.5 minutes), in bursts of up to 7 per second, each after ~40 ms. Zonal clusters without routing showed none. The transparent in-memory requests (/allowed, authorized_keys: 22k in the same window, all 200) were unaffected, because Go can retry those.
Staging never showed it: at ~0.07 ops/s each request got a fresh connection, so there was never a queued request to lose.
Fix: git.ReplayBody, a pipe that records consumed bytes (up to 1 MiB) and can hand out fresh readers from byte 0. Pack requests built from it set GetBody, so the transport resends them.
- Streaming is preserved, so Git's multi-round negotiation still works; readers block for data, and the writer blocks when it runs 256 KiB ahead.
- Reader
Closeonly flags that reader but wakes a blockedRead. The transport closes the abandoned body while it may be waiting for stdin; without this the retry would deadlock. - Beyond 1 MiB,
GetBodyfails and the transport behaves exactly as before (no retry), so a large push never replays a truncated body. Abort()is deferred by callers so the stdin-forwarding goroutine cannot block forever after the request ends.
Replaying is safe: Go only replays when the server did not process the request (or sent GOAWAY naming a lower last-stream-id). In the worst case for a push, receive-pack rejects the second attempt as non-fast-forward.
2. Treat Topology Service NotFound as the default-host answer
NotFound is the Topology Service's answer for every claim not assigned to a cell, which is most projects today. It was already not retried, but it was logged at WARN as "classify failed after retries, falling back to default host" (~600/min on canary) and counted under gitlab_shell_topology_requests_total{status="fail"}, which made the fail rate useless as a rollback signal.
Fix: log it at debug and count it under a new label, status="not_found". fail now only reflects real errors. Latency is still observed since the round trip completed.
Dashboards/alerts: anything using
status!="ok"as an error rate should move tostatus="fail".
How to set up and validate locally
go test ./internal/gitlabnet/git ./internal/topology ./internal/command/githttp
make lintKey tests:
TestSSHUploadPackRetriesAfterGoAwaydrives a real HTTP/2 server whose first connection sendsGOAWAYafter the request HEADERS and first DATA frame, reproducing the production failure deterministically. With aReplayBodythe request is resent on a new connection and handled once; with anio.Pipeit fails withcannot retry err [...] after Request.Body was written.TestReplayBody*cover streaming, replay, the 1 MiB limit, trimming, backpressure, and the two unblocking cases (readerClose,Abort).TestResolveNotFoundIsNotAFailure/TestIsClaimNotFoundcover theNotFoundhandling and metric label.
Production validation plan
Build a custom image from this branch and run it on gprd-cny (same procedure as the staging validation in #874): Internal API unreachable with cannot rewind body should drop to ~0 while the pull rate stays the same, and topology_requests_total{status="fail"} should go to ~0 with not_found taking the volume.
Follow-ups (not in this MR)
- HTTP/2 health checks (
ReadIdleTimeout/PingTimeout) and a shorterIdleConnTimeoutwould reduce how often a dead pooled connection is picked in the first place. The replay fix handles the race regardless.
Related: gitlab-com/gl-infra/production#22944, #874, !1570 (merged)