Support framed Cells SSH request bodies
What does this MR do and why?
Contributes to gitlab-com/gl-infra/production#22944
GitLab-Shell MR: gitlab-shell!1663 (merged)
Why we need it
Cells-routed SSH fetches can fail when the Git client pauses for more than about 15 seconds. The HTTP request stays open but silent, and the edge resets it. A valid pause after ls-refs, or while processing a received pack, can therefore fail the operation.
Transport-level keepalives can't fix this. The reset comes from Cloudflare's Workers runtime, which only sees request body bytes. HTTP/2 PING frames and TCP keepalives end at the edge and never reach the Worker, and these paths can't bypass the Worker route (details).
How it works
This adds optional framing to the HTTP request body between GitLab Shell and Workhorse:
- Rails advertises framing support through
/allowed, behindcells_ssh_body_framing. - Shell opts in for Cells-routed requests using
Gitlab-Shell-Body-Framing: v1. - Shell wraps Git data in frames and sends a keepalive every five idle seconds.
- Workhorse removes frame headers, discards keepalives, and forwards the original Git bytes to Gitaly.
Neither the Git client nor Gitaly needs to understand framing.
Frame format
Every frame starts with a five-byte binary header:
[type: 1 byte][payload length: 4 bytes, big-endian][payload]| Frame | Type | Payload length | Payload |
|---|---|---|---|
| Data | 0x01 |
1–65,536 bytes | Original Git bytes |
| Keepalive | 0x02 |
0 | None |
For example, the Git flush packet 0000 becomes these bytes, shown in hexadecimal:
01 00 00 00 04 30 30 30 30
└── header ──┘ └─ "0000" ─┘A keepalive is just:
02 00 00 00 00A conversation might look like:
[data header][Git bytes]
[keepalive] after 5 seconds idle
[keepalive] after another 5 seconds idle
[data header][more Git bytes]We cannot insert keepalives into an otherwise raw body. Git pack data is binary and could contain the same bytes as a keepalive. Framing tells Workhorse which bytes to preserve and which to discard.
Facts
- Each frame adds a five-byte header.
- Data streams immediately, without waiting to fill a frame.
- The current encoder reserves roughly 64 KiB of reusable memory per active framed request. That is memory usage, not transfer overhead.
- Clients that omit the framing header continue using raw bodies.
- Existing Geo paths do not opt in.
References
gitlab-com/gl-infra/production#22944
How to set up and validate locally
Enable cells_ssh_body_framing for an instance and verify that /allowed
includes ssh_over_http_body_framing: 'v1'.
MR acceptance checklist
Evaluated against the MR acceptance checklist.