Keep Cells SSH pack requests alive with body framing
What does this MR do and why?
Contributes to gitlab-com/gl-infra/production#22944
Rails MR: gitlab-org/gitlab!269512
Problem
Cloudflare resets Cells-routed SSH pack requests after about 15 seconds
without request-body bytes. Client pauses during fetches and pushes can
therefore fail with PROTOCOL_ERROR.
Solution
- Negotiate v1 body framing from the
/allowedresponse for routed sessions. - Frame request data in 64 KiB chunks and send keepalives after five idle seconds.
- Mark framed requests, including the receive-pack advertisement, and log framing mode.
How it works
GitLab Shell is the encoding side of the framing protocol added in the Rails MR:
- Rails returns
ssh_over_http_body_framing: "v1"from/allowedwhencells_ssh_body_framingis enabled. - Shell turns framing on only when the session is Cells-routed (
cell_addressis set) and the version is exactlyv1. In all other cases, including Geo paths, Shell sends raw bodies as before. - When framing is on, Shell sets
Gitlab-Shell-Body-Framing: v1on everyssh-upload-packandssh-receive-packrequest, including the receive-pack advertisement request. framedBodyWriterwraps each write in data frames of up to 64 KiB. A timer sends a keepalive frame when no data has been written for five seconds.- Workhorse removes the frame headers, drops keepalives, and forwards the original Git bytes to Gitaly.
Neither the Git client nor Gitaly needs to understand framing.
Frame format
The format matches Workhorse's sshBodyFrameReader:
[type: 1 byte][payload length: 4 bytes, big-endian][payload]| Frame | Type | Payload length | Payload |
|---|---|---|---|
| Data | 0x01 |
1–65,536 bytes | Original Git bytes |
| Keepalive | 0x02 |
0 | None |
For example, the advertisement request body 0000 becomes:
01 00 00 00 04 30 30 30 30
└── header ──┘ └─ "0000" ─┘A paused fetch looks like:
[data header][want/have lines]
[keepalive] after 5 seconds idle
[keepalive] after another 5 seconds idle
[data header][done]Facts
- Data is written right away. Shell does not wait to fill a frame.
- The replay body stores framed bytes, so retried requests resend valid frames.
- Frame headers and keepalives count toward the replay limit. For upload-pack, the headers add about 10% for have/want lines, so negotiations near 1 MiB stop being retryable a bit sooner.
- The
Cells: using SSH-over-HTTP ...log line includesbody_framing(v1ornone). - Disabling
cells_ssh_body_framingturns framing off without a Shell rollback.
References
Depends on the Rails MR gitlab-org/gitlab!269512.
How to set up and validate locally
Deploy !269512 to every cell, release gitlab-shell and bump
GITLAB_SHELL_VERSION, then enable cells_ssh_body_framing.
Disabling the flag turns framing off.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.