Keep Cells SSH pack requests alive with body framing

What does this MR do and why?

Contributes to gitlab-com/gl-infra/production#22944

Rails MR: gitlab-org/gitlab!269512

Problem

Cloudflare resets Cells-routed SSH pack requests after about 15 seconds without request-body bytes. Client pauses during fetches and pushes can therefore fail with PROTOCOL_ERROR.

Solution

  • Negotiate v1 body framing from the /allowed response for routed sessions.
  • Frame request data in 64 KiB chunks and send keepalives after five idle seconds.
  • Mark framed requests, including the receive-pack advertisement, and log framing mode.

How it works

GitLab Shell is the encoding side of the framing protocol added in the Rails MR:

  1. Rails returns ssh_over_http_body_framing: "v1" from /allowed when cells_ssh_body_framing is enabled.
  2. Shell turns framing on only when the session is Cells-routed (cell_address is set) and the version is exactly v1. In all other cases, including Geo paths, Shell sends raw bodies as before.
  3. When framing is on, Shell sets Gitlab-Shell-Body-Framing: v1 on every ssh-upload-pack and ssh-receive-pack request, including the receive-pack advertisement request.
  4. framedBodyWriter wraps each write in data frames of up to 64 KiB. A timer sends a keepalive frame when no data has been written for five seconds.
  5. Workhorse removes the frame headers, drops keepalives, and forwards the original Git bytes to Gitaly.

Neither the Git client nor Gitaly needs to understand framing.

Frame format

The format matches Workhorse's sshBodyFrameReader:

[type: 1 byte][payload length: 4 bytes, big-endian][payload]
Frame Type Payload length Payload
Data 0x01 1–65,536 bytes Original Git bytes
Keepalive 0x02 0 None

For example, the advertisement request body 0000 becomes:

01 00 00 00 04  30 30 30 30
└── header ──┘  └─ "0000" ─┘

A paused fetch looks like:

[data header][want/have lines]
[keepalive]                  after 5 seconds idle
[keepalive]                  after another 5 seconds idle
[data header][done]

Facts

  • Data is written right away. Shell does not wait to fill a frame.
  • The replay body stores framed bytes, so retried requests resend valid frames.
  • Frame headers and keepalives count toward the replay limit. For upload-pack, the headers add about 10% for have/want lines, so negotiations near 1 MiB stop being retryable a bit sooner.
  • The Cells: using SSH-over-HTTP ... log line includes body_framing (v1 or none).
  • Disabling cells_ssh_body_framing turns framing off without a Shell rollback.

References

Depends on the Rails MR gitlab-org/gitlab!269512.

How to set up and validate locally

Deploy !269512 to every cell, release gitlab-shell and bump GITLAB_SHELL_VERSION, then enable cells_ssh_body_framing. Disabling the flag turns framing off.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasilii Iakliushin

Merge request reports

Loading
Loading