Sign Cells Shell JWT per request

Vladimir Glafirovrequested to merge
vg/cells-refresh-shell-jwt into main

What does this MR do and why?

Cells SSH-over-HTTP routing signs a Shell JWT (1-minute TTL) once, when the Cells git client is built, and reuses it for every request.

Since !1561 (merged), a Cells push makes two requests: the ref advertisement, then the actual receive-pack, which opens only after the client starts sending, i.e. after its pre-push hook finishes. With Git LFS the pre-push hook uploads objects and routinely takes minutes, so the second request carries an expired JWT. Rails rejects it and, by design, answers 404:

remote: ERROR: cells routing: receive-pack response: Not found

Observed on staging: a 524 MB LFS upload in the pre-push hook took ~155s, and the following receive-pack request failed with the stale token.

This MR adds an optional per-request HeaderFunc to git.Client and uses it in buildCellsGitClient, so the Shell JWT is signed when each request starts, for both push and pull. Pull was not affected in practice (its single request starts immediately after the client is built), but it now follows the same rule.

The JWT is only validated when a request starts, so long-running requests (for example clones longer than 60s) are unaffected by the TTL.

How to set up and validate locally

go test ./internal/gitlabnet/git ./internal/command/githttp ./internal/command/receivepack ./internal/command/uploadpack
make lint

New tests:

  • push: each of the two requests carries a separately signed token
  • pull: the token is signed at request time, not at client construction
  • signing failure: push and pull return the error without contacting the cell
  • git.Client: HeaderFunc runs on every request; on error the request body is closed

Staging validation plan

  • Push with a pre-push hook sleeping 90s
  • Single-command push with a fresh >= 500 MB LFS object to gitlab-com/gl-infra/cmbr
  • Clone longer than 60s (gitlab-org/gitlab) via Cells routing

Related: #864 (closed), gitlab-com/gl-infra/production#22944, follow-up to !1561 (merged)

🤖 Created with the help of OpenCode Duo

Merge request reports