Fix Workhorse hang when rejecting Geo SSH git requests
What does this MR do and why?
With Geo direct-to-primary SSH proxying enabled (geo_proxy_push_ssh_to_primary, now on by default), a git push over SSH to a Geo secondary by a user with no access to the project hangs instead of failing immediately. The user expects the usual "The project you were looking for could not be found or you don't have permission to view it." error, but the client just sits there until they cancel it.
This MR fixes a bug in Workhorse, on the primary, so the rejection is sent right away under HTTP/1.1 protocol.
Why the rejection never arrived
gitlab-shell on the secondary sends one streaming POST /<repo>.git/ssh-receive-pack (or ssh-upload-pack) request to the primary, with the request body wired to the SSH client's stdin. Git sends nothing until it receives the server's ref advertisement, so the request body stays open and silent until the server responds.
Workhorse runs a preauth check against Rails before handling the request. When Rails rejects the request (a 404 here), Workhorse tries to pass that response back through passResponseBack in workhorse/internal/api/api.go. Go's net/http server, before it writes response headers, tries to read and discard up to 256 KB of any unread request body, unless the handler has already enabled full duplex. Workhorse only enabled full duplex inside handleSSHUploadPack and handleSSHReceivePack, which run after preauth succeeds. On the rejection path, full duplex was never enabled, so the server waited for request-body bytes that never arrive, and the 404 response was never sent.
Primaries behind the Linux package's bundled NGINX do not hit this. That NGINX sends Connection: close to Workhorse, and Go's server skips the drain for a connection it is about to close. Envoy Gateway keeps its upstream connections open, so Cloud Native Hybrid primaries hit the hang.
The fix in workhorse/internal/git/ssh.go adds a withFullDuplex wrapper around the preauth handler for the ssh-upload-pack and ssh-receive-pack routes, so full duplex is enabled before preauth runs. The now-redundant EnableFullDuplex calls inside the two inner handlers are removed. If enabling full duplex fails, the request still fails with a 500, the same as before.
This MR also adds a regression test, TestSSHPreAuthorizeRejection_OpenRequestBody, in workhorse/internal/git/ssh_test.go, and replaces two now-invalid full-duplex tests with TestWithFullDuplex_Unsupported. go test ./internal/git/ passes, and golangci-lint 2.11.4 reports nothing for the package.
References
Related to #604776 (closed).
Note that the original report used an HTTPS primary, so this MR does not close it. On the path this MR fixes, gitlab-shell already passes the error to the client within a millisecond of receiving it; the delay comes from Workhorse not sending the 404 instead.
Screenshots or screen recordings
This is a backend-only fix, validated end-to-end on a Cloud Native Hybrid 3k Geo pair (GitLab v19.4.1, Envoy Gateway, primary URL over plain HTTP, so HTTP/1.1). The primary's gitlab-webservice-default Workhorse container ran v19.4.1 with only this patch applied.
| Scenario | Before | After |
|---|---|---|
| Push by a user with no access | Hangs; secondary gitlab-shell logs the 404 from POST .../ssh-receive-pack with duration_s: 45.8, which is when the push was cancelled with Ctrl-C |
Fails immediately with "could not be found or you don't have permission"; gitlab-shell logs the 404 with duration_s: 0.708 |
| Push by a user with access | Succeeds | Succeeds |
How to set up and validate locally
- From the
workhorsedirectory, rungo test ./internal/git/ -run 'TestSSHPreAuthorizeRejection_OpenRequestBody|TestWithFullDuplex_Unsupported' -count=1. - To see the test catch the bug, change
SSHUploadPackandSSHReceivePackback to returningrepoPreAuthorizeHandler(...)withoutwithFullDuplex, and rerun: both subtests fail after 5 seconds. - Optional end-to-end with a Geo pair: with
geo_proxy_push_ssh_to_primaryenabled, push over SSH to the secondary as a user who cannot see the project, and confirm the error appears immediately; then push as a user with access and confirm it succeeds.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.