## Summary The `geo_proxy_fetch_ssh_to_primary` feature flag causes proxied SSH clones to hang indefinitely when the transfer takes longer than 60 seconds on slow network connections. ## Customer Issue A customer upgraded from v15 to v18.2.8 and experienced the following behavior: - **Initial Problem**: Proxied clones from secondary nodes failed with "Remote repository is unavailable" error - **Workaround**: Enabling `geo_proxy_fetch_ssh_to_primary` FF fixed smaller repository clones - **New Problem**: Clones taking >60 seconds would hang indefinitely at exactly the 60-second mark ## Environment Details - **Setup**: Omnibus with single Primary and two geographically-distant Secondary nodes - **Bandwidth**: Secondary has 1-10 MiB/s transfer rate to primary ## Impact The feature flag `geo_proxy_fetch_ssh_to_primary` may not be resilient to slow network conditions (or handle very large repos?) and cannot be safely released until this issue is resolved or denied as not reproducible. ## Required Actions - [ ] Try to reproduce the 60-second timeout behavior in a test environment ## References <details> <summary>Click to expand transcript of Slack thread</summary> >>> ## Slack Thread Transcript (some text redacted) **Support Engineer** - Yesterday at 3:55 PM Hi - I'm on a call with a customer who has just upgraded from v15 to v18.2.8 and they are seeing proxied clones from their secondary hanging at exactly 60 seconds. They have enabled the geo_proxy_fetch_ssh_to_primary FF which fixed clones of smaller repos, but something is still going wrong if it takes >60s. Is anyone aware of any issues that might be related? **Engineer 1** - Yesterday at 6:14 PM We suspect this is related to HTTP_READ_TIMEOUT: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/geo/git_ssh_proxy.rb?ref_type=heads#L6 Looks like the secondary node is terminating the request to the gitlab-workhorse in the primary node after 60 seconds, at least based on the gitlab-workhorse logs in the primary node: ``` {"correlation_id":"[CORRELATION_ID]","error":"upload pack failed: rpc error: code = Canceled desc = context canceled","level":"error","method":"POST","msg":"","time":"2025-12-21T18:01:56-08:00","uri":"[REPO_PATH]/ssh-upload-pack"} 44284:{"backend_id":"gitaly","body_limit":104857600,"content_type":"","correlation_id":"[CORRELATION_ID]","duration_ms":61134,"host":"[PRIMARY_NODE]","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","read_bytes":557,"referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"[IP_ADDRESS]","route":"^/.+\\.git/ssh-upload-pack\\z","route_id":"ssh_upload_pack","status":200,"system":"http","time":"2025-12-21T18:01:56-08:00","ttfb_ms":86,"uri":"[REPO_PATH]/ssh-upload-pack","user_agent":"GitLab-Shell","written_bytes":244599940} ``` **Engineer 1** - Yesterday at 6:21 PM After that, the git clone from the workstation to the secondary node will hang and run indefinitely. **Team Member** - Today at 5:19 AM Does this ring a bell with you at all? **Engineer 1** - Today at 5:26 AM BTW we disabled the feature flag again after fixing the synchronization issues to prevent the clone from redirecting. It looks like there are certain scenarios where the feature flag will cause a clone to hang indefinitely on slow networks. **Team Member** - Today at 5:29 AM Let me dig through our old issues/RFHs to see if I can find anything. Here's one that popped up that looks similar: https://gitlab.com/gitlab-com/request-for-help/-/issues/3332#note_2714007973 Could be also related to this old bug that we have which is blocked: https://gitlab.com/gitlab-org/gitlab/-/issues/454707 Also: https://gitlab.com/gitlab-org/gitlab/-/issues/545153 **Engineering Manager** - Today at 7:19 AM No, I don't recall that but it seems related to the issues you linked above. Pinging you since you are very familiar with the proxy code. **Senior Engineer** - Today at 9:55 AM I don't think HTTP_READ_TIMEOUT can be a factor if geo_proxy_fetch_ssh_to_primary is enabled, since that FF uses a new codepath which does not use the old Gitlab::Geo::GitSSHProxy code. Could you please clarify this? - What was the behavior of small repos before and after enabling the FF? - What was the behavior of large repos before and after enabling the FF? - What is the environment architecture? CNG? Omnibus? - Did they disable proxy request buffering when enabling geo_proxy_fetch_ssh_to_primary? - What components are between the GitLab Rails app and the user? NGINX, HAProxy, cloud load balancer? Please link an RFH and do we have GitLab SOSs from all nodes? **Support Engineer** - Today at 10:41 AM Hi, I was working on this yesterday and can provide some of the info you requested. After upgrading from v15 to v18.2.8 when cloning any repo from the secondary which was proxied they saw the following error: ``` $ cd /tmp && git clone -vv --depth 1 ssh://git@[SECONDARY_NODE]/[CUSTOMER_PROJECT]/[REPO].git -b production --single-branch --recurse-submodules [REPO] Cloning into '[REPO]'... remote: remote: This request to a Geo secondary node will be forwarded to the remote: Geo primary node: remote: remote: git@[PRIMARY_NODE]:[CUSTOMER_PROJECT]/[REPO].git remote: Server version is git/2.50.1.gl1-Linux Server supports shallow Server supports multi_ack_detailed Server supports no-done Server supports side-band-64k Server supports allow-tip-sha1-in-want Server supports allow-reachable-sha1-in-want Server supports thin-pack Server supports no-progress Server supports include-tag Server supports ofs-delta Server supports filter Server supports deepen-since Server supports deepen-not Server supports deepen-relative want [COMMIT_SHA] (refs/heads/production) remote: remote: ======================================================================== remote: remote: Remote repository is unavailable remote: remote: ======================================================================== remote: fatal: the remote end hung up unexpectedly ``` We asked them to enable the geo_proxy_fetch_ssh_to_primary FF and that seemed to fix it. However, they later discovered that when a proxied clone took >60s to complete it would hang at exactly the 60s mark. They have multiple secondaries and the transfer rate when cloning from one secondary is around 1-10MiB/s - not sure if this is down to bandwidth between the secondary and primary, or between client and secondary, or both. We observed that for the same repo the clone would succeed if the average bandwidth at the time was high enough to allow it to complete in <60s - otherwise it would hang (and not disconnect even after several minutes). We tried manually adding the nginx setting on the primary and secondary rails nodes even though this is the default in 18.2.8, to no effect: ``` nginx['request_buffering_off_path_regex'] = "/api/v\\d/jobs/\\d+/artifacts$|/import/gitlab_project$|\\.git/git-receive-pack$|\\.git/gitlab-lfs/objects|\\.git/info/lfs/objects/batch$|\\.git/git-upload-pack$|\\.git/ssh-upload-pack$" ``` They use Omnibus and have a single Primary and two secondary nodes. I'm not sure about what other network services are between the sites, but they were not changed as part of the upgrade. Which is not to rule out something like the absence of keepalives or a LB session timeout manifesting itself with the v18 Geo codebase and the changes to ssh/https proxy handling. FWIW, at one point we asked them to do a clone with ssh debugging and ssh keepalives enabled - the trace shows a continuous flow of data until the 60s point it just stops, but the keepalive packets keep getting responded to. **Support Engineer** - Today at 10:53 AM I've asked about the network setup. One thought I had is that maybe the root cause of the 60s problem is something in the configuration of a network service between the secondary and primary sites e.g. a load balancer session timeout that is now surfacing due to changes in how Geo proxying data transfer works in v18. I know when using an AWS ALB you need to increase the default 60s timeout to avoid issues with GitLab. Would you be able to provide details of all networking services/devices that are in use between the sites? And check whether there is perhaps a session timeout option set to 60 seconds? **Senior Engineer** - Today at 11:16 AM Thank you, I don't have a root cause yet. This reminds me of one customer with a large monorepo who had problems cloning because the Git client asks for info_refs and then asks for the blobs, and the monorepo took so long generating the response to info_refs that by the time the client asked for blobs, it was over a minute later. And both requests used the same JWT token. So if the token expired in a minute, then the primary rejected the request for the blobs since the token was already expired. (I don't know if this could be related yet. This was probably blocking syncing rather than user clones.) **Engineer 1** - Today at 20 minutes ago So the customer has two issues: 1. Clone which is redirected to the primary that's taking more than 60 seconds will indefinitely hang 2. Project synchronization not reaching 100% due to checksum mismatch errors and the record of the checksum is wrong for the primary During the call, we decided to fix the project synchronization first so we gave them a script that would recompute the hash in both the primary and secondary. However, the errors would resurface intermittently after fixing it. With this they are no longer redirecting in the clone. Even if it takes more than 60 seconds, it will resume. The customer also wanted to disable the feature flag. After a while, the bandwidth for the affected secondary appears to have improved and clones are now performing faster. With the faster bandwidth we observed the following behavior: - The checksum errors for project synchronization is no longer intermittently happening - After disabling the feature flag, if the clone redirected us to the primary and it took more than 60 seconds, it will still proceed I suspect the feature flag might not be resilient to network issues (e.g. if the connection of the secondary to the primary is slow and the connection of the client to the secondary is also slow). **Senior Engineer** - Today at 7 minutes ago Thank you for this interesting information. Regarding out-of-date primary checksums when bandwidth is slow, that indicates a race condition on the primary. I think there is already a bug issue about this. I'll link it if I find it. Regarding geo_proxy_fetch_ssh_to_primary hanging if over 60s, that will be a problem for us at some point since we do intend to release that change. The current design is unfortunately fundamentally flawed. I will open an issue blocking the release on this potential bug. At minimum we would need to attempt to reproduce the bug before releasing geo_proxy_fetch_ssh_to_primary. >>> </details>