Geo: POC: Proxy Git push over SSH via SSH
Problem
See &8819
Proposal
The high-level idea is that when a client does a Git push over SSH to a secondary site, the secondary site should proxy the SSH connection to the primary site. Hopefully with little to no modification. For example:
sequenceDiagram
participant Git on client
participant GitLab SSHD
participant Rails
participant Primary Site GitLab SSHD
Note left of Git on client: git fetch
Git on client->>+GitLab SSHD: ssh git fetch-pack request
GitLab SSHD->>+Rails: GET /internal/api/authorized_keys?key=AAAA...
Note right of Rails: Lookup key ID
Rails-->>-GitLab SSHD: 200 OK, command="gitlab-shell upload-pack key_id=1"
GitLab SSHD->>+Rails: GET /internal/api/allowed?action=upload_pack&key_id=1
Note right of Rails: Auth check
Rails-->>-GitLab SSHD: 200 OK, { forward: ... }
GitLab SSHD->>+Primary Site GitLab SSHD: ssh git fetch-pack request
Note over Git on client,Primary Site GitLab SSHD: Bidirectional communication between Git client and server
Primary Site GitLab SSHD-->>-GitLab SSHD: ssh git fetch-pack response
GitLab SSHD-->>-Git on client: ssh git fetch-pack response
As shown, the only component that seems to need modification is gitlab-sshd
. Instead of the final ReceivePack Gitaly call, gitlab-sshd
should connect to the primary site's gitlab-sshd
, repeat the command, and forward the streams in both directions.
POC acceptance criteria
- The user does Git push over SSH against the secondary
- The Git push should be handled by the primary
- The Git push should succeed
If it makes the iteration smaller, then improving Git pull can be deferred, but leave that door open.
If it makes the iteration smaller, then it is acceptable to make this functionality depend on gitlab-sshd
, since we expect gitlab-sshd
to replace the legacy SSH implementation at some point.
We must use a feature flag for this critical functionality.
Start with a proof of concept because this is only an idea at the moment, and there may be hidden obstacles.
Previous discussion
The Geo primary site already does have an SSH server. Git push over SSH doesn't need to be fiddled with by the Geo secondary site at all, it just needs to get proxied. The harder one is Git pulls need to consult with Rails to conditionally proxy. But it seems like maybe there is another idea here. Maybe with GitLab SSHD?
yes using SSH would fit the bill well.
Are customers willing to open up SSH in firewalls etc. so that the Geo secondary can make an SSH connection to the Geo primary
We'd need to add to https://docs.gitlab.com/ee/administration/geo/#firewall-rules, and we have good reason to make this change. IIUC, if they use Git over SSH, then they already have the port open to all users.
Streamline the distribution of public keys: both the client key the secondary pushes with, and the (signature of) the host key the primary answers with
It happens that we already instruct to copy the primary host keys to secondaries. I'm not sure how we'd handle the client key, but in the worst case, we do already have API endpoints on both the primary and secondary side which are used by other Geo sites, so we could add more if needed.
From what I remember, early on Geo used SSH for all Git replication but then they moved away from it in favor of HTTPS. I wasn't involved closely enough to know why we did that.
Relying on HTTPS was easier for some practical reasons:
- Firewall
- ssh-keys
Instead of generating a new ssh-key and having to update how (at time) gitlab-shell identifies a user, we could more easily create the alternative route via HTTPS, which did not require rearchitecting gitlab-shell.
At time some users would no5 be super happy to have to enable ssh just so replication can happen (think non public non internet available instances)
There is nothing good or bad other than those two practical things
I expect the firewall problem is still annoying. But what is different now is that SSH push is optional. Back then, getting SSH through the firewall from secondary to primary was a hard requirement for getting Geo working at all. SSH-through-firewall was causing pain for everybody. Now it would only be for Geo installations that want to support SSH push to secondary.
What has also changed is how flexible we can be when developing SSH features. It should be easier to implement special behavior in gitlab-sshd because we don't have to know what OpenSSH server version the user has.