Geo SSH proxy fails with SSH certificate authentication

Summary

Currently, Geo's SSH proxying is incompatible with SSH certificate authentication. It has been so since the time when both SSH proxying and SSH certificate authentication existed, so technically it was an unimplemented feature addition, and now it's a bug.

More details according to Duo

When using SSH certificates for authentication (via AuthorizedPrincipalsCommand), Geo SSH proxy operations fail with "Geo push user is invalid" error. This occurs because Geo::PushUser only supports the key-{ID} format for gl_id, not the username-{KEY_ID} format used by SSH certificates.

Click to expand the flow in the codebase that leads to the error
  1. Secondary's GeoGitAccess#geo_custom_ssh_action creates payload with gl_id in username-{KEY_ID} format (from certificate)
  2. Primary's GitHttpController#ssh_upload_pack handles the request
  3. access_actor method calls Geo::PushUser.new(geo_gl_id)
  4. Geo::PushUser#user calls identify_using_ssh_key(gl_id)
  5. Gitlab::Identifier#identify_using_ssh_key expects key-{ID} format and queries User.find_by_ssh_key_id(key_id)
  6. This fails for SSH certificate users because there's no record in the keys table
  7. Returns nil and access_actor raises 'Geo push user is invalid.'
Click to expand details on `gl_id` formats

The Different gl_id Formats and Their Origins

The gl_id format depends on how the user authenticates via SSH:

1. Traditional SSH Keys (key-{ID}): When a user authenticates with an SSH key stored in GitLab's database, the AuthorizedKeysCommand returns:

command="gitlab-shell upload-pack key_id=1"

GitLab Shell then uses key-{ID} as the gl_id, and Gitlab::Identifier#identify_using_ssh_key looks up the user via User.find_by_ssh_key_id(key_id).

2. SSH Certificates (username-{USERNAME}): When using SSH certificate authentication via AuthorizedPrincipalsCommand, the command outputs:

command="/opt/gitlab/.../gitlab-shell username-{KEY_ID}",... {PRINCIPAL}

Here, {KEY_ID} is the username extracted from the certificate (not a database ID). There's no database lookup needed because the username comes directly from the cryptographically-signed certificate.

3. HTTP Push (user-{ID}): For HTTP Git operations, the user is already authenticated via the web session/API, so gl_id uses user-{ID} with the numeric user ID.

Why Gitlab::Identifier#identify Only Handles user-* and key-*

The identify method was designed for the internal API flow where:

  • user-{ID} comes from HTTP authentication (user already identified)
  • key-{ID} comes from traditional SSH key lookup (key exists in database)

The username-{USERNAME} format was added later for SSH certificates and bypasses the need for database lookup during initial authentication. The username is trusted because it's embedded in a certificate signed by a trusted CA.

The Bug

Geo::PushUser directly calls identify_using_ssh_key(gl_id), which assumes key-{ID} format. When Geo proxies an SSH request from a certificate-authenticated user, the gl_id is username-{USERNAME}, which fails because there's no SSH key record to look up.

Steps to reproduce

  1. Configure GitLab with SSH certificate authentication using AuthorizedPrincipalsCommand
  2. Disable traditional SSH key uploads (common security policy)
  3. Set up Geo with a secondary site configured as read-only (GEO_SECONDARY_PROXY=0)
  4. Attempt to clone/pull a repository via SSH from the secondary when the repository is out-of-date

What is the current bug behavior?

When the secondary proxies an SSH request to the primary for an out-of-date repository:

Error message:

remote: Geo push user is invalid.
fatal: Could not read from remote repository.

What is the expected correct behavior?

Geo::PushUser should support the username-{KEY_ID} format used by SSH certificate authentication, allowing users to be identified by username lookup.

Potential solutions

Click to expand an idea from Duo

This is Duo's idea which I have not vetted as of 13 Jan 2026. I added the security items.

  1. Update Geo::PushUser#user to use the generic identify() method
  2. Extend Gitlab::Identifier#identify to handle the username-{KEY_ID} pattern
  3. Add an identify_using_username method to look up users by username
  4. Consider security implications
  5. Get app sec review

Relevant code

ee/app/models/geo/push_user.rb:

def user
  @user ||= identify_using_ssh_key(gl_id)
end

lib/gitlab/identifier.rb already has a generic identify() method that handles multiple formats, but it does not include username-*:

def identify(identifier)
  case identifier
  when /\Auser-\d+\Z/
    identify_using_user(identifier)
  when /\Akey-\d+\Z/
    identify_using_ssh_key(identifier)
  end
end

Possible Workarounds

  1. If you do not use CNH/CNG, you might try enabling Feature.enable(:geo_proxy_fetch_ssh_to_primary) && Feature.enable(:geo_proxy_push_ssh_to_primary) since I believe the unreleased change in flow does not run the offending code. This possible workaround has not been verified empirically, as of 13 Jan 2026. References [Feature flag] Rollout of `geo_proxy_fetch_ssh_... (#466045), [Feature flag] Rollout of `geo_proxy_push_ssh_t... (#466057). (This is not working for CNH/CNG yet because you cannot disable proxy request buffering by path in the NGINX ingress. You could enable the FFs if you disable proxy request buffering completely. This is low risk if your env is not on the public internet.)

Related issues

Customer request for help: https://gitlab.com/gitlab-com/request-for-help/-/issues/4040

Edited by Michael Kozono