Do not retry 429 responses from /authorized_certs

Fred Reininkrequested to merge
skip-retries-authorixed-certs into main

Human written/readable tl;dr

As part of adding audit events on failed certificate authorization, I'll be adding a rate limit to the /authorized_certs endpoint (rails MR: gitlab!263054 (merged)). The problem is that Shell will automatically retry 429s, which doesn't work with this endpoint as the SSH public-key callback runs with a 10-second deadline. Additionally, the error message returned to the user by Shell when denied by the rate limit would be a generic Permission denied (publickey) error, even if their certificate was valid. This MR also returns a banner so the user sees:

GitLab: Too many requests. Try again later.
git@gitlab.example.com: Permission denied (publickey).

What does this MR do?

Adds a per-request opt-out for retrying 429 Too Many Requests responses in the internal API client, and uses it for /authorized_certs, so a rate-limited SSH certificate resolution is rejected immediately as a policy denial instead of being retried into a timeout.

Because SSH gives the client no reason for a failed public-key attempt, gitlab-sshd also sends an auth banner on 429, so a user with a valid certificate can tell a rate limit from a bad key.

This is MR 2 of the plan in https://gitlab.com/gitlab-org/gitlab/-/work_items/611349#note_4015768715.

Why

Rails is adding a per-client-IP rate limit on /internal/authorized_certs (MR 3 of the plan) that answers throttled requests with 429 and Retry-After: 60.

go-retryablehttp retries 429 by default, and its backoff honors Retry-After without capping it at RetryWaitMax. The SSH public-key callback runs under a 10-second deadline, so today gitlab-sshd would:

  1. Sleep toward 60 seconds and hit the 10-second deadline.
  2. Fail with a transport error, reported as Internal API unreachable with System: true, which counts as an infrastructure failure in the error SLIs.

How to set up and validate locally

These steps exercise the full path: an OpenSSH client → gitlab-sshd → GET /internal/authorized_certs → 429 → banner. They need the Rails MR !263054 checked out in the same GDK.

Setup

  1. Build this branch and restart gitlab-sshd (GDK's sshd service runs gitlab-shell/bin/gitlab-sshd, which is not rebuilt automatically):

    cd gitlab-shell && make build && gdk restart sshd

  2. In the Rails checkout, temporarily lower the limit so it can be tripped by hand. In lib/gitlab/application_rate_limiter/labkit_adapter/supported_rate_limits.rb, change the ssh_certificate_authentication rule from limit: 300 to limit: 1.

  3. Remove the loopback exemption. When rack_attack.git_basic_auth.ip_whitelist is unset, Rails defaults it to ["127.0.0.1"], which exempts every local SSH client from the limit. In gitlab/config/gitlab.yml, under rack_attack.git_basic_auth, add:

    ip_whitelist: []

  4. Restart Rails and enable the flag:

    gdk restart rails-web
cd gitlab && bin/rails runner 'Feature.enable(:ssh_certificate_authentication_rate_limit)'

  5. Create a trusted CA and an untrusted CA, each with a user certificate. The principal (-n) must be the SSH login user git; the GitLab username goes in the key ID (-I).

    mkdir -p /tmp/sshqa && cd /tmp/sshqa
for n in good bad; do
  ssh-keygen -q -t ed25519 -f ${n}_ca   -N '' -C ${n}_ca
  ssh-keygen -q -t ed25519 -f ${n}_user -N ''
  ssh-keygen -q -s ${n}_ca -I root -n git -V +2h ${n}_user.pub
done

  6. Register only good_ca.pub as a trusted instance CA, either in Admin → Settings → SSH certificates or:

    cd gitlab && bin/rails runner 'InstanceSshCertificate.create!(title: "QA", key: File.read("/tmp/sshqa/good_ca.pub").strip)'

  7. Define a helper. IdentityAgent=none and IdentitiesOnly=yes stop the client from also offering agent keys (which take the /authorized_keys path and are not rate limited).

    sshcert() { ssh -T -p 2222 -o IdentitiesOnly=yes -o IdentityAgent=none \
  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
  -i /tmp/sshqa/${1}_user -o CertificateFile=/tmp/sshqa/${1}_user-cert.pub git@127.0.0.1; }

Validate

  1. Happy path. sshcert good prints Welcome to GitLab, @root! with no banner.

  2. Rejections below the limit are not announced. Run sshcert bad twice. Each prints only Permission denied (publickey). These two failures put 127.0.0.1 over the limit of 1.

  3. Throttled untrusted certificate. time sshcert bad prints the banner before the denial and returns in well under a second:

    GitLab: Too many requests. Try again later.
git@127.0.0.1: Permission denied (publickey).

    A client that still retried 429 would hang for ~10 s (honouring Retry-After: 60 until the publickey callback deadline) and then fail without a banner.

  4. Throttled trusted certificate. Within the same minute, time sshcert good prints the same banner and also returns immediately. A source over the limit is blocked for everything.

  5. Logs. gdk tail sshd shows, for each throttled login, one Internal API returned a client error entry with status: 429 followed by certificate lookup rate limited by internal API. No retries. Rails log/auth_json.log has an Application_Rate_Limiter_Request entry for ssh_certificate_authentication.

  6. Git operation under throttle. Still within the minute:

    GIT_SSH_COMMAND="ssh -p 2222 -o IdentitiesOnly=yes -o IdentityAgent=none -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /tmp/sshqa/good_user -o CertificateFile=/tmp/sshqa/good_user-cert.pub" \
  git ls-remote ssh://git@127.0.0.1:2222/gitlab-org/gitlab-test.git HEAD

    Output is the banner, Permission denied (publickey), then fatal: Could not read from remote repository.

  7. Recovery. Wait 60 s. sshcert good succeeds again and the git ls-remote above returns a SHA.

  8. Flag off. Disable the flag, then wait about a minute or restart rails-web; the running process caches the flag value briefly.

    cd gitlab && bin/rails runner 'Feature.disable(:ssh_certificate_authentication_rate_limit)'

    Run sshcert bad four times: all four print only Permission denied (publickey), no banner. sshcert good succeeds.

Clean up

  • Revert the limit: 1 and ip_whitelist: [] edits, then gdk restart rails-web.
  • Delete the QA instance CA from Admin → Settings → SSH certificates.
  • rm -rf /tmp/sshqa

Rollout

The Rails rate limit (MR 3) is behind a feature flag that should only be enabled after this and !1578 (merged) ship in a GitLab Shell release and GITLAB_SHELL_VERSION is bumped.

Edited by Fred Reinink

Merge request reports