Rate-limit failed SSH certificate authentication per client IP

Fred Reininkrequested to merge
freinink/rate-limit-authorized-certs into master

What does this MR do and why?

Adds a per-SSH-client-IP limit on failed resolutions to GET /api/v4/internal/authorized_certs, the endpoint gitlab-sshd calls to resolve an SSH certificate authority fingerprint and user identifier to a GitLab user.

Anyone can generate a CA, sign a structurally valid certificate with it, and have gitlab-sshd forward it to Rails. They need no GitLab account, no trusted certificate, and no Shell secret. No existing control covers this path: Rack::Attack excludes internal API paths, gitlab_shell_operation limiting runs only in /internal/allowed (which rejected certificates never reach), and unknown identities have no lockout. An upcoming MR adds an audit event for every failed resolution, which would otherwise be an unbounded database write driven by unauthenticated traffic.

Design: count failures, block everything once over

The attacker in this threat model holds no trusted certificate, so every request they can produce is a failure. Counting only failures is exactly as effective against them as counting every request, and it never penalizes legitimate traffic on its own: a busy runner fleet or office behind NAT cannot reach the limit without someone behind that address producing 300 bad certificates in a minute.

Once a source has reached the limit, every request from it is rejected until the one-minute window expires, including requests that would have resolved successfully. This is the same shape as the failed authentication ban for Git and the container registry: a source that just produced 300 bad certificates is treated as hostile for a short time. Rejecting only the failures would require running both lookups first, which spends the resolution work and leaves only the audit write to save.

The threshold is 300 failures per minute per IP, the same figure GitLab.com uses to ban Git and registry authentication. Legitimate failure sources (foreign certificates in an SSH agent, a misconfigured runner, a lapsed group licence) cannot plausibly reach it.

Changes

  • Register a fixed ssh_certificate_authentication rule: 300 per minute, keyed on ip.
  • In authorized_cert_payload, peek at the limit (check_rate_limit! with peek: true) as the first statement, after Shell-token authentication and before either resolution service. A peek does not increment; a source already over the limit gets the standard 429.
  • After both the group and instance lookups fail, immediately before the error response, increment the counter with mark_throttle!. Successful resolutions are never counted and do not reset the counter. This is the same point where the follow-up MR records the audit event.
  • Add a CE ssh_client_ip helper that normalizes check_ip (strips IPv6 zone IDs, collapses IPv4-mapped IPv6 to IPv4, returns nil for anything invalid). The scope falls back to the request IP (the GitLab Shell node) when check_ip is missing or invalid, which only happens during mixed-version upgrade windows. Because only failures count, clients behind an older Shell node are unaffected unless the node as a whole produces 300 failures in a minute.
  • Exempt IP addresses in the rack_attack.git_basic_auth.ip_whitelist setting, the same allowlist check_allowed honours for its per-IP limit in this file. This gives administrators an escape hatch for a shared egress address that legitimately produces many failures.
  • Gate the whole thing behind the ssh_certificate_authentication_rate_limit feature flag (gitlab_com_derisk, default off). The same flag will gate the audit event in the follow-up MR so audit writes never happen without the limit in place.

A throttled request gets the standard check_rate_limit! response: 429 with Retry-After: 60 and RateLimit-Name: ssh_certificate_authentication. The standard Application_Rate_Limiter_Request entry in auth.log records the check_ip in the request path, so the client IP is logged without extra code. Increments produce no log line.

This is MR 3 of 4 in the implementation plan. The flag must stay off until the two GitLab Shell MRs (send check_ip, do not retry 429) ship and GITLAB_SHELL_VERSION is bumped.

References

Screenshots or screen recordings

No UI changes.

How to set up and validate locally

See the steps in the shell MR: gitlab-shell!1616 (merged)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Fred Reinink

Merge request reports