LabKit rack rate limiter emits user_authenticated_using_job_token audit event on every JOB-TOKEN request, exhausts Redis (19.3 regression)

Summary

audit_job_token_authentication in ee/lib/ee/gitlab/auth/auth_finders.rb unconditionally emits a user_authenticated_using_job_token audit event every time find_user_from_job_token resolves a user. That helper is called from any code path that runs the auth chain — including rate-limiter identity resolution — not only from real controller-facing authentication.

Every request carrying a JOB-TOKEN header therefore emits multiple user_authenticated_using_job_token audit events per request, one per rate-limiter classification call:

  • Rack::Attack::Request#throttled_identifer calls authenticated_identifier → Gitlab::Auth::RequestAuthenticator#find_authenticated_requester → find_user_from_job_token. This runs once per matching throttle during Rack::Attack evaluation (typically 1–3 times per request). This has been true since the audit was introduced in !164546 (merged) (17.4, September 2024) — it is pre-existing behaviour, not a 19.3 regression by itself.
  • Gitlab::Middleware::LabkitRackRateLimit (introduced by !250401 (merged), default-enabled for self-managed in 19.3) adds one additional identity-resolution call per request via ClassifiedRequest#labkit_facts, running whenever any of the six rate_limiter_use_labkit_rack_cohort_{1,2,3}[_enforce] flags is enabled.

On instances with an audit event streaming destination configured and no event-type filter, each emission becomes a Sidekiq job on queue:audit_events. On instances where the request rate to JOB-TOKEN-authenticated endpoints exceeds the audit workers' drain rate, the queue accumulates and can saturate the shared Redis (noeviction), taking down webservice, Sidekiq, and rate limiting together.

Impact

Found during this incident on a GitLab Dedicated tenant on 19.3.2.

  • Sustained accumulation on queue:audit_events under normal traffic, with the queue growing faster than the audit workers could drain it.
  • Redis memory pressure on the shared ElastiCache instance rose significantly over the incident window and would have led to write failures without intervention (webservice, Sidekiq, and rate limiting all share this Redis).
  • Customer observed the audit-event Sidekiq surge and opened a support ticket during the incident.
  • Mitigation on the tenant: disabled all six flags (three shadow + three _enforce) via Rails console to remove the LabKit-side classification and cut audit-event ingress by roughly the LabKit-added share. The queue drained. Rack::Attack's calls to the same auth chain continue at the pre-19.3 rate on this tenant. 
    rate_limiter_use_labkit_rack_cohort_{1,2,3}
rate_limiter_use_labkit_rack_cohort_{1,2,3}_enforce

The tenant-local Rails-console flag disable is a partial mitigation: it removes the LabKit-added identity resolution but not Rack::Attack's. On tenants where the audit workers cannot keep up even at the Rack::Attack-only ingress rate, disabling the flags will not be sufficient.

Root cause

Two independent contributions to the same symptom, both in the auth-emission code path:

  1. Pre-existing (17.4, September 2024): !164546 (merged) added audit_job_token_authentication to the EE override of find_user_from_job_token in ee/lib/ee/gitlab/auth/auth_finders.rb. The feature spec (#481325 (closed), group::compliance) asked for an audit event on job-token authentication. The emission was placed inside find_user_from_job_token itself, so it fires for every caller of that method, not only for controller-facing authentication. This includes rate-limiter identity resolution (both Rack::Attack::Request#throttled_identifer and, later, LabKit's ClassifiedRequest#requester).
  2. Amplifier (19.3, !250401 (merged)): default-enabled the six LabKit rack cohort flags for self-managed as part of the Rack::Attack → Labkit::RateLimit migration tracked in #602804. Gitlab::Middleware::LabkitRackRateLimit#run executes on every request when any cohort flag is on, calls ClassifiedRequest#labkit_facts → requester([:api, :rss, :ics]) → authenticated_identifier → the auth chain → find_user_from_job_token. That is one additional emission per request beyond what Rack::Attack was already doing.

Disabling the LabKit shadow flags removes contribution (2) but not contribution (1). The 19.3 rollout made the pre-existing over-emission from (1) severe enough to notice at the fleet edge (tenants with audit-event streaming + high JOB-TOKEN request volume), but the fundamental design issue lives in (1).

Neither !164546 (merged) nor !250401 (merged) is individually a bug in isolation. The bug is the invariant: an audit-emitting side effect placed at a shared auth-primitive seam, called by any code that needs to know the requester.

Fix direction (speculative — for the owning group to confirm)

The following is a speculative fix direction based on discussion during the incident (see the incident issue) and a read of the affected code path. The owning group should confirm.

The audit emission should only fire when find_user_from_job_token is called from a controller-facing authentication path, not from a rate-limiter (or any future) identity-resolution caller. Possible seams:

  • Move the audit emission out of the finder and up to its controller-facing callers (ApplicationController before_actions, API::Helpers, Rack::Attack explicitly opts out by not calling the emitter). The finder becomes side-effect-free; callers that want to audit call the emitter explicitly.
  • Alternatively, keep the emission where it is but guard it on a request-scoped flag ("this call is from a controller-facing auth pass") set by the controller-facing callers before invoking the finder. Rate-limiter callers wouldn't set the flag, so their invocations wouldn't emit.
  • Add a spec that runs both Rack::Attack::Request#throttled_identifer and the LabKit shadow pass with a JOB-TOKEN request, and asserts zero user_authenticated_using_job_token audit events emitted from either. A separate spec covers the controller-facing path and asserts exactly one emission.
  • Backport the fix to 19.3.

The Compliance group should be consulted on the semantics: is the intent of the audit event "record every time a job token authenticates a user in any context" (current behaviour, amplified) or "record every time a job token authenticates a user for the purpose of performing an action" (implied by the feature spec, but not what the code does)?

Relates to

Partial workaround for affected instances

Rails console:

flags = %i[
  rate_limiter_use_labkit_rack_cohort_1
  rate_limiter_use_labkit_rack_cohort_2
  rate_limiter_use_labkit_rack_cohort_3
  rate_limiter_use_labkit_rack_cohort_1_enforce
  rate_limiter_use_labkit_rack_cohort_2_enforce
  rate_limiter_use_labkit_rack_cohort_3_enforce
]

# Record current state
flags.to_h { |f| [f, Feature.enabled?(f, type: :beta)] }

# Disable
flags.each { |f| Feature.disable(f) }

# Confirm all false
flags.to_h { |f| [f, Feature.enabled?(f, type: :beta)] }

Rack::Attack resumes as the sole rate limiter (parity with 19.2 behaviour). Confirm GITLAB_THROTTLE_* env is intact if the instance relied on Rack::Attack throttles.

This is a partial mitigation. It removes only the LabKit-added identity-resolution call. Rack::Attack's own identity resolution continues to emit user_authenticated_using_job_token audit events at the pre-19.3 rate. On instances where the audit workers cannot keep up even at that rate, the queue will still accumulate; the only complete workaround until the fix ships is disabling the audit-event streaming destination or adding a user_authenticated_using_job_token event-type filter on the destination.

Edited by Chris Taylor