## Summary `audit_job_token_authentication` in `ee/lib/ee/gitlab/auth/auth_finders.rb` unconditionally emits a `user_authenticated_using_job_token` audit event every time `find_user_from_job_token` resolves a user. That helper is called from **any** code path that runs the auth chain — including rate-limiter identity resolution — not only from real controller-facing authentication. Every request carrying a `JOB-TOKEN` header therefore emits multiple `user_authenticated_using_job_token` audit events per request, one per rate-limiter classification call: - **`Rack::Attack::Request#throttled_identifer`** calls `authenticated_identifier` → `Gitlab::Auth::RequestAuthenticator#find_authenticated_requester` → `find_user_from_job_token`. This runs **once per matching throttle** during Rack::Attack evaluation (typically 1–3 times per request). This has been true since the audit was introduced in !164546 (17.4, September 2024) — it is pre-existing behaviour, not a 19.3 regression by itself. - **`Gitlab::Middleware::LabkitRackRateLimit`** (introduced by !250401, default-enabled for self-managed in 19.3) adds **one additional** identity-resolution call per request via `ClassifiedRequest#labkit_facts`, running whenever any of the six `rate_limiter_use_labkit_rack_cohort_{1,2,3}[_enforce]` flags is enabled. On instances with an audit event streaming destination configured and no event-type filter, each emission becomes a Sidekiq job on `queue:audit_events`. On instances where the request rate to JOB-TOKEN-authenticated endpoints exceeds the audit workers' drain rate, the queue accumulates and can saturate the shared Redis (`noeviction`), taking down webservice, Sidekiq, and rate limiting together. ## Impact Found during [this incident](https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/4504) on a GitLab Dedicated tenant on `19.3.2`. - Sustained accumulation on `queue:audit_events` under normal traffic, with the queue growing faster than the audit workers could drain it. - Redis memory pressure on the shared ElastiCache instance rose significantly over the incident window and would have led to write failures without intervention (webservice, Sidekiq, and rate limiting all share this Redis). - Customer observed the audit-event Sidekiq surge and opened a support ticket during the incident. - Mitigation on the tenant: disabled all six flags (three shadow + three `_enforce`) via Rails console to remove the LabKit-side classification and cut audit-event ingress by roughly the LabKit-added share. The queue drained. `Rack::Attack`'s calls to the same auth chain continue at the pre-19.3 rate on this tenant. ``` rate_limiter_use_labkit_rack_cohort_{1,2,3} rate_limiter_use_labkit_rack_cohort_{1,2,3}_enforce ``` The tenant-local Rails-console flag disable is a partial mitigation: it removes the LabKit-added identity resolution but not `Rack::Attack`'s. On tenants where the audit workers cannot keep up even at the Rack::Attack-only ingress rate, disabling the flags will not be sufficient. ## Root cause Two independent contributions to the same symptom, both in the auth-emission code path: 1. **Pre-existing (17.4, September 2024)**: !164546 added `audit_job_token_authentication` to the EE override of `find_user_from_job_token` in `ee/lib/ee/gitlab/auth/auth_finders.rb`. The feature spec (#481325, `group::compliance`) asked for an audit event on job-token authentication. The emission was placed inside `find_user_from_job_token` itself, so it fires for **every caller** of that method, not only for controller-facing authentication. This includes rate-limiter identity resolution (both `Rack::Attack::Request#throttled_identifer` and, later, LabKit's `ClassifiedRequest#requester`). 2. **Amplifier (19.3, !250401)**: default-enabled the six LabKit rack cohort flags for self-managed as part of the Rack::Attack → Labkit::RateLimit migration tracked in #602804. `Gitlab::Middleware::LabkitRackRateLimit#run` executes on every request when any cohort flag is on, calls `ClassifiedRequest#labkit_facts` → `requester([:api, :rss, :ics])` → `authenticated_identifier` → the auth chain → `find_user_from_job_token`. That is one additional emission per request beyond what Rack::Attack was already doing. Disabling the LabKit shadow flags removes contribution (2) but not contribution (1). The 19.3 rollout made the pre-existing over-emission from (1) severe enough to notice at the fleet edge (tenants with audit-event streaming + high JOB-TOKEN request volume), but the fundamental design issue lives in (1). Neither !164546 nor !250401 is individually a bug in isolation. The bug is the invariant: an audit-emitting side effect placed at a shared auth-primitive seam, called by any code that needs to know the requester. ## Fix direction (speculative — for the owning group to confirm) _The following is a speculative fix direction based on discussion during the incident (see [the incident issue](https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/4504)) and a read of the affected code path. The owning group should confirm._ The audit emission should only fire when `find_user_from_job_token` is called from a controller-facing authentication path, not from a rate-limiter (or any future) identity-resolution caller. Possible seams: - Move the audit emission out of the finder and up to its controller-facing callers (`ApplicationController` before_actions, `API::Helpers`, `Rack::Attack` explicitly opts out by not calling the emitter). The finder becomes side-effect-free; callers that want to audit call the emitter explicitly. - Alternatively, keep the emission where it is but guard it on a request-scoped flag ("this call is from a controller-facing auth pass") set by the controller-facing callers before invoking the finder. Rate-limiter callers wouldn't set the flag, so their invocations wouldn't emit. - Add a spec that runs both `Rack::Attack::Request#throttled_identifer` and the LabKit shadow pass with a JOB-TOKEN request, and asserts zero `user_authenticated_using_job_token` audit events emitted from either. A separate spec covers the controller-facing path and asserts exactly one emission. - Backport the fix to 19.3. The Compliance group should be consulted on the semantics: is the intent of the audit event "record every time a job token authenticates a user in any context" (current behaviour, amplified) or "record every time a job token authenticates a user for the purpose of performing an action" (implied by the feature spec, but not what the code does)? ## Relates to - Feature-added the audit emission: gitlab-org/gitlab!164546 (17.4, September 2024). Feature spec: gitlab-org/gitlab#481325 (`group::compliance`). - Amplifier: gitlab-org/gitlab!250401 (19.3). Rollout tracker: gitlab-org/gitlab#602804. - Sibling 19.3 regression from !250401 (unrelated to this bug): gitlab-org/gitlab#630787. - Incident: [gitlab-com/gl-infra/gitlab-dedicated/incident-management#4504](https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/4504) ## Partial workaround for affected instances Rails console: ```ruby flags = %i[ rate_limiter_use_labkit_rack_cohort_1 rate_limiter_use_labkit_rack_cohort_2 rate_limiter_use_labkit_rack_cohort_3 rate_limiter_use_labkit_rack_cohort_1_enforce rate_limiter_use_labkit_rack_cohort_2_enforce rate_limiter_use_labkit_rack_cohort_3_enforce ] # Record current state flags.to_h { |f| [f, Feature.enabled?(f, type: :beta)] } # Disable flags.each { |f| Feature.disable(f) } # Confirm all false flags.to_h { |f| [f, Feature.enabled?(f, type: :beta)] } ``` `Rack::Attack` resumes as the sole rate limiter (parity with 19.2 behaviour). Confirm `GITLAB_THROTTLE_*` env is intact if the instance relied on Rack::Attack throttles. **This is a partial mitigation.** It removes only the LabKit-added identity-resolution call. `Rack::Attack`'s own identity resolution continues to emit `user_authenticated_using_job_token` audit events at the pre-19.3 rate. On instances where the audit workers cannot keep up even at that rate, the queue will still accumulate; the only complete workaround until the fix ships is disabling the audit-event streaming destination or adding a `user_authenticated_using_job_token` event-type filter on the destination.