Rate limit AI audit event ingest

What does this MR do and why?

Adds rate limits to AI audit event ingest:

Limit Endpoints Counts Default
ai_duo_workflow_audit_event_ingest POST /api/v4/ai/duo_workflows/workflows/:id/audit_events Requests per user 120/min
ai_audit_event_ingest_by_top_level_namespace The Duo Workflow endpoint and POST /api/v4/projects/:id/ai_agent/audit_events Events per top-level namespace 20,000/min
  • The Duo Workflow endpoint had no limit. The external agent endpoint keeps its existing limit of 10 requests a minute per user and project.
  • The namespace limit counts events, not requests, using the rate limiter's cost:, because one request carries anywhere from 1 to 500 events.
  • For composite identity tokens, current_user is the human, so the per-user limit doesn't put everyone behind the service account in one bucket.
  • The Duo Workflow endpoint now rejects batches over 500 events in parameter validation, as the external agent endpoint does, so an oversized request can't use up the namespace limit. It answered 400 before too, from the service.
  • On the external agent endpoint, the namespace limit counts after the event and session checks, so requests that are rejected anyway don't count. The existing user and project limit stays first.

Both limits always count, but they only answer 429 while the enforce_ai_audit_event_ingest_limits flag (gitlab_com_derisk, actor: top-level namespace) is on. With the flag off, requests over a limit are still logged to the auth log and recorded in rule_evaluations_total, so the limits can be checked against real traffic before they reject anything. Labkit's action: :log would do this without a flag, but the application rate limiter requires :limit rules and treats an exceeded :log rule as throttled.

Why these numbers

  • glab posts once per prompt from Claude Code hooks, and in batches of up to 500 events when it uploads older sessions. It waits as Retry-After asks, and hooks don't wait, so a 429 delays uploads rather than losing them.
  • A heavy Claude Code user with several sessions running sends at most about 50 events a minute, and p99 is under 40, from local transcripts. 20,000 events a minute is about 400 such users at their peak at the same time.
  • On GitLab.com over the last 7 days, the Duo Workflow endpoint had 2.37M requests, 235 a minute on average and 539 at the peak. Requests carry around 3 events each, so the peak is roughly 1,600 events a minute across all of GitLab.com. The external agent endpoint had about 190 requests.
  • In the peak minute, the busiest real user sent 9 requests, and a QA test account sent 55. 120 a minute per user leaves room for that account while still being over 10 times the busiest real user.
  • At 120 requests of 500 events, one user could send 60,000 events a minute, more than the namespace limit. Real traffic is nowhere near that, but it means one runaway client could use up its namespace's limit. That's part of what the flag-off period should show before enforcing.

How to set up and validate locally

  1. In a Rails console, lower the namespace limit for a quick check, or post a 500-event batch repeatedly to POST /api/v4/projects/:id/ai_agent/audit_events.
  2. With Feature.disable(:enforce_ai_audit_event_ingest_limits), requests over the limit are accepted and logged to log/auth_json.log with env ai_audit_event_ingest_by_top_level_namespace_request_limit.
  3. With Feature.enable(:enforce_ai_audit_event_ingest_limits), they get 429 with Retry-After.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jean van der Walt

Merge request reports

Loading
Loading