Rate limit the AI Catalog bulk enable mutation per user

Jaydip Pansuriyarequested to merge
635004-rate-limit-bulk-item-consumer-create into master

What does this MR do and why?

The aiCatalogItemConsumerBulkCreate mutation accepts up to 100 projects per call and queues an Ai::Catalog::ItemConsumers::BulkCreateWorker job that configures the item in each of them, adding memberships, authorization refreshes, audit events and a result email. It had no rate limit, so one user, bot or agent could call it in a loop and queue unbounded background work.

This MR:

  • Registers ai_catalog_item_consumer_bulk_create in the Labkit rate limit registry at 10 requests per minute per user, following the existing ai_catalog_item_report entry.
  • Rejects calls over the limit in Mutations::Ai::Catalog::ItemConsumer::BulkCreate#resolve, after authorization and before the job is queued. The error tells the caller to try again in a minute so scripts can back off. The UI already renders mutation errors in an alert, so no frontend change is needed.
  • Logs each rejected call to the auth log through ApplicationRateLimiter.throttled_request?, with the user ID, username, IP and path, so we can count how many users hit the limit. Entry points without a Rack request, such as GraphqlChannel, still count against the limit without the log entry, following the pattern in Mutations::Repositories::Tags::Create.
  • Adds a description to the mutation so the generated GraphQL reference documents the limit, and adds one sentence to the user docs for enabling agents and flows in multiple projects.

The limit is keyed on the user only. Scoping by item would protect nothing, because a caller can switch items. 10 per minute allows up to 1,000 projects a minute per user, well above legitimate usage, while still stopping a tight loop.

This is the second of three small MRs for the linked issue. The first, !265704 (merged), added a concurrency_limit to the worker.

References

How to set up and validate locally

  1. Create a public agent in the AI Catalog and note its global ID, plus the global IDs of two projects you maintain.

  2. Run this mutation 11 times in quick succession in GraphiQL (/-/graphql-explorer), replacing the IDs:

    mutation {
  aiCatalogItemConsumerBulkCreate(input: {
    itemId: "gid://gitlab/Ai::Catalog::Item/1",
    projectIds: ["gid://gitlab/Project/1", "gid://gitlab/Project/2"]
  }) {
    errors
  }
}

  3. The first 10 calls return empty errors. The 11th returns:

    You have enabled items in bulk too many times. Try again in a minute.

  4. Confirm no BulkCreateWorker job was queued for the 11th call, and that log/auth_json.log has an entry like:

    {"severity":"ERROR","message":"Application_Rate_Limiter_Request","env":"ai_catalog_item_consumer_bulk_create_request_limit","remote_ip":"127.0.0.1","method":"POST","path":"/api/graphql","user_id":1,"username":"root"}

  5. Wait a minute and call again. It succeeds.

Rollout

No feature flag or migration. Rollback is a revert.

After deploy, watch for a week:

  • Who hit the limit: in Kibana, the Rails auth_json subcomponent with json.env: ai_catalog_item_consumer_bulk_create_request_limit. A unique count on json.user_id gives the number of affected users. A hit from a legitimate user means the limit should go up.
  • How often: Prometheus gitlab_labkit_rate_limiter_rule_evaluations_total{rule="limit_ai_catalog_item_consumer_bulk_create_by_user", result="exceeded"}.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jaydip Pansuriya

Merge request reports