Rate limit bulk add-on seat assignment per user
What does this MR do and why?
The GraphQL mutation UserAddOnAssignmentBulkCreate (ee/app/graphql/mutations/gitlab_subscriptions/user_add_on_assignments/bulk_create.rb) assigns add-on seats (for example GitLab Duo) to up to 100 users per call. On GitLab.com, each newly assigned user enqueues one Onboarding::CreateIterableTriggerWorker job, which makes one CustomersDot request. The number of calls one user could make was not limited. One actor could add unbounded load on CustomersDot by cycling unassign and reassign.
This MR adds an application rate limit, key user_add_on_assignment_bulk_create. The limit is the number of userAddOnAssignmentBulkCreate calls allowed every minute for each user. It is set by the new application setting user_add_on_assignment_bulk_create_limit:
- The setting is an integer in the
rate_limitscolumn ofApplicationSetting. It is EE only and API only, with no Admin UI. - The default is
0, which disables the limit. The mutation also runs on GitLab Self-Managed and GitLab Dedicated, so a non-zero default would start rejecting calls on upgrade. !271238 shows the same default applied to three 19.6 limits for the same reason. - When the setting is above
0and a user goes over it, the mutation responds with HTTP 429 and assigns no seats. The message "This endpoint has been requested too many times. Try again later." is in the top-level GraphQLerrors, not in the mutation payload'serrorsfield.
The rule is a Labkit::RateLimit::Rule with action: :limit in ee/lib/ee/gitlab/application_rate_limiter/labkit_adapter/supported_rate_limits.rb. Its limit lambda reads the setting. The check runs in resolve, after authorization and argument validation, so unauthorized or invalid calls do not count.
The limit is an application setting instead of a feature flag, following the new Project Bulkhead standard for new rate limits. Example: !271030 (merged).
This MR also adds the setting to the JSON schema, the application settings API, the visible attributes helper, doc/api/settings.md, doc/rate_limits/_index.md, and the OpenAPI doc. It removes the feature flag enforce_user_add_on_assignment_bulk_create_rate_limit.
Changelog: added, because the setting is new in the application settings API.
GitLab.com rollout
After this MR is merged, a production change request sets the value to 20 on GitLab.com. That allows at most 2,000 users per minute for each caller (20 calls, up to 100 users per call). Setting the value back to 0 turns the limit off. The change request is tracked on https://gitlab.com/gitlab-org/gitlab/-/work_items/635547.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/635547 (Project Bulkhead Phase 2). This MR covers the "Rate-limit the BulkCreate seat-assignment mutation per user" item of its plan.
- !271238 (reference for defaulting 19.6 limits to
0) - !271030 (merged) (Project Bulkhead standard example)
Screenshots or screen recordings
Not applicable, backend only.
How to set up and validate locally
-
Simulate SaaS and have a group with a GitLab Duo add-on purchase with free seats. Sign in as its owner.
-
In the Rails console, set the limit:
ApplicationSetting.current.update!(user_add_on_assignment_bulk_create_limit: 2)Or use the API:
PUT /api/v4/application/settings?user_add_on_assignment_bulk_create_limit=2. -
Call the
userAddOnAssignmentBulkCreatemutation 3 times within a minute, for example from/-/graphql-explorer. -
The 3rd call returns HTTP 429, with the rate limit message in the top-level GraphQL
errorsrather than the payload'serrorsfield. -
Set the limit back to
0and repeat. All calls succeed.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.