Add log-only rate limits to the billable members and subscription APIs
What does this MR do and why?
The billable members API and the subscription API for free and trial groups calculate a group's billed users on every request, scanning the whole group hierarchy.
The MR counts those requests per user and top-level group in windows (minute, hour, day) using staged_rule log mode. Until enforce_billed_users_scan_rate_limits is enabled for a TLG, requests over a threshold are recorded in rate_limit_state and still succeed. When the feature flag is enabled, they return 429 Too Many Requests.
Note: The thresholds are initial values. After reviewing the log-only data in Kibana over ~1 week, we'll adjust them as needed and add user documentation before enabling the flag globally.
This is the first of two stacked MRs. The second MR !269383 (merged) adds the same limits to GraphQL Group.billableMembersCount and the seat usage CSV export.
References
- Related to: https://gitlab.com/gitlab-org/gitlab/-/work_items/636108
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/639222
How to set up and validate locally
-
In
ee/lib/ee/gitlab/application_rate_limiter/labkit_adapter/supported_rate_limits.rb, set thebilled_users_scan_per_minutelimit to2and restart Rails. -
As an owner of a top-level group, call the billable members API 3 times:
curl --header "PRIVATE-TOKEN: <your_access_token>" \ --url "http://gdk.test:3000/api/v4/groups/<group_id>/billable_members"All 3 calls return
200. Inlog/api_json.log, the third request'srate_limit_stateincludesapplimiter_billed_users_scan_per_minute:limit_billed_users_scans_per_minute_by_user_namespace_log:log. -
In the Rails console, enable the flag for that group:
Feature.enable(:enforce_billed_users_scan_rate_limits, Group.find(<group_id>)) -
Call the API 3 more times. The third call returns
429 Too Many Requests.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.