Rate limit group saves that cascade settings
What does this MR do and why?
A group save that changes a cascading setting queues a background job that updates every subgroup and project in the hierarchy. Nothing bounded how often those saves could run. This MR adds a limit, behind the rate_limit_cascading_settings_updates feature flag (default off).
Groups::UpdateServicechecks two newApplicationRateLimiterrules before it saves a change that would cascade: 10 per group and 30 per user, every 10 minutes. Over either limit, nothing is saved and the group gets the error "Too many changes to inherited settings. Try again later."- The REST API returns that error as
429 Too Many Requestswith aRetry-Afterheader. Any other failure still returns 400. The settings page and GraphQL show the error message as they do for other errors. - Saves that don't cascade, such as a rename, don't count toward the limit.
- Turning on the lock of one of those settings counts too, because it clears that lock on every descendant in one UPDATE. Turning a lock off doesn't count.
- When the submitted foundational flow selection is the same set as the current one, in any order, the save no longer queues
Namespaces::CascadeDuoSettingsWorker. The settings page sends the flow list on every save, so most saves of that page no longer start a cascade.
Notes for reviewers:
capture_previous_foundational_flow_refsnow checksparams.key?instead of the value. A request that sendsenabled_foundational_flows: nilstill records the old selection, so the change is detected and the flows are cleared.- Re-saving an unchanged flow selection no longer re-syncs descendants that drifted from the root. The boolean cascading settings already work this way.
ee/spec/services/groups/update_service_spec.rbandee/spec/requests/api/groups_spec.rbnow clean the rate limiting Redis store for every example. Redis isn't rolled back between examples, so a sharedlet_it_begroup would otherwise go over the limit across examples.
Thresholds
10 saves per group and 30 per user, each over 10 minutes, come from the proposal in the issue. That's about one cascade per minute for a group and three per minute for a user. An Owner changing settings by hand stays well under that, and a script flipping a setting in a loop gets stopped after the tenth change. Other ApplicationRateLimiter rules for actions that start background or hierarchy-wide work use similar or stricter numbers:
| Rule | Limit | Scope |
|---|---|---|
update_environment_canary_ingress |
1 per minute | environment |
job_retry, job_play |
3 per minute | user and job |
pipeline_retry, pipeline_cancel, pipeline_delete |
5 per minute | user and pipeline |
group_permanent_delete_api |
10 per minute | user |
semantic_code_search_ad_hoc_indexing |
10 per hour | namespace |
security_policies_resync |
1 per hour | project or group |
The rules are hard-coded, like their neighbours in the registry. Once the flag is removed, the limit applies on every instance, including self-managed.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/635299
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/635500
Screenshots or screen recordings
Not applicable. The change has no new UI. The settings page shows the error through its existing error message.
| Before | After |
|---|---|
How to set up and validate locally
-
In the Rails console, enable the feature flag:
Feature.enable(:rate_limit_cascading_settings_updates) -
As the Owner of a group with Duo features available and
duo_features_enabledset tofalse, change that setting through the API 11 times, switching the value each time:for value in true false true false true false true false true false true; do curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ --data "duo_features_enabled=$value" \ "http://gdk.test:3000/api/v4/groups/<group_id>" --silent --output /dev/null --write-out "%{http_code}\n" doneThe first 10 requests return
200and the 11th returns429. -
Rename the group through the API. It returns
200, because a rename doesn't cascade.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.