Rate limit group saves that cascade settings

What does this MR do and why?

A group save that changes a cascading setting queues a background job that updates every subgroup and project in the hierarchy. Nothing bounded how often those saves could run. This MR adds a limit, behind the rate_limit_cascading_settings_updates feature flag (default off).

  • Groups::UpdateService checks two new ApplicationRateLimiter rules before it saves a change that would cascade: 10 per group and 30 per user, every 10 minutes. Over either limit, nothing is saved and the group gets the error "Too many changes to inherited settings. Try again later."
  • The REST API returns that error as 429 Too Many Requests with a Retry-After header. Any other failure still returns 400. The settings page and GraphQL show the error message as they do for other errors.
  • Saves that don't cascade, such as a rename, don't count toward the limit.
  • Turning on the lock of one of those settings counts too, because it clears that lock on every descendant in one UPDATE. Turning a lock off doesn't count.
  • When the submitted foundational flow selection is the same set as the current one, in any order, the save no longer queues Namespaces::CascadeDuoSettingsWorker. The settings page sends the flow list on every save, so most saves of that page no longer start a cascade.

Notes for reviewers:

  • capture_previous_foundational_flow_refs now checks params.key? instead of the value. A request that sends enabled_foundational_flows: nil still records the old selection, so the change is detected and the flows are cleared.
  • Re-saving an unchanged flow selection no longer re-syncs descendants that drifted from the root. The boolean cascading settings already work this way.
  • ee/spec/services/groups/update_service_spec.rb and ee/spec/requests/api/groups_spec.rb now clean the rate limiting Redis store for every example. Redis isn't rolled back between examples, so a shared let_it_be group would otherwise go over the limit across examples.

Thresholds

10 saves per group and 30 per user, each over 10 minutes, come from the proposal in the issue. That's about one cascade per minute for a group and three per minute for a user. An Owner changing settings by hand stays well under that, and a script flipping a setting in a loop gets stopped after the tenth change. Other ApplicationRateLimiter rules for actions that start background or hierarchy-wide work use similar or stricter numbers:

Rule Limit Scope
update_environment_canary_ingress 1 per minute environment
job_retry, job_play 3 per minute user and job
pipeline_retry, pipeline_cancel, pipeline_delete 5 per minute user and pipeline
group_permanent_delete_api 10 per minute user
semantic_code_search_ad_hoc_indexing 10 per hour namespace
security_policies_resync 1 per hour project or group

The rules are hard-coded, like their neighbours in the registry. Once the flag is removed, the limit applies on every instance, including self-managed.

References

Screenshots or screen recordings

Not applicable. The change has no new UI. The settings page shows the error through its existing error message.

Before After

How to set up and validate locally

  1. In the Rails console, enable the feature flag:

    Feature.enable(:rate_limit_cascading_settings_updates)
  2. As the Owner of a group with Duo features available and duo_features_enabled set to false, change that setting through the API 11 times, switching the value each time:

    for value in true false true false true false true false true false true; do
      curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \
        --data "duo_features_enabled=$value" \
        "http://gdk.test:3000/api/v4/groups/<group_id>" --silent --output /dev/null --write-out "%{http_code}\n"
    done

    The first 10 requests return 200 and the 11th returns 429.

  3. Rename the group through the API. It returns 200, because a rename doesn't cascade.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Raounak Sharma

Merge request reports

Loading
Loading