Rate limit secrets manager enablement per top-level group

Erick Bajaorequested to merge
eb-rate-limit-gsm-provision into master

What does this MR do and why?

Problem

  • Each secrets manager adds an OpenBao namespace. OpenBao loads every namespace on startup and failover, so each new one makes recovery slower for every tenant.
  • Nothing limits how fast one top-level group can enable secrets managers. A script or agent can create a lot of namespaces in a short time.

Solution

  • Add a secrets_manager_initialize rate limit: 30 enablements per minute per top-level group.
  • Check it in the projectSecretsManagerInitialize and groupSecretsManagerInitialize GraphQL mutations. Projects and groups in the same top-level group share the limit.
  • Do not count requests for a project or group that already has a secrets manager, for a project in a user namespace (the service rejects those), or requests rejected earlier (not authorized, inactive namespace, entitlement denied).
  • Ship it as a staged_rule behind the secrets_manager_initialize_rate_limit flag. While the flag is off, the limit only logs. Nothing is blocked.
  • Document the limit in the non-configurable rate limits page and the Secrets Manager performance page.

Design notes

  • Mutations, not InitializeServiceHelpers#execute. StartTrial and EnableAddOn also call the service, but only after they change billing state. A rate limit error there would leave a running trial or a recorded add-on next to a failed UI. These two paths only provision the top-level group itself, once, so they cannot grow the namespace count fast. Specs make sure they stay unlimited.
  • Per minute, not per hour. The goal is to protect OpenBao from bursts, not to slow down large groups. The limiter uses a fixed window, so a blocked group waits at most one minute. A script that enables 1,000 projects is paced to about 35 minutes instead of being blocked for hours.
  • Counted after the access checks. The check runs after authorization, the inactive namespace check, and the entitlement check. A user without access cannot use up a group's quota.
  • The provision-in-request plan. If provisioning moves into the request (#638671), the Initialize mutations carry the OpenBao load, so the limit already sits in the right place.
Why 30 per minute

Each provision makes more than 10 OpenBao calls, so one top-level group can cause about 300 or more OpenBao calls per minute through this path. That stops tight loops but leaves room for normal use. 30 is a starting value. Log mode shows real traffic on GitLab.com before we enforce, and the rollout issue has a step to raise it if a legitimate group goes over. Concurrency across all groups is handled separately in https://gitlab.com/gitlab-org/gitlab/-/work_items/638687.

Rollout and rollback

  • Flag: secrets_manager_initialize_rate_limit (gitlab_com_derisk, default off). The actor is the top-level group.
  • Flag off: log mode. Requests over the limit show up as limit_secrets_manager_initializations_by_root_namespace_log in the gitlab_labkit_rate_limiter_rule_evaluations_total metric and in the rate_limit_state log field.
  • Flag on: requests over the limit get This endpoint has been requested too many times. Try again later. in the payload errors. The UI shows that message as-is.
  • Rollback: disable the flag. The limit goes back to log mode right away.
  • Rollout issue: #639391

Databases touched

None. The counter lives in Redis (rate limiting).

References

Screenshots or screen recordings

Not applicable. No UI changes.

How to set up and validate locally

  1. Follow the Secrets Manager local setup.

  2. In the Rails console, enable the flag for a top-level group, and use up its quota:

    group = Group.find_by_full_path('gitlab-org')
Feature.enable(:secrets_manager_initialize_rate_limit, group)
30.times { Gitlab::ApplicationRateLimiter.throttled?(:secrets_manager_initialize, scope: { namespace: group }) }

  3. In GraphiQL (/-/graphql-explorer), as an owner, initialize a project in that group that has no secrets manager:

    mutation {
  projectSecretsManagerInitialize(input: { projectPath: "gitlab-org/gitlab-test" }) {
    errors
    projectSecretsManager { status }
  }
}

    The payload errors has This endpoint has been requested too many times. Try again later.

  4. Run it again for a project that already has a secrets manager. It answers Secrets manager already initialized for the project. and is not blocked.

  5. Disable the flag, and check that the same quota no longer blocks (log mode):

    Feature.disable(:secrets_manager_initialize_rate_limit)
Array.new(31) { Gitlab::ApplicationRateLimiter.throttled?(:secrets_manager_initialize, scope: { namespace: group }) }.any?
# => false

    Run step 3 again on a project without a secrets manager. It succeeds.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Erick Bajao

Merge request reports