[FF] protected_branch_write_rate_limit_enforce -- enforce the protected branch write rate limit

Summary

Roll out enforcement of the protected branch write rate limit, which is behind protected_branch_write_rate_limit_enforce.

The key counts protected branch creations and deletions against the application setting protected_branch_write_limit (default 5,000 per hour, per user, per top-level group). It is registered with staged_rule, so counting starts as soon as the release is deployed and nothing is rejected until this flag is on. The flag takes the root namespace as its actor, so rollout is per top-level group.

Introduced in !267714 (merged).

  • DRI: @j.seto
  • Team Slack channel: #g_create_source_code

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

Once the flag is on, a user over the limit gets 429 from the protected branch REST endpoints and an error from the branchRuleCreate / branchRuleDelete mutations. No data loss, and no effect on reads, on updates to an existing protected branch, or on push access checks.

The blast radius is bounded by the log phase that runs before it. With the flag off, every write that enforcement would reject is counted on the :log twin and reported as:

  • gitlab_labkit_rate_limiter_rule_evaluations_total with rule="limit_protected_branch_writes_by_user_namespace_log" and result="log".
  • rate_limit_state on the request log line: applimiter_protected_branch_write:limit_protected_branch_writes_by_user_namespace_log:log. The meta fields on the same line say who made the request.

Do not enable the flag until that is quiet enough for the namespaces in scope. Any namespace appearing there is one enforcement would start rejecting.

If it is not quiet, correct the threshold first. It is an application setting, so this is a live change and needs no deploy:

PUT /api/v4/application/settings?protected_branch_write_limit=<value>

Setting it to 0 stops the check before it touches Redis, which also stops the counting.

The :log and :limit rules keep separate Redis counters, so the hour the flag flips starts from zero: a user already over the limit gets a fresh budget at that moment rather than an immediate 429.

Events

Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging

Rollout

Run all production /chatops in #production and cross-post the results to #g_create_source_code. Background: incremental rollout process, feature actors.

Phase 1 — log only. No command to run: the key counts on its :log twin from the moment the release is deployed. Review the metric and the rate_limit_state field above, and correct the threshold if the volume is not what we expect.

Non-production:

/chatops gitlab run feature set protected_branch_write_rate_limit_enforce true --dev --pre --staging --staging-ref

Phase 2 — enforce on production. Only after phase 1 has been reviewed. Start with specific namespaces rather than a percentage, and wait ≥15 min between steps:

/chatops gitlab run feature set --group=<group-path> protected_branch_write_rate_limit_enforce true
/chatops gitlab run feature set protected_branch_write_rate_limit_enforce <percentage> --actors

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. It is a gitlab_com_derisk flag, so it is expected to go within two months of merge.

Cleanup is a code change as well as a chatops one: replace staged_rule with Labkit::RateLimit::Rule.new and action: :limit in SupportedRateLimits, which drops the :log twin. The :limit rule keeps its name, so its counter is not reset again. Self-managed instances do not enforce the limit until this lands.

protected_branch_write_limit stays as the operator control, including the 0 escape hatch, so no instance is left without a way to turn the limit off.

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete protected_branch_write_rate_limit_enforce --dev --pre --staging --staging-ref --production

Rollback

Disabling the flag returns the key to log mode: nothing is rejected, and counting and reporting continue.

/chatops gitlab run feature set protected_branch_write_rate_limit_enforce false
/chatops gitlab run feature set protected_branch_write_rate_limit_enforce false --dev --pre --staging --staging-ref

To stop counting as well, set the threshold to 0:

PUT /api/v4/application/settings?protected_branch_write_limit=0
Edited by Jerry Seto