## Summary Roll out [the feature](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30224) currently behind the `rate_limiter_endpoint_block_counter` feature flag. - DRI: @nindurkar - Team Slack channel: `#g_tenant-controls` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag only adds work on requests the rack limiter has already blocked. The 429 does not change: the increment runs in its own guard, so a raise is tracked and the block still returns. - Extra time on each blocked request, from the route lookup. Measured on a test environment, not production: p99 at or under 0.49 ms per lookup (2,000 lookups on each of 10 paths). - Longer boot per Puma process: the route matcher is built once at boot, about 162 ms (median of 5 builds, test environment). This happens even with the flag off. - New series of `gitlab_rate_limit_rack_blocks_total`, one per `endpoint_id` and `rule` pair that blocks. In the 24h to 2026-10-29 10:00 UTC, 17 rack rules blocked at least once on gprd. ## Dashboards and queries - [rate-limiting: Overview](https://dashboards.gitlab.net/d/rate-limiting-main/rate-limiting3a-overview) - [Mimir / Cardinality Overview](https://dashboards.gitlab.net/d/cardinality-management/mimir-cardinality-overview), for the series count - [Mimir Tenants, gitlab-gprd ingestion](https://dashboards.gitlab.net/d/35fa247ce651ba189debf33d7ae41611/mimir-tenants?orgId=1&var-datasource=mimir-metamonitoring&var-namespace=mimir&var-user=gitlab-gprd&viewPanel=14), for E against the 8M limit - The two Kibana links under Events, for exceptions with the flag on Queries, all on Mimir gprd: ```promql # A: blocks the new counter saw sum(rate(gitlab_rate_limit_rack_blocks_total{env="gprd"}[5m])) # B: blocks LabKit reported sum(rate(gitlab_labkit_rate_limiter_checks_total{env="gprd", rate_limiter=~"rack_request.*", action="block"}[5m])) # C: share of blocks with no endpoint found sum(rate(gitlab_rate_limit_rack_blocks_total{env="gprd", endpoint_id="unknown"}[5m])) / sum(rate(gitlab_rate_limit_rack_blocks_total{env="gprd"}[5m])) # D: series count count(gitlab_rate_limit_rack_blocks_total{env="gprd"}) ``` Queries on the Mimir metamonitoring datasource: ```promql # E: gitlab-gprd ingestion, samples/s sum(rate(cortex_distributor_received_samples_total{user="gitlab-gprd"}[5m])) # F: samples Mimir rejected for the rate limit sum(rate(cortex_discarded_samples_total{user="gitlab-gprd", reason="rate_limited"}[5m])) ``` ## Expected values - Today B is about 100.7 million blocks a day (observed, 24h to 2026-10-29 10:00 UTC, `increase(...[24h])`), almost all on `rack_request`. - At N% rollout, A should be about N% of B (inferred). The flag is checked per request. - At 100%, A should be close to B, and never above it (inferred). A request blocked by two limiters counts twice in B and once in A. - C is the share we could not map to an endpoint. No baseline yet. Record it at the first step. - D is 0 today (observed, 2026-11-03 04:38 UTC). `gitlab_sli_rails_request_total` has 2,592,973 series (observed, same time), and D should stay under 10% of that. - E is about 7.35M samples/s against an 8M `ingestion_rate` limit with a 10M burst (observed, 2026-11-03 04:40 UTC). Peaks already touch 8M. F is 0. ## Stop and roll back if - Any exception from the endpoint lookup or the counter shows in Sentry or in the Kibana exceptions link. - A goes above B. - An apdex or error-ratio alert fires for web, api or git during a step. - D at 5%: above 12,950, read 1 hour after the step. The full-rollout line is 259,000 (10% of 2,592,973, rounded down), and 5% of that is 12,950. - D at later steps: above 259,000 times the step fraction (for example 64,750 at 25%), or above 259,000 at any step. - F above 0. - E above 8M for 15 minutes. A short spike over 8M is fine, the burst limit is 10M. ## Continue if All the above hold for 15 minutes or more at the current step, then go to the next one. At 5%, wait 1 hour, so the D reading is complete. ## Events - [Exceptions with rate_limiter_endpoint_block_counter:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22rate_limiter_endpoint_block_counter:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with rate_limiter_endpoint_block_counter:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22rate_limiter_endpoint_block_counter:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_tenant-controls`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set rate_limiter_endpoint_block_counter 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set rate_limiter_endpoint_block_counter true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set rate_limiter_endpoint_block_counter <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss rate_limiter_endpoint_block_counter true /chatops gitlab run feature set --group=gitlab-org,gitlab-com rate_limiter_endpoint_block_counter true /chatops gitlab run feature set --user=nindurkar rate_limiter_endpoint_block_counter true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete rate_limiter_endpoint_block_counter --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set rate_limiter_endpoint_block_counter false # production /chatops gitlab run feature set rate_limiter_endpoint_block_counter false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete rate_limiter_endpoint_block_counter --dev --pre --staging --staging-ref --production # remove entirely ```