Roll out [the feature](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30224) currently behind the `rate_limiter_endpoint_block_summary` feature flag. It replaces `rate_limiter_endpoint_block_counter` (gitlab-org/gitlab#636996), which was rolled back. - DRI: @nindurkar - Team Slack channel: `#g_tenant-controls` > [!note] > Process and guidance live in the docs. This issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability. It is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## Before the first change - [x] `Labkit::RateLimit::BlockSummary` released in labkit-ruby, and `gitlab-labkit` bumped in gitlab-rails. - [x] The Rails MR merged and deployed to gprd. - [x] Daily line limit agreed with the logging team, and written into the stop lines below. The logging team picked 15 minutes and 5 pairs ([Slack thread](https://gitlab.slack.com/archives/C0ATYKN2NTG/p1793684290777599), 2026-11-09). - [x] gitlab-org/gitlab!269205 (15 minutes, 5 pairs) deployed to gprd. ## What could go wrong? The flag only adds work on requests the rack limiter has already blocked. The 429 does not change: the call runs in its own guard, so a raise is tracked and the block still returns. - Each Puma worker runs one more thread. It waits on a queue and writes the counts every 15 minutes, then once more on a graceful stop. - New lines in `pubsub-rails-inf-gprd`, at most workers x 96 flushes x (`max_pairs` + 1) a day. With 20,137 workers (2026-11-03 09:00 UTC) and `max_pairs` 5, that is 11.6M lines a day (inferred ceiling, not a forecast). - `max_pairs` is 5. A worker that blocks more than 5 pairs in one flush counts the rest on one `other` line, so in a broad attack the endpoint ranking can miss endpoints. A large `other` `block_count` is the sign to raise it, within the line ceiling the logging team accepts. - A worker killed before it flushes loses up to 15 minutes of its counts. This loses data, not requests. - During OOM kills or Puma worker timeouts, a low `block_count` can mean lost counts, not fewer blocks. ## Dashboards and queries - [Block summary lines](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.message:%22rate_limit_block_summary%22'),columns:!(json.endpoint_id,json.rate_limit_rule,json.block_count),sort:!(!(json.time,desc)))), for the line count and `sum(json.block_count)` - [Failed flushes](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.message:%22rate_limit_error%22%20and%20json.dropped_pairs:*'),columns:!(json.class_name,json.error_message,json.dropped_pairs),sort:!(!(json.time,desc)))) - [rate-limiting: Overview](https://dashboards.gitlab.net/d/rate-limiting-main/rate-limiting3a-overview) - The two Kibana links under Events, for exceptions with the flag on On Mimir gprd: ```promql # B: blocks LabKit reported sum(increase(gitlab_labkit_rate_limiter_checks_total{env="gprd", rate_limiter=~"rack_request.*", action="block"}[1h])) ``` ## Expected values - Lines today: 0. - B is about 85.0M blocks a day (observed, `increase(...[24h])` to 2026-11-03 00:00 UTC), almost all `rack_request`. - At N%, `sum(json.block_count)` over an hour should be about N% of B over the same hour, and never above it (inferred). The flag is checked per request. - So until 100%, `block_count` counts only part of the blocks. Do not compare it to the 429 rate before then. - Lines count distinct `(endpoint_id, rule)` pairs, not blocks. So lines a day at 5% may already be close to 100% (inferred). The 5% step is the real test of log volume. - The share of `endpoint_id: unknown`: no baseline yet. Record it at the first step. - Failed flushes: 0. ## Stop and roll back if - Any exception from the block summary shows in Sentry or in the Kibana exceptions link. - Any failed flush line. - `sum(json.block_count)` above B over the same hour. - An apdex or error-ratio alert fires for web, api or git during a step. - More than 483k lines in any hour. That is the 11.6M lines a day ceiling divided by 24 (inferred). ## Continue if All the above hold for 15 minutes or more at the current step, then go to the next one. At 5%, wait 24 hours, so the lines-a-day reading covers a full day. ## Events - [Exceptions with rate_limiter_endpoint_block_summary:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22rate_limiter_endpoint_block_summary:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with rate_limiter_endpoint_block_summary:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22rate_limiter_endpoint_block_summary:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_tenant-controls`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set rate_limiter_endpoint_block_summary 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set rate_limiter_endpoint_block_summary true --dev --pre --staging --staging-ref ``` **Production**, percentage rollout (wait at least 15 min between steps, 24h at 5%, watch dashboards): ``` /chatops gitlab run feature set rate_limiter_endpoint_block_summary <percentage> --actors ``` ## Before global rollout Confirm the relevant gotchas before going to 100%. See [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle). See [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete rate_limiter_endpoint_block_summary --dev --pre --staging --staging-ref --production ``` After the MR deploys, delete the old flag's records: ``` /chatops gitlab run feature delete rate_limiter_endpoint_block_counter --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set rate_limiter_endpoint_block_summary false # production /chatops gitlab run feature set rate_limiter_endpoint_block_summary false --dev --pre --staging --staging-ref # non-production ```