## Summary Switches Free-plan MCP Server traffic from log-only counting to enforcement: requests over 60 a minute get rejected with a 429. Second of three issues; follows #631323 and precedes #631325. Tracking issue #629883, epic `gitlab-org&22800`, milestone 19.5. - DRI: `@terrichu` - Team Slack channel: `#g_agent_execution` > [!note] > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What the flags do | Flag | Effect when enabled | | --- | --- | | `rate_limiter_mcp_limits_free_enforce` | `burst_mcp_traffic_per_user_free_plan` (the `:limit` rule) starts matching instead of its `_log` twin: Free-plan MCP traffic over 60 a minute gets rejected with a 429 | The Premium and Ultimate `_enforce` flags stay off; those plans keep logging only until #631325. ## Prerequisites - [ ] #631323 completed and the `burst_mcp_traffic_per_user_free_plan_log` counters reviewed, so the 60-a-minute limit is known not to reject normal Free-plan traffic - [ ] `rate_limiter_mcp_limits_free_info` still enabled (from #631323); the `:limit` rule needs it true alongside `_enforce` ## What could go wrong - Matching switches from the `_log` rule's counter key to the `:limit` rule's counter key for Free-plan traffic; still one rate-limiting Redis key per requester per minute, same as during observe. - Labkit fails open: if rule evaluation errors, the request still goes through. - Free-plan users sending more than 60 MCP requests a minute start getting 429s. 60 a minute is the tightest limit of the three plans, so this is the plan most likely to surface a client that polls too aggressively. ## Rollout Per [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), run in `#production` and cross-post to the team channel. No percentage step here: the actor is `Feature.current_request`, so a partial flip would reject a user on some requests and let the same user through on others, at the same request rate. Enforcement is only coherent at 0% or 100%. The flag goes straight from off to fully on. Non-production: ``` /chatops gitlab run feature set rate_limiter_mcp_limits_free_enforce true --dev --pre --staging --staging-ref ``` Production: ``` /chatops gitlab run feature set rate_limiter_mcp_limits_free_enforce true ``` ## Verify Grafana: the [rate-limiting detail dashboard](https://dashboards.gitlab.net/d/rate-limiting-detail/rate-limiting-detail), `Application::Labkit Rate Limiter` row. The tier-aware panels added by gitlab-com/runbooks!11499 read `rate_limiter="rack_request"` with `rule=~".*_traffic_per_.*"`, so the MCP rules show up as their own series with no dashboard change. Enforcement is live, and `result="block"` is the 429 count. There is no dedicated per-throttle 429 counter, and `gitlab_labkit_rate_limiter_enforced_total` does not exist in gprd. ```promql sum by (rule, result) (sli_aggregations:gitlab_labkit_rate_limiter_rule_evaluations_total:rate_5m{env="gprd", rate_limiter="rack_request", rule=~"burst_mcp_traffic_per_user_free_plan"}) ``` Expect `block` to appear and the `_log` twin to fall away. The `block` rate should sit at or below the `log` rate measured in #631323 — materially higher means the flip changed which requests match, not just what happens to them. Fail-open stays flat: ```promql sum(gitlab_component_errors:rate_5m{env="gprd", type="rate-limiting", component="rate_limiter_checks"}) ``` Kibana: on `pubsub-rails-inf-gprd*`, `json.rate_limit_state` carries one `<limiter>:<rule>:<result>` string per counted rule. ``` json.path:"/api/v4/mcp" and json.rate_limit_state:*burst_mcp* ``` A rejected request never reaches Grape, so a 429 produces no `rate_limit_state` entry. Use Kibana for what still gets through and Prometheus for what is blocked. **Roll back if:** the fail-open rate rises, `redis-cluster-ratelimiting` saturation moves, or the `block` rate exceeds the log-mode share from #631323. A flag flip does not reach every Puma worker at once, so give it a minute before reading counters. Observed on a GDK: straight after disabling an enforce flag, a 65-request run split 30 on the enforcing rule and 35 on the `_log` twin; re-run once the flag cache had settled it was 65 on the twin and nothing on the enforcing rule. A mixed first minute after each flip is expected, not a bug. Runbook: [Tier-aware throttles (labkit)](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/rate-limiting/README.md#tier-aware-throttles-labkit). ## Rollback Production: ``` /chatops gitlab run feature set rate_limiter_mcp_limits_free_enforce false ``` Non-production: ``` /chatops gitlab run feature set rate_limiter_mcp_limits_free_enforce false --dev --pre --staging --staging-ref ``` ## Cleanup Remove per [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up): ``` /chatops gitlab run feature delete rate_limiter_mcp_limits_free_enforce --dev --pre --staging --staging-ref --production ``` Hold off until #631325 is also stable. The YAML removal for `rate_limiter_mcp_limits_free_enforce` happens together with the other five MCP flags in one cleanup MR, tracked on #631325.