### Problem When a request carries the Rack::Attack bypass header (`ENV['GITLAB_THROTTLE_BYPASS_HEADER']` set to `'1'`), `ApplicationRateLimiter.throttled_request?` short-circuits and returns `false` before the labkit adapter path is reached ([`application_rate_limiter.rb:203`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/application_rate_limiter.rb#L203)): ```ruby def throttled_request?(request, current_user, key, scope:, **options) if ::Gitlab::Throttle.bypass_header.present? && request.get_header(Gitlab::Throttle.bypass_header) == '1' return false end throttled?(key, scope: scope, **options) end ``` Bypassed requests are completely invisible to labkit's Redis counters and Prometheus metrics. In contrast, Rack::Attack's safelist logs the bypass via `throttle_safelist` in `production_json.log`, so operators can still observe bypass traffic. As we migrate rate limiting to labkit (Stage 2a, https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28808), bypass traffic becomes a blind spot: we cannot answer "how much bypassed traffic would have been throttled by labkit?" — a question that matters for capacity planning, shadow validation, and abuse detection. Stage 2b (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28852) already addresses this for Rack::Attack at the middleware layer with a high-priority `bypass_header_present` `:log` rule. This issue covers the equivalent gap in the `ApplicationRateLimiter` controller/API layer. ### Proposal Labkit supports an `action: :allow` rule type ([`labkit-ruby` rule.rb:16-18](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/blob/main/lib/labkit/rate_limit/rule.rb#L16-L18)). An `:allow` rule: - Increments the Redis counter and emits Prometheus metrics - **Terminates evaluation** on match — no subsequent `:block` rule is evaluated ([`evaluator.rb:53`](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/blob/main/lib/labkit/rate_limit/evaluator.rb#L53)) - Never causes a block, regardless of whether the counter exceeds the limit For each rate limit registered in `SupportedRateLimits`, define a companion `:allow` rule. When `throttled_request?` detects the bypass header, it passes the header value into the labkit identifier. Labkit's first-match-wins evaluation handles the rest. This gives us: 1. **Visibility**: bypass traffic is counted in labkit's Redis counters and Prometheus metrics (`gitlab_labkit_rate_limiter_calls_total{action="allow"}`). 2. **No wasted work**: the `:allow` action terminates evaluation immediately — the `:block` rule is never reached, no extra Redis round-trips. 3. **Shadow validation**: during the shadow phase, bypass traffic participates in the labkit divergence counter, improving confidence before flipping enforce. 4. **Static rule definition**: the `:allow` rules are defined statically in `SupportedRateLimits`, aligning with the direction in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28853 where each limiter has its rules defined declaratively. 5. **Parity with Stage 2b**: matches the pattern Stage 2b uses for Rack::Attack bypass observability at the middleware layer. ### Implementation sketch The bypass header value is passed into the labkit identifier as a regular characteristic. No new adapter method is needed — the existing `run!` path handles everything. 1. In `SupportedRateLimits`, each entry gains a companion `:allow` rule ordered **before** the existing `:block` rule. The `:allow` rule uses `match: { bypass: "1" }` to only fire when the bypass header is present. The naming convention could be `bypass_<rule_name>` (e.g. `bypass_limit_ai_actions_by_user`). ```ruby ai_action: { limiter_name: 'applimiter_ai_action', rules: [ { rule_name: 'bypass_limit_ai_actions_by_user', characteristics: %i[user], match: { bypass: '1' }, action: :allow }, { rule_name: 'limit_ai_actions_by_user', characteristics: %i[user], action: :block } ], flag_scope: :cohort_2 } ``` 2. Remove the bypass short-circuit from `throttled_request?` for labkit-handled keys. Instead, pass the bypass header value into the identifier so labkit's first-match-wins handles it: ```ruby def throttled_request?(request, current_user, key, scope:, **options) bypass_value = if ::Gitlab::Throttle.bypass_header.present? request.get_header(Gitlab::Throttle.bypass_header) end throttled?(key, scope: scope, bypass: bypass_value, **options).tap do |throttled| log_request(request, :"#{key}_request_limit", current_user) if throttled end end ``` For keys handled by labkit, the `bypass` value flows into the identifier. The `:allow` rule matches when `bypass: "1"`, terminates evaluation, and emits metrics — the request is never blocked. For keys still on the legacy path, `_throttled?` checks the `bypass` kwarg and returns `false` (preserving existing behaviour). 3. The Redis key shapes are naturally disjoint. Labkit's `_unknown_` sentinel fills the `bypass` characteristic when the header is absent, producing keys like `labkit:rl:...:bypass:_unknown_` for normal traffic vs `labkit:rl:...:bypass:1` for bypass traffic. The counters never interfere. ### Scope - Only affects `throttled_request?` callers: controllers via [`check_rate_limit.rb:11`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/concerns/check_rate_limit.rb#L11) and Grape API via [`api/helpers/rate_limiter.rb:13`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/helpers/rate_limiter.rb#L13). - Direct `throttled?` callers (e.g. internal API SSH path at `lib/api/internal/base.rb:74,79`) do not check the bypass header today — out of scope. ### Acceptance criteria - [ ] Bypassed requests flowing through `throttled_request?` increment labkit Redis counters and Prometheus metrics with `action="allow"` - [ ] Bypassed requests are never blocked (existing behaviour preserved) - [ ] The `:allow` rule terminates evaluation — no `:block` rule is evaluated for bypass traffic - [ ] Bypass header value is passed as a characteristic in the labkit identifier — no separate `record_bypass!` method - [ ] Redis key shapes for bypass vs normal traffic are disjoint (`:bypass:1` vs `:bypass:_unknown_`) - [ ] Shadow divergence counter includes bypass traffic when the shadow flag is on - [ ] Existing Rack::Attack safelist logging is unaffected - [ ] `:allow` rules are defined statically in `SupportedRateLimits` with `match: { bypass: "1" }` ### Related - MR that surfaced the question: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235588 - Stage 2a (ApplicationRateLimiter migration): https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28808 - Stage 2b (Rack::Attack migration, covers middleware-level bypass): https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28852 - Design: static config evolution: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28853 - Labkit `:allow` action: https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/blob/main/lib/labkit/rate_limit/rule.rb#L16-L18 - Bypass header source of truth: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/throttle.rb#L25-L30