Merge branch 'ashs/split-rate-limiter-metrics' into 'master'

its limit. For a `:log` rule that means shadow traffic surfaces to the caller

as `exceeded? == true` (with `action` still `:allow`) — visible, but unable to

block. A `:log`-only path that matches therefore returns `matched? == true`

and emits no `rule="unmatched"` metric.

and its check is counted as `checks_total{matched="true"}`.

```mermaid

flowchart TD

    Iter -->|yes| Match{rule.match<br/>all satisfied?}

    Match -->|no| Iter

    Match -->|yes| Skip{rule.action<br/>== :skip?}

    Skip -->|"yes (no Redis op)"| SkipEmit[Emit calls_total<br/>action=skip]

    Skip -->|"yes (no Redis op)"| SkipEmit[Emit rule_evaluations_total<br/>action=skip, result=skip]

    SkipEmit --> SkipReturn([Return matched=true<br/>action=:allow])

    Skip -->|no| Eval["INCR Redis counter<br/>(see Redis sequence below)"]

    Eval --> Build[Build Evaluation<br/>resolve limit/period]

    Build --> Add[Add evaluation to Result]

    Add --> Emit[Emit calls_total + limit/period gauges]

    Add --> Emit[Emit rule_evaluations_total<br/>+ limit/period gauges]

    Emit --> Act{"result.block?<br/>(:limit rule over limit)"}

    Act -->|yes| Return([Return Result<br/>action=:block])

    Act -->|"no (:log, or :limit under limit)"| Iter

    Iter -->|no more rules| Any{any rule<br/>evaluated?}

    Any -->|yes| ReturnFold([Return Result reporting<br/>most-constraining evaluation])

    Any -->|no| Unmatched[Emit calls_total<br/>rule=unmatched, action=allow]

    Unmatched --> ReturnUnmatched([Return matched=false<br/>action=:allow])

    Eval -. StandardError .-> Error[Emit errors_total<br/>log warn]

    Iter -->|no more rules| Return2([Return Result reporting<br/>most-constraining evaluation,<br/>or matched=false if none])

    Return --> Check[Emit checks_total<br/>action, matched, error]

    SkipReturn --> Check

    Return2 --> Check

    Eval -. StandardError .-> Error[Emit errors_total +<br/>checks_total error=true<br/>log warn]

    Error --> ReturnErr([Return error=true<br/>action=:allow])

```

were incremented. Those requests are counted but produce no verdict — the

fail-open trade-off is that a request is never blocked on a partial evaluation.

Note that `calls_total` is emitted **per matched rule**, so summing it by

`rate_limiter` counts rule evaluations, not requests.

`rule_evaluations_total` is emitted **per evaluated rule** (including matched

`:skip` rules); `checks_total` is emitted **exactly once per check**, whatever

path the evaluation takes — so summing `checks_total` by `rate_limiter` counts

requests through the limiter, and "no rule matched" is `matched="false"` on the

check rather than a placeholder rule.

Only matched rules are evaluated: a rule whose `match:` conditions are not

satisfied emits nothing to `rule_evaluations_total` (match-testing is not an

evaluation), so the counter has no "didn't match" population and needs no

label for it. A matched `count_distinct` rule skipped by the missing-key

fail-open also emits nothing here — it was never evaluated; that check is

visible via `checks_total{error="true"}` and the

`rate_limit_missing_count_distinct` log, which carries the rule name.

During the transition the deprecated `calls_total` counter is additionally

emitted at every point the diagram emits `rule_evaluations_total` (with its

historical per-rule semantics, including the `rule="unmatched"` placeholder

after the loop). It is not shown above to keep the diagram legible; see the

Metrics table below.

### Actions

The rule's `action` describes what the rule does; the result's `action`

describes the outcome — what the caller should do — and is only ever `:allow`

or `:block`. The counter is always incremented when a rule matches, except for

`:skip` rules, which never touch Redis:

| rule action | what it does                                  | exceeded? | result action | terminating?  |

|-------------|-----------------------------------------------|-----------|---------------|---------------|

| `:limit`    | count against the limit                       | no        | `:allow`      | no — continue |

| `:limit`    | count against the limit                       | yes       | `:block`      | yes — stop    |

| `:log`      | count against the limit (observability only)  | no        | `:allow`      | no — continue |

| `:log`      | count against the limit (observability only)  | yes       | `:allow`      | no — continue |

| `:skip`     | don't count (bypass)                          | n/a       | `:allow`      | yes — stop    |

or `:block`. The Redis counter is always incremented when a rule matches,

except for `:skip` rules, which never touch Redis. The `result` label on

`rule_evaluations_total` records what each evaluation decided:

| rule action | what it does                                  | exceeded? | result action | `result` label | terminating?  |

|-------------|-----------------------------------------------|-----------|---------------|----------------|---------------|

| `:limit`    | count against the limit                       | no        | `:allow`      | `allow`        | no — continue |

| `:limit`    | count against the limit                       | yes       | `:block`      | `block`        | yes — stop    |

| `:log`      | count against the limit (observability only)  | no        | `:allow`      | `allow`        | no — continue |

| `:log`      | count against the limit (observability only)  | yes       | `:allow`      | `log`          | no — continue |

| `:skip`     | don't count (bypass)                          | n/a       | `:allow`      | `skip`         | yes — stop    |

- `:limit` — when exceeded, `Result#action` is `:block` and evaluation

  terminates. Caller should reject the request (e.g. with HTTP 429). When under

  `Result#action` `:allow` **without any Redis operation**: nothing is

  counted, so `limit`, `period`, `characteristics`, and `count_distinct` are

  inert and the result carries no `info` (`to_response_headers` is `{}`).

  The match is still observable via `calls_total{action="skip"}`. Use this

  for bypasses.

  The match is still observable via `rule_evaluations_total{action="skip"}`.

  Use this for bypasses.

### Redis keys

The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`

(Redis connection failure, timeout, OOM in user-supplied callables, …) is

logged at WARN with `message: "rate_limit_error"` and returned as an error

`Result` (`matched?` false, `error?` true, `action` `:allow`). The

`gitlab_labkit_rate_limiter_errors_total` counter is incremented. The caller

logged at WARN with `error_type: "rate_limit_error"` and returned as an error

`Result` (`matched?` false, `error?` true, `action` `:allow`). The caller

should treat the request as allowed.

A failed-open `check` is still counted: it emits

`checks_total{action="allow", matched="false", error="true"}`, so the fraction

of checks that encountered an error is `checks_total{error="true"}` over

`checks_total`. `error="true"` also covers a check where a matched

`count_distinct` rule was skipped because the identifier was missing its

`count_distinct` key — that check completes (`action` and `matched` describe

its outcome as usual), so `error="true"` is not exclusively fail-open traffic.

The deprecated `gitlab_labkit_rate_limiter_errors_total` counter is still

incremented on every fail-open (whole-check and per-rule `count_distinct`),

and remains the only error metric for `peek`, which emits no `checks_total`

(peek must not inflate the per-check counter). The follow-up MR that removes

`errors_total` must first decide where `peek` errors go — a dedicated peek

metric, or logs only.

Metric emission itself is best-effort: a failure in the metrics stack never

alters the verdict or breaks fail-open, and is logged at WARN with

`error_type: "rate_limit_metrics_error"` (once per process, to avoid

flooding).

## Metrics

`Labkit::RateLimit::Metrics` emits the following Prometheus metrics through

`Labkit::Metrics::Client`:

| metric                                              | type    | labels                                       | meaning                                                              |

|-------------------------------------------------|---------|-------------------------------------|----------------------------------------------------------------------|

| `gitlab_labkit_rate_limiter_calls_total`        | counter | `rate_limiter`, `rule`, `action`    | One increment per counted rule (plus one per matched `:skip` rule). `action` is the rule-level outcome: `"allow"` (under limit), `"limit"` (blocking `:limit` rule), `"log"` (exceeded `:log` rule), `"skip"`. `rule="unmatched", action="allow"` when no rule matched. |

| `gitlab_labkit_rate_limiter_errors_total`       | counter | `rate_limiter`                      | Fail-open events (any `StandardError` in the labkit path).            |

|-----------------------------------------------------|---------|----------------------------------------------|----------------------------------------------------------------------|

| `gitlab_labkit_rate_limiter_checks_total`           | counter | `rate_limiter`, `action`, `matched`, `error` | Exactly one increment per `check` call, including fail-open. `action` is what the caller should do (`"allow"` or `"block"`); `matched` and `error` are `"true"`/`"false"`. |

| `gitlab_labkit_rate_limiter_rule_evaluations_total` | counter | `rate_limiter`, `rule`, `action`, `result`   | One increment per evaluated rule (plus one per matched `:skip` rule). `action` is the configured rule action (`"limit"`, `"log"`, `"skip"`); `result` is what the evaluation decided (`"allow"`, `"block"`, `"log"`, `"skip"` — see the Actions table). |

| `gitlab_labkit_rate_limiter_calls_total`            | counter | `rate_limiter`, `rule`, `action`             | **Deprecated** — superseded by `checks_total` + `rule_evaluations_total`. Historical per-rule counter: one increment per counted rule (plus one per matched `:skip` rule), `rule="unmatched", action="allow"` when no rule matched. Emitted unchanged during the transition. |

| `gitlab_labkit_rate_limiter_errors_total`           | counter | `rate_limiter`                               | **Deprecated** — use `checks_total{error="true"}`. Fail-open events (any `StandardError` in the labkit path); still the only error metric for `peek`. |

| `gitlab_labkit_rate_limiter_limit`                  | gauge   | `rate_limiter`, `rule`                       | Resolved limit at the last check (useful when `limit:` is callable). |

| `gitlab_labkit_rate_limiter_period_seconds`         | gauge   | `rate_limiter`, `rule`                       | Resolved period at the last check.                                   |

Because every matching rule is counted, a single `check` call can emit

**multiple** `calls_total` increments: one per counted rule (or a single

`rule="unmatched"` increment if nothing matched).

`sum by (rate_limiter) (rate(gitlab_labkit_rate_limiter_checks_total[5m]))` is

the request rate through a limiter — no exclusions or dedup needed. A single

`check` call emits **one** `checks_total` increment and as many

`rule_evaluations_total` increments as rules it evaluated (possibly zero).

**Transition:** the deprecated `calls_total` and `errors_total` counters keep

emitting exactly as before this split, so existing dashboards and alerts stay

correct while consumers migrate to the new counters. Both are removed in a

follow-up major release once nothing consumes them.

## Dev/test vs production guards

      def check(identifier, cost: 1, rule_context: nil)

        cursor = RuleCursor.new

        check_rules(identifier, cost, rule_context, cursor)

        result = check_rules(identifier, cost, rule_context, cursor)

      rescue StandardError => e

        # Intentionally broad: fail-open applies to any unexpected error (network,

        # timeout, OOM) not only Redis protocol errors.

        report_error_metrics

        log_error(e, identifier, cursor.rule)

        Result.error

        result = Result.error

      ensure

        # StandardError-safe emission, so it cannot mask a propagating error.

        # result is nil when a non-StandardError unwinds: emit nothing then.

        report_check_metrics(result) if result

      end

      # Read-without-increment counterpart to {#check}. Same matching and Result

      # - :skip terminates on match without touching Redis. No counter is

      #   incremented, so the branch sits before the count_distinct check

      #   (identifier completeness is irrelevant to a rule that builds no key).

      #   calls_total still increments so the bypass stays observable.

      #   rule_evaluations_total still increments so the bypass stays observable.

      # - a :limit rule over its limit terminates, because the request is

      #   rejected and later rules cannot change that. Rules declared after it

      #   are neither counted nor evaluated, so a blocked request debits every

      # most-constraining evaluation (ranking lives in Result::Evaluation#<=>).

      # cost is therefore debited from every matching rule, not just the first.

      #

      # Metrics: each evaluated rule emits rule_evaluations_total (plus the

      # deprecated per-rule calls_total); the per-check checks_total is

      # emitted once in #check. Full contract in the README's Metrics section.

      #

      # SET-mode rules (rule.count_distinct set) that match but whose identifier

      # is missing the count_distinct key fail open + log + bump errors_total, and

      # the loop continues to the next rule (the rule is treated as not applicable

      # rather than aborting the whole evaluation).

      # is missing the count_distinct key fail open + log + bump errors_total +

      # flag the Result, and the loop continues to the next rule (the rule is

      # treated as not applicable rather than aborting the whole evaluation).

      #

      # Error handling stays whole-check (see #check): a raise part-way through

      # discards the results of the rules already evaluated, even though their

          if rule.count_distinct && missing_count_distinct_value?(rule, identifier)

            log_missing_count_distinct(rule, identifier)

            report_error_metrics

            result.degraded!

            next

          end

          evaluation = evaluate_rule(rule, identifier, cost, rule_context)

          result.add_evaluation(evaluation)

          report_matched_metrics(evaluation)

          report_evaluation_metrics(evaluation)

          return result if result.block?

        end

        report_unmatched_metrics unless result.matched?

        result

      rescue StandardError => e # binds e for the ensure

        raise

      ensure

        # Keep the rule attributed while an exception unwinds; clear it on

        # every normal exit (early returns included).

        cursor.rule = nil unless e

      end

      # Mirror of check_rules without metrics or writes. :log rules are read

          return result if result.block?

        end

        cursor.rule = nil

        result

      rescue StandardError => e # binds e for the ensure

        raise

      ensure

        cursor.rule = nil unless e

      end

      def rule_matches?(rule, identifier)

      # loop reached one, or after it finished - so the field is logged as null

      # rather than omitted, the same way identifier is. A named rule is the rule

      # whose match or evaluation raised.

      # Never raises: a logging failure must not break fail-open.

      def log_error(error, identifier, rule = nil)

        @logger.warn(

          name: @name,

          Labkit::Fields::ERROR_MESSAGE => error.message,

          identifier: identifier&.to_h

        )

      rescue StandardError

        nil

      end

      def log_missing_count_distinct(rule, identifier)

        )

      end

      def report_matched_metrics(evaluation)

        Metrics.calls_total.increment(

          rate_limiter: @name,

          rule: evaluation.rule.name,

          action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s

        )

        Metrics.limit_gauge.set(

          { rate_limiter: @name, rule: evaluation.rule.name },

          evaluation.info.resolved_limit

      def report_evaluation_metrics(evaluation)

        rule_labels = { rate_limiter: @name, rule: evaluation.rule.name }

        Metrics.safe_increment(

          :rule_evaluations_total,

          rule_labels.merge(action: evaluation.rule.action.to_s, result: evaluation_result(evaluation)),

          logger: @logger

        )

        Metrics.period_gauge.set(

          { rate_limiter: @name, rule: evaluation.rule.name },

          evaluation.info.resolved_period

        # Deprecated dual emission - remove together with Metrics.calls_total.

        Metrics.safe_increment(

          :calls_total,

          rule_labels.merge(action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s),

          logger: @logger

        )

        Metrics.safe_set(:limit_gauge, rule_labels, evaluation.info.resolved_limit, logger: @logger)

        Metrics.safe_set(:period_gauge, rule_labels, evaluation.info.resolved_period, logger: @logger)

      end

      # An exceeded :log rule reports "log" rather than the "allow" the caller

      # sees, so shadow rules over their limit stay visible.

      def evaluation_result(evaluation)

        return "allow" unless evaluation.exceeded?

        evaluation.rule.action == :limit ? "block" : "log"

      end

      # calls_total carries action="skip" (the rule action, not the :allow the

      # caller sees) so bypass traffic stays distinguishable from counted

      # allows. No limit/period gauges: a skip rule has no limit to report.

      def report_skipped_metrics(rule)

        Metrics.calls_total.increment(

          rate_limiter: @name,

          rule: rule.name,

          action: "skip"

        Metrics.safe_increment(

          :rule_evaluations_total,

          { rate_limiter: @name, rule: rule.name, action: "skip", result: "skip" },

          logger: @logger

        )

        # Deprecated dual emission - remove together with Metrics.calls_total.

        Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: rule.name, action: "skip" }, logger: @logger)

      end

      # Deprecated dual emission - remove together with Metrics.calls_total.

      def report_unmatched_metrics

        Metrics.calls_total.increment(

        Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: "unmatched", action: "allow" }, logger: @logger)

      end

      def report_check_metrics(result)

        Metrics.safe_increment(

          :checks_total,

          {

            rate_limiter: @name,

          rule: "unmatched",

          action: "allow"

            action: result.action.to_s,

            matched: result.matched?.to_s,

            error: (result.error? || result.degraded?).to_s

          },

          logger: @logger

        )

      end

      def report_error_metrics

        Metrics.errors_total.increment(rate_limiter: @name)

        Metrics.safe_increment(:errors_total, { rate_limiter: @name }, logger: @logger)

      end

    end

  end

# frozen_string_literal: true

require "concurrent-ruby"

module Labkit

  module RateLimit

    module Metrics

      # Process-wide once-latch for log_failure; specs reset via make_false.

      FAILURE_LOGGED = Concurrent::AtomicBoolean.new(false)

      module_function

      # Emitted once per *matched rule*, not once per check: every rule that

      # matches is evaluated, so summing this by rate_limiter counts rule

      # evaluations rather than requests. rule="unmatched", action="allow" is

      # emitted only when no rule matched at all.

      # Metric emission must never affect the caller's outcome. Resolving the

      # metric by name keeps a raising getter inside the rescue.

      def safe_increment(counter, labels, logger: nil)

        public_send(counter).increment(**labels) # rubocop:disable GitlabSecurity/PublicSend

      rescue StandardError => e

        log_failure(e, counter, labels, logger)

      end

      # Gauge counterpart of safe_increment.

      def safe_set(gauge, labels, value, logger: nil)

        public_send(gauge).set(labels, value) # rubocop:disable GitlabSecurity/PublicSend

      rescue StandardError => e

        log_failure(e, gauge, labels, logger)

      end

      # make_true returns true only for the flipping caller, so exactly one

      # warn per process. Never raises - a raise here would defeat the

      # callers' rescues.

      def log_failure(error, metric, labels, logger)

        return unless logger && FAILURE_LOGGED.make_true

        logger.warn(

          name: labels[:rate_limiter],

          metric: metric.to_s,

          Labkit::Fields::ERROR_TYPE => "rate_limit_metrics_error",

          Labkit::Fields::CLASS_NAME => error.class.to_s,

          Labkit::Fields::ERROR_MESSAGE => error.message

        )

      rescue StandardError

        nil

      end

      # Emitted exactly once per #check call, including calls that fail open;

      # summing by rate_limiter gives the request rate through the limiter.

      def checks_total

        Labkit::Metrics::Client.counter(

          :gitlab_labkit_rate_limiter_checks_total,

          'Total number of rate limit checks',

          { rate_limiter: nil, action: nil, matched: nil, error: nil }

        )

      end

      # Deprecated: superseded by checks_total and rule_evaluations_total.

      # Emitted unchanged (once per matched rule, rule="unmatched" when none

      # matched) until consumers migrate; will get removed in a follow-up release.

      def calls_total

        Labkit::Metrics::Client.counter(

          :gitlab_labkit_rate_limiter_calls_total,

        )

      end

      # Emitted once per *evaluated* rule. Only matched rules are evaluated:

      # Unmatched traffic is checks_total{matched="false"}

      def rule_evaluations_total

        Labkit::Metrics::Client.counter(

          :gitlab_labkit_rate_limiter_rule_evaluations_total,

          'Total number of rate limit rule evaluations',

          { rate_limiter: nil, rule: nil, action: nil, result: nil }

        )

      end

      # Deprecated: superseded by checks_total{error="true"}. Still emitted

      # because it remains the only error metric for #peek.

      def errors_total

        Labkit::Metrics::Client.counter(

          :gitlab_labkit_rate_limiter_errors_total,

    #               false). Other rules may also have matched and been counted;

    #               see #evaluations.

    # error?      - true if Redis was unavailable; result fails open (exceeded? is false)

    # degraded?   - true if a matched count_distinct rule was skipped by the

    #               missing-key fail-open path; the check still completed

    # info        - Result::Info with per-window counters for the reported rule;

    #               nil when matched? is false, error?, or the matched rule is

    #               :skip (no counter exists)

        @evaluations = []

        @skip_rule = nil

        @error = error

        @degraded = false

        @most_constraining = nil

      end

        @error

      end

      # Unlike error?, the check keeps going: the flag marks the outcome as

      # degraded without changing it.

      def degraded!

        @degraded = true

        self

      end

      def degraded?

        @degraded

      end

      def action

        block? ? :block : :allow

      end