Remove the labkit rack shadow comparison machinery

Nidhey Indurkarrequested to merge
nindurkar/remove-labkit-rack-shadow-comparator into master

What

Removes the labkit rack shadow comparison machinery. Closes gitlab-com/gl-infra/production-engineering#29541 (closed).

The comparator existed to check labkit's block decision against Rack::Attack's during the migration. With Rack::Attack safelisted there is nothing left to compare against, so it ran on every request and recorded nothing. The comparator has produced no output since 2026-08-12, when GitLab.com went labkit-only.

Deletes:

  • Gitlab::RackAttack::LabkitRateLimit::Divergence and its spec
  • the gitlab_rate_limiter_labkit_rack_shadow_total metric it wrote
  • the log_labkit_rack_divergence ops flag that gated its sampled log
  • the record call path in Gitlab::Middleware::LabkitRackRateLimit

Why this does not alter behaviour

@reprazent asked to confirm this specifically, so setting out the reasoning rather than just asserting it.

The comparator is observation-only and runs after @app.call, so it never participates in the decision or the response. Deleting it is therefore behaviour-neutral by construction.

The part that deserves scrutiny is not the deletion but the small restructuring that came with it, in the middleware:

  • run no longer returns facts: in its hash. Only record consumed that key. The outbound path reads decision[:results], which is unchanged.
  • blocking_result is removed. Only record called it. Note this is not the same method as blocked?, which is still used by enforced_response.
  • The outbound guard collapses from two calls inside if decision to one call with the same if decision. The condition and the surviving call are identical.

Nothing in the inbound path changed: the request is still built, the facts still computed, every limiter still checked, and enforced_response still decides whether to short-circuit with the byte-identical 429.

Verification

spec/lib/gitlab/middleware/labkit_rack_rate_limit_spec.rb: 24 examples, 0 failures, run against a local cluster.

The middleware spec asserted on divergence.record in several places. Those examples were rewritten to assert the surrounding behaviour rather than deleted, so the outbound path keeps its coverage.

Searched the tree for Divergence, log_labkit_rack_divergence and labkit_rack_shadow_total: no remaining references. The remaining hits for the word "divergence" are ordinary prose in comments, such as "a known, accepted divergence", and are unrelated.

Deliberately kept

spec/support/rate_limiter_labkit_rack_shadow.rb stays. It forces the cohort feature flags off suite-wide, and those flags are removed separately in gitlab-com/gl-infra/production-engineering#29663. #29541 (closed) is explicit that this file waits for that work.

Dashboards

The shadow metric appears in no committed dashboard in gitlab-com/runbooks, so there is nothing to retire.

Merge request reports