Add sampled divergence log to labkit rack shadow

What does this MR do and why?

Adds a temporary, sampled structured log line for divergent block decisions in the labkit rack shadow middleware. The existing gitlab_rate_limiter_labkit_rack_shadow_total counter labels each data point only by whichever side blocked, so it cannot distinguish an identity-taxonomy defect from window-phase skew. Each logged divergence now carries, side by side:

  • both stacks' verdicts (labkit_blocked / rackattack_blocked)
  • every matched labkit rule with its action, count, and limit (shows where labkit routed the request, including synthetic skip/bypass matches)
  • every Rack::Attack throttle annotation with its discriminator, count, and limit (a throttle absent here means Rack::Attack did not count the request at all)
  • the labkit identity facts (requester_type, requester_id, runner_id, remote_ip, path, method)

This is the first unchecked item in the diagnosis plan for the parity divergence classes blocking the per-cohort enforce gates: it separates the api over-blocking class from the git-LFS window-phase class mechanically, and pins the remaining identity-attribution asymmetry that metrics cannot distinguish.

Volume is operator-controlled: logging is gated by the new default-off ops flag log_labkit_rack_divergence, sampled per request via percentage-of-actors on Feature.current_request, and only divergences (never matches) are logged. With the flag off the only new work per divergence is one Feature.enabled? call.

References

How to set up and validate locally

  1. Enable a shadow cohort flag and the log flag: Feature.enable(:rate_limiter_use_labkit_rack_cohort_2), Feature.enable(:log_labkit_rack_divergence) (or Feature.enable_percentage_of_actors(:log_labkit_rack_divergence, 1) for sampling).
  2. Drive requests past one stack's limit and watch log/application_json.log for Labkit rack shadow divergence lines.
Edited by Max Woolf

Merge request reports

Loading