Add sampled divergence log to labkit rack shadow
What does this MR do and why?
Adds a temporary, sampled structured log line for divergent block decisions in the labkit rack shadow middleware. The existing gitlab_rate_limiter_labkit_rack_shadow_total counter labels each data point only by whichever side blocked, so it cannot distinguish an identity-taxonomy defect from window-phase skew. Each logged divergence now carries, side by side:
- both stacks' verdicts (
labkit_blocked/rackattack_blocked) - every matched labkit rule with its action, count, and limit (shows where labkit routed the request, including synthetic skip/bypass matches)
- every Rack::Attack throttle annotation with its discriminator, count, and limit (a throttle absent here means Rack::Attack did not count the request at all)
- the labkit identity facts (
requester_type,requester_id,runner_id,remote_ip,path,method)
This is the first unchecked item in the diagnosis plan for the parity divergence classes blocking the per-cohort enforce gates: it separates the api over-blocking class from the git-LFS window-phase class mechanically, and pins the remaining identity-attribution asymmetry that metrics cannot distinguish.
Volume is operator-controlled: logging is gated by the new default-off ops flag log_labkit_rack_divergence, sampled per request via percentage-of-actors on Feature.current_request, and only divergences (never matches) are logged. With the flag off the only new work per divergence is one Feature.enabled? call.
References
- https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362
- Rollout issue: #605893
How to set up and validate locally
- Enable a shadow cohort flag and the log flag:
Feature.enable(:rate_limiter_use_labkit_rack_cohort_2),Feature.enable(:log_labkit_rack_divergence)(orFeature.enable_percentage_of_actors(:log_labkit_rack_divergence, 1)for sampling). - Drive requests past one stack's limit and watch
log/application_json.logforLabkit rack shadow divergencelines.