[FF] log_labkit_rack_divergence -- sampled divergence log for the labkit rack shadow

Summary

Operational rollout tracking for the log_labkit_rack_divergence ops flag introduced in !245389 (merged).

The flag gates a temporary sampled diagnostic log for the labkit rack shadow: one application_json.log line per divergent block decision between labkit and Rack::Attack, carrying both stacks' verdicts, counters, and discriminators. It exists to root-cause the parity divergence classes tracked in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362, which block the per-cohort enforce gates.

  • DRI: @mwoolf
  • Team Slack channel: #g_production_engineering

Note

This is not a feature release. The flag is a sampling dial for a diagnostic log line: it will be raised for bounded diagnosis windows, returned to off between them, and removed together with the log once https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362 is diagnosed. "Stable at 100%" is explicitly not the goal. Feature flag controls

What could go wrong?

  • Log volume. Only divergent decisions are logged (never matches), currently a few hundred per second fleet-wide at worst, dominated by throttle_authenticated_git_lfs during active LFS throttling events. 100% sampling during such an event is the worst case; prefer raising the percentage outside LFS abuse events or keeping windows short. Watch the divergence-by-throttle dashboard (linked from the work item) to predict line rate before raising.
  • No user-facing behavior. The logging happens on the outbound path after the response is decided, inside the middleware's guard: a logging failure is tracked to error tracking and the request is unaffected (verified by unit spec and GDK smoke test).
  • Sampling granularity is global. Percentage-of-actors applies per request across all fleets; there is no per-fleet targeting. Getting a useful sample of the low-rate ai-assisted divergence (~0.2/s) requires a higher global percentage for a few hours, which oversamples the high-rate classes at the same time.

Rollout

Run all production /chatops in #production and cross-post results to the team channel.

Non-production (verify line shape in logs before production):

/chatops gitlab run feature set log_labkit_rack_divergence true --staging

Production (percentage of actors; each step is a bounded diagnosis window, not a ratchet):

/chatops gitlab run feature set log_labkit_rack_divergence 1 --actors
  1. Start at 1%, confirm line rate and shape in Kibana (json.message: "Labkit rack shadow divergence").
  2. Raise to 25-100% for a bounded window (hours) to capture the ai-assisted and api identity samples needed for https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362.
  3. Return to false between diagnosis windows:
/chatops gitlab run feature set log_labkit_rack_divergence false

Cleanup

The flag and the log line it gates are removed together once the divergence classes have documented root causes and dispositions (the diagnosis-plan exit criteria in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362). Remove the flag, the log_divergence path in lib/gitlab/rack_attack/labkit_rate_limit/divergence.rb, and the YAML definition, then:

/chatops gitlab run feature delete log_labkit_rack_divergence --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set log_labkit_rack_divergence false