[FF] application_rate_limiter_static_labkit_limiters -- Use static Labkit limiter objects for ApplicationRateLimiter

Summary

Roll out the feature currently behind the application_rate_limiter_static_labkit_limiters feature flag.

  • DRI: @hmerscher
  • Team Slack channel: #proj-ai-to-prod-rate-limits

What could go wrong?

ApplicationRateLimiter checks switch from dynamic hash-based limiters to pre-built static Labkit limiter objects. If a limiter is misconfigured or missing, rate limiting for that action could behave unexpectedly (allow too much or block too aggressively). Blast radius is limited to rate-limited endpoints; no data loss risk. Watch error rate and rate-limit dashboards.

Rollout

Run all production /chatops in #production and cross-post the results to #proj-ai-to-prod-rate-limits.

Non-production

/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters 10 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters 1 --actors
/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters 10 --actors
/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters 50 --actors
/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters 100 --actors
/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters true

Before global rollout

  • Docs + version history updated (n/a — internal implementation change)
  • No breaking changes for external API consumers
  • Change management issue opened, if required

Cleanup

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242446 19.2
/chatops gitlab run feature delete application_rate_limiter_static_labkit_limiters --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters false
/chatops gitlab run feature set application_rate_limiter_static_labkit_limiters false --dev --pre --staging --staging-ref
/chatops gitlab run feature delete application_rate_limiter_static_labkit_limiters --dev --pre --staging --staging-ref --production

Verification dashboards

Use these dashboards during each production rollout step. Wait at least 15 minutes between percentage changes and check for regressions around the ChatOps annotation timestamp.

  • Rate Limiting: Overview
    • Watch the Application::Labkit Rate Limiter section for block/allow patterns, configured limits, periods, and unexpected changes after each FF step.
  • General: Platform Triage
    • Watch broad platform health, especially error rate, latency, Apdex, and saturation around each rollout step.
    • Roll back if there is a clear platform regression correlated with the FF step.
Edited by Hercules Merscher