[FF] use_labkit_git_basic_auth_ban -- route the git basic auth ban through labkit

Summary

Roll out the feature currently behind the use_labkit_git_basic_auth_ban feature flag.

  • DRI: @nindurkar
  • Team Slack channel: #g_tenant-controls

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

Routes the Git and container registry authentication IP ban through Labkit::RateLimit instead of Rack::Attack::Allow2Ban. Introduced in !252213 (merged).

Three deliberate behaviour differences, all agreed with @reprazent:

  • The ban lands one attempt later. Allow2Ban banned when the count reached maxretry; labkit blocks only once the count is above the limit.
  • The counting window is anchored on the first write rather than a wall-clock bucket, so attempts spread across a bucket boundary now accumulate where they previously did not. Quantifying this is why the counter in !250836 (merged) shipped first.
  • register_fail! returns true on the attempt that creates the ban rather than the one after, which moves the "threshold exceeded" auth log to ban creation. On the old path that log almost never fired.

Blast radius: failed Git and container registry authentication only. gitlab-ci-token is exempt on both paths, and the IP allowlist is applied before either. In-flight bans are abandoned at the flip, since the two implementations use different Redis keys, up to bantime which is 15 minutes on GitLab.com.

Watch gitlab_rate_limiter_git_basic_auth_ban_events_total, which both paths emit, so before and after are directly comparable. Note ban and already_banned are rare per process and are registered lazily, so use sum(metric) rather than rate or increase on those two.

Rollout

Run all production /chatops in #production and cross-post the results to #g_tenant-controls. Background: incremental rollout process, feature actors.

Always pass --actors. Without it ChatOps sets a percentage of time, which is a fresh coin flip on every call. This code asks the flag more than once per request (banned?, then register_fail!), so a per-call coin flip could read the ban from Rack::Attack and write the failure into labkit within one request. The flag uses Feature.current_request as the actor precisely to keep one request on one implementation.

For the same reason, the --project, --group and --user targeting in the template cannot work for this flag. The actor is the request, not a user or project. Percentage is the only lever.

Non-production

/chatops gitlab run feature set use_labkit_git_basic_auth_ban 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set use_labkit_git_basic_auth_ban true --dev --pre --staging --staging-ref

Do not move to production until rate_limiter="git_basic_auth" appears in the staging metrics. That is the proof the flag took effect, rather than just that ChatOps accepted it.

Production — post a dashboard screenshot as a comment before each increment, showing the current step is healthy.

  • /chatops gitlab run feature set use_labkit_git_basic_auth_ban 5 --actors
  • /chatops gitlab run feature set use_labkit_git_basic_auth_ban 10 --actors
  • /chatops gitlab run feature set use_labkit_git_basic_auth_ban 25 --actors
  • /chatops gitlab run feature set use_labkit_git_basic_auth_ban 50 --actors
  • /chatops gitlab run feature set use_labkit_git_basic_auth_ban 75 --actors
  • /chatops gitlab run feature set use_labkit_git_basic_auth_ban 100 --actors

Rollback at any point:

/chatops gitlab run feature set use_labkit_git_basic_auth_ban false

What to watch

These two prove the new path is live and healthy:

sum(gitlab_labkit_rate_limiter_checks_total{rate_limiter="git_basic_auth"})
sum(gitlab_labkit_rate_limiter_checks_total{rate_limiter="git_basic_auth", error="true"})

The first is currently absent, because this limiter has never run in production. It must become non-zero after enabling. The second must stay at zero: a rising error count means the limiter is failing open and bans have stopped working.

Volume, but not attribution:

sum by (event) (gitlab_rate_limiter_git_basic_auth_ban_events_total)

increment_ban_metric is called with the same event names on both the Rack::Attack and labkit branches, with no label distinguishing them, so this counter shows volume but cannot tell you which implementation ran.

Use sum, never rate or increase. These counters are registered lazily and pods churn, so the rate functions are wildly wrong here: increase(...[24h]) reports about 49M failures against a counter whose cumulative value is about 1.2M.

Baseline captured 2026-09-01 before rollout, flag off:

event count
reset 12,933,567
failure 1,219,322
blocked 129,809
ban 14
already_banned 3

Bans are genuinely rare, order tens per day fleet-wide. Seeing zero bans at 5% or 10% for a few hours is expected and is not evidence of a problem.

Roll back if: error="true" climbs on the git_basic_auth limiter, blocked or git-auth 403s rise beyond the baseline trend, or anyone reports legitimate git operations returning 403.

Expected behaviour changes

These are intended and were agreed in #29557. They are not faults.

  1. The ban lands one attempt later than Rack::Attack::Allow2Ban, because labkit blocks when the count is above the limit rather than when it reaches it.
  2. The counting window anchors on first write instead of a wall-clock bucket, so in-flight counters reset once at the moment of the flip.
  3. The "threshold exceeded" auth log starts firing. On the old path it needed a concurrent request to have created the ban first, so it was effectively unreachable. Expect that line to go from roughly zero to real volume.

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete use_labkit_git_basic_auth_ban --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set use_labkit_git_basic_auth_ban false                                         # production
/chatops gitlab run feature set use_labkit_git_basic_auth_ban false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete use_labkit_git_basic_auth_ban --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Nidhey Indurkar