[FF] skip_protected_branch_push_cache -- skip per-ref protected-branch Redis cache on large pushes

Summary

Roll out the feature currently behind the skip_protected_branch_push_cache feature flag.

• DRI: @stanhu
• Team Slack channel: #g_source_code

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

Blast radius: the git push access-check path only. When the flag is on, BranchCheck matches protected branches in Ruby against the preloaded all_protected_branches instead of the per-ref ProtectedBranches::CacheService Redis cache. The web read path is untouched. No data-loss risk.

Risks to watch:

• Wrong protected? result — a mismatch between the Ruby match and the cached value would let a push be wrongly allowed or blocked. Watched by the cache-mismatch log and the exceptions link below.
• Added SQL latency on /internal/allowed from loading all_protected_branches per request. Watched by the push access-check latency query below.

Rollback is the flag off (see Rollback section).

Monitoring

Prometheus (links open Grafana Explore on Mimir - Gitlab Gprd, verified rendering):

• Primary confirmation — protected-branch cache hit/miss rate. The cache_hit="false" (miss) series is the push-heavy share and should drop as the flag ramps; cache_hit="true" (web read path) stays flat. It won't reach zero, since web reads still use the cache.

  sum by (cache_hit) (rate(redis_hit_miss_operations_total{env="gprd", cache_identifier="ProtectedBranches::CacheService#fetch"}[5m]))

• Cache-generation p95 in this path — time spent generating the value on a miss; sample count drops with the miss rate.

  histogram_quantile(0.95, sum by (le) (rate(redis_cache_generation_duration_seconds_bucket{env="gprd", cache_identifier="ProtectedBranches::CacheService#fetch"}[5m])))

• redis-cluster-cache service overview — confirm no regression on the backing Redis (apdex, CPU, ops). Service-level effect is small; only large pushes bypass.

Kibana (rails index):

• Redis-cache calls on the push access-check endpoint, heaviest first — on a large push, json.redis_cache_calls and json.duration_s drop once the flag is on.
• Cache-mismatch log (correctness guard) — any hit means the Ruby match and the cached value diverged.

The exceptions and FF-event links are in the Events section below.

Events

• Exceptions with skip_protected_branch_push_cache:1
• Events with skip_protected_branch_push_cache:1
• Error rate and other graphs by modifying the examples in the Visualization Library

Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging

Rollout

Run all production /chatops in #production and cross-post the results to #g_source_code. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set skip_protected_branch_push_cache 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set skip_protected_branch_push_cache true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set skip_protected_branch_push_cache <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss skip_protected_branch_push_cache true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com skip_protected_branch_push_cache true
/chatops gitlab run feature set --user=stanhu skip_protected_branch_push_cache true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

• Docs + version history updated
• Breaking changes announced, if any
• Change management issue opened, if required
• External API consumers handled with a fail-open mechanism, if applicable

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete skip_protected_branch_push_cache --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set skip_protected_branch_push_cache false                                         # production
/chatops gitlab run feature set skip_protected_branch_push_cache false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete skip_protected_branch_push_cache --dev --pre --staging --staging-ref --production  # remove entirely