[FF] `rate_limit_pipeline_retry` -- Add a rate limit to pipeline retry

Summary

Roll out the feature currently behind the rate_limit_pipeline_retry feature flag.

DRI: @srajadas
Team Slack channel: #g_pipeline_execution

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

Blast radius is bounded: the flag only adds two rate limits to pipeline retry, so the worst case is rejecting retries that should have been allowed. No data loss. Both buckets are keyed on the calling user, so one actor cannot exhaust another user's budget.

Watch the 429 rate on json.route.keyword: "/api/:version/projects/:id/pipelines/:pipeline_id/retry" and the json.rate_limiting_gates field for pipeline_retry and pipeline_retry_per_project. Rejections should be close to zero at the current thresholds. Any rejection is a signal to inspect the actor before raising the limit.

The per-project threshold is the pipeline_retry_limit_per_user_project application setting (default 200), so it can be tuned without a deploy. The per-pipeline limit is fixed at 5 and needs a code change.

Events

Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging

Rollout

Run all production /chatops in #production and cross-post the results to #g_pipeline_execution. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set rate_limit_pipeline_retry 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set rate_limit_pipeline_retry true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set rate_limit_pipeline_retry <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss rate_limit_pipeline_retry true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com rate_limit_pipeline_retry true
/chatops gitlab run feature set --user=srajadas rate_limit_pipeline_retry true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Docs + version history updated
Breaking changes announced, if any
Change management issue opened, if required
External API consumers handled with a fail-open mechanism, if applicable

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254367 19.4
/chatops gitlab run feature delete rate_limit_pipeline_retry --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set rate_limit_pipeline_retry false                                         # production
/chatops gitlab run feature set rate_limit_pipeline_retry false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete rate_limit_pipeline_retry --dev --pre --staging --staging-ref --production  # remove entirely

[FF] rate_limit_pipeline_retry -- Add a rate limit to pipeline retry