## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/627274) currently behind the `rate_limit_job_erase` feature flag. - DRI: @srajadas - Team Slack channel: `#g_pipeline_execution` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag adds per-user rate limits to job erasure. The worst case is rejecting an erase that should have been allowed. There is no data loss. Both buckets are keyed on the calling user, so one actor cannot exhaust another user's budget. Only the user-facing entry points are throttled: the REST endpoint `POST /api/:version/projects/:id/jobs/:job_id/erase` and the web UI. Housekeeping paths are not affected. Watch the `429` rate on `json.route.keyword: "/api/:version/projects/:id/jobs/:job_id/erase"` with `json.method: POST`, and the `json.rate_limiting_gates` field for `job_erase` and `job_erase_per_project`. Rejections should be close to zero at the current thresholds. Any rejection is a signal to inspect the actor before raising the limit. The per-project threshold is the `job_erase_limit_per_user_project` application setting (default 600), so it can be tuned without a deploy. The per-job limit is fixed at 5 per minute and needs a code change. One nuance: the rate limit is counted before the permission check, so a call that gets a `403` still counts against the user's buckets. This is on purpose. ## Events - [Exceptions with rate_limit_job_erase:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22rate_limit_job_erase:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with rate_limit_job_erase:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22rate_limit_job_erase:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production commands in #production and cross-post the results to #g_pipeline_execution. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` gitlab run feature set rate_limit_job_erase 50 --actors --dev --pre --staging --staging-ref gitlab run feature set rate_limit_job_erase true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` gitlab run feature set rate_limit_job_erase <percentage> --actors ``` Or target specific actors instead: ``` gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss rate_limit_job_erase true gitlab run feature set --group=gitlab-org,gitlab-com rate_limit_job_erase true gitlab run feature set --user=srajadas rate_limit_job_erase true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then: ``` gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/259439 19.5 gitlab run feature delete rate_limit_job_erase --dev --pre --staging --staging-ref --production ``` ## Rollback ``` gitlab run feature set rate_limit_job_erase false # production gitlab run feature set rate_limit_job_erase false --dev --pre --staging --staging-ref # non-production gitlab run feature delete rate_limit_job_erase --dev --pre --staging --staging-ref --production # remove entirely ```