## Summary Roll out the per-user rate limit on issue and work item moves and clones. The limit is the `work_item_move_limit` application setting (30 moves and clones every minute for each user, labeled beta) and one key covers the REST API, GraphQL and the web UI. The key is a `staged_rule`, so it runs in [log mode](https://docs.gitlab.com/development/application_limits/#roll-out-a-gitlabapplicationratelimiter-key-in-log-mode) from the deploy: requests are counted and the ones over the limit are logged but not blocked. The `work_item_move_rate_limit_enforce` feature flag turns on enforcement. Introduced in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/268181 for https://gitlab.com/gitlab-org/gitlab/-/work_items/637476. - DRI: daniyalAD - Team Slack channel: `#g_work_items` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? - Automation that moves or clones issues above 30 times each minute with one token gets `429` responses. The customer impact record in https://gitlab.com/gitlab-org/gitlab/-/work_items/634694 is checked before the flag goes to 100%. - The issue page shows a bare `429` response when a move or clone is throttled. There is no friendly error yet. - From the deploy, every move and clone adds one Redis rate limit check, on every instance, with no flag to turn it off. Watch Redis rate limiting latency after the deploy. If it is a problem, revert the MR. - A bulk move is charged one request per item and, once enforced, rejected up front when it has more items than the limit. The child items moved with an issue are charged too, but never rejected. - Rollback: `/chatops gitlab run feature set work_item_move_rate_limit_enforce false`. The limit goes back to log mode: requests are still counted and the ones over the limit are logged, but nothing is blocked. - To see who would be limited, search the Rails logs in https://log.gprd.gitlab.net for `json.rate_limit_state: "applimiter_work_item_move:limit_work_item_moves_by_user_log:log"` while the flag is off, and for `json.rate_limit_state: "applimiter_work_item_move:limit_work_item_moves_by_user:block"` once it is enforced. The `json.meta.user` field of the same line says who made the request. The metric `gitlab_labkit_rate_limiter_rule_evaluations_total` with `rule="limit_work_item_moves_by_user_log"` and `result="log"` shows the rate. ## Events - [Exceptions with work_item_move_rate_limit_enforce:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22work_item_move_rate_limit_enforce:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with work_item_move_rate_limit_enforce:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22work_item_move_rate_limit_enforce:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_work_items`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). The flag is checked for the user who moves or clones, so `--actors` rolls it out by user. ### Step 1: review log mode Log mode starts with the deploy. Review it for at least a few days before step 2: - [ ] Busiest users in the `_log` rule reviewed, `work_item_move_limit` raised if needed - [ ] Customer impact checked in https://gitlab.com/gitlab-org/gitlab/-/work_items/634694 ### Step 2: enforce **Non-production** ``` /chatops gitlab run feature set work_item_move_rate_limit_enforce 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set work_item_move_rate_limit_enforce true --dev --pre --staging --staging-ref ``` Check on staging that a move past the limit returns `429` through the API, GraphQL and the UI, and that a parent move still moves its children. **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set work_item_move_rate_limit_enforce <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss work_item_move_rate_limit_enforce true /chatops gitlab run feature set --group=gitlab-org,gitlab-com work_item_move_rate_limit_enforce true /chatops gitlab run feature set --user=daniyalAD work_item_move_rate_limit_enforce true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable - [ ] Confirm the customer impact record in https://gitlab.com/gitlab-org/gitlab/-/work_items/634694 still shows no caller above the limit - [ ] Add the move and clone limit (30 requests each minute for a user) to the "Current rate limits" table in `doc/user/gitlab_com/rate_limits.md` - [ ] Mention the limit in the move and clone sections of the issue user docs - [ ] Open a frontend follow-up to show a friendly error instead of the bare `429` response on the issue page move and clone actions ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, replace `staged_rule` with `Labkit::RateLimit::Rule.new` and `action: :limit` for `work_item_move`, and drop the `enforced?` gate in `WorkItems::BulkMoveService#move_limit`. Then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete work_item_move_rate_limit_enforce --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set work_item_move_rate_limit_enforce false # production /chatops gitlab run feature set work_item_move_rate_limit_enforce false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete work_item_move_rate_limit_enforce --dev --pre --staging --staging-ref --production # remove entirely ```