## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/636123) currently behind the `rate_limit_context_commits_write` feature flag. The flag enables a per-user rate limit of 10 requests per minute, shared by the REST endpoints that create and delete merge request context commits (!267499). The **Add or remove previously merged commits** modal uses the same endpoints. - DRI: @egrieff - Team Slack channel: `#<team channel>` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? - **Blast radius:** only `POST` and `DELETE /projects/:id/merge_requests/:merge_request_iid/context_commits`. No data changes, and no data loss. - **Risk:** a user or integration that legitimately sends more than 10 requests a minute gets `429 Too Many Requests`. In the modal, a limited save shows the existing "Please try again" alert, unless the save both adds and removes commits and only one request is limited. Then the page reloads without an alert. In 3 days of GitLab.com logs, the most requests one user sent in a minute was 2. - **What to watch** after each step, in the Rails logs: - 429s on the two routes: ``` json.meta.caller_id:("POST /api/:version/projects/:id/merge_requests/:merge_request_iid/context_commits" or "DELETE /api/:version/projects/:id/merge_requests/:merge_request_iid/context_commits") and json.status:429 ``` - Throttled requests in `auth.log`: `json.message:"Application_Rate_Limiter_Request" and json.env:"context_commits_write_request_limit"` - If real users keep hitting the limit, roll back and revisit the threshold. ## Events - [Exceptions with rate_limit_context_commits_write:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22rate_limit_context_commits_write:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with rate_limit_context_commits_write:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22rate_limit_context_commits_write:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#<team channel>`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). These endpoints get very little traffic (about 14 requests a day on GitLab.com), so short percentage steps would show almost nothing. Leave each production step on for at least a day. **Non-production** ``` /chatops gitlab run feature set rate_limit_context_commits_write 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set rate_limit_context_commits_write true --dev --pre --staging --staging-ref ``` - [ ] On staging, confirm that the 11th request within a minute returns `429`, and that the modal still saves. **Production** - [ ] Enable for the DRI and check the modal and the API: ``` /chatops gitlab run feature set --user=egrieff rate_limit_context_commits_write true ``` - [ ] 50% of users, then check the logs above after a day: ``` /chatops gitlab run feature set rate_limit_context_commits_write 50 --actors ``` - [ ] 100% of users, then check the logs above after a day: ``` /chatops gitlab run feature set rate_limit_context_commits_write 100 --actors ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete rate_limit_context_commits_write --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set rate_limit_context_commits_write false # production /chatops gitlab run feature set rate_limit_context_commits_write false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete rate_limit_context_commits_write --dev --pre --staging --staging-ref --production # remove entirely ```