## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/636584) currently behind the `enforce_audit_streaming_destination_limits` feature flag. - DRI: @andrew.jung - Team Slack channel: `#sec-compliance` > [!note] > Process and guidance live in the docs. This issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability. It is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? - The MR is !267270. With the flag off, nothing is blocked and over-limit calls are only logged. With the flag on, two limits are enforced on audit streaming destinations. - A group can have at most 50 destinations in total, inactive ones included. The 51st create is rejected with "Maximum number of external audit event destinations (50) exceeded, including inactive ones. Delete unused destinations to add more." Only create is checked. Existing destinations can still be updated and deleted. - `groupAuditEventStreamingDestinationsCreate` and `groupAuditEventStreamingDestinationsUpdate` share a limit of 10 calls per user per minute. Over the limit they return "This endpoint has been requested too many times. Try again later." and write nothing. - Who could be affected: an integration or script that creates or updates destinations in bursts over 10 a minute per user, or a group that legitimately wants more than 50 destinations. On 2026-11-02 the most destinations any one group had was 11, and the mutations peaked at about 13 calls a day. - Watch two logs. Both are written whether the flag is on or off. `auth_json.log`: `json.message: "Application_Rate_Limiter_Request"` with `json.env: "audit_event_streaming_destination_mutation_request_limit"`. `application_json.log`: `json.message: "Audit streaming destination total limit exceeded"` (has `group_id`, `model`, `limit`). - No data loss. The limits only reject new writes. Rollback is turning the flag off. ## Events - [Exceptions with enforce_audit_streaming_destination_limits:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22enforce_audit_streaming_destination_limits:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with enforce_audit_streaming_destination_limits:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22enforce_audit_streaming_destination_limits:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#sec-compliance`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). Let the log-only behavior run on GitLab.com for about a week and confirm both log queries above are empty or near empty. Then enable the flag and remove it. The code checks the flag with the destination's top-level group as the actor, so rollout can target groups or a percentage of actors. **Non-production** ``` /chatops gitlab run feature set enforce_audit_streaming_destination_limits 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set enforce_audit_streaming_destination_limits true --dev --pre --staging --staging-ref ``` **Production**: percentage rollout. Wait at least 15 minutes between steps and watch the logs above. ``` /chatops gitlab run feature set enforce_audit_streaming_destination_limits <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --group=gitlab-org,gitlab-com enforce_audit_streaming_destination_limits true ``` ## Before global rollout Confirm the relevant gotchas before going to 100%, see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle), see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete enforce_audit_streaming_destination_limits --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set enforce_audit_streaming_destination_limits false # production /chatops gitlab run feature set enforce_audit_streaming_destination_limits false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete enforce_audit_streaming_destination_limits --dev --pre --staging --staging-ref --production # remove entirely ```