# Production Change ## Change Summary Re-enable audit event streaming for the `gitlab-org` namespace, which was disabled in [production#22114](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22114) to prevent a repeat of [production#22103](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22103). The circuit breaker MR [gitlab-org/gitlab!235349](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235349) has been merged and deployed to production. This change restores full audit event coverage for the `gitlab-org` namespace. Related issue: [production-engineering#29044](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29044) ## Change Details 1. **Services Impacted** - Audit event streaming for the `gitlab-org` namespace; Sidekiq `catchall-b` shard 1. **Change Technician** - @nrosandich 1. **Change Reviewer** - @sabrams 1. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-05-19 06:30 1. **Time tracking** - ~15 minutes (including monitoring period) 1. **Downtime Component** - None expected > [!IMPORTANT] > If your change involves scheduled maintenance, add a step to set and > [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) > per our runbooks. This will make sure SLA calculations adjust for the maintenance period. ## Preparation > [!NOTE] > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist ~C3: - [ ] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [x] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [ ] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [ ] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps > [!NOTE] > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are > listed below. - [ ] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [ ] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [ ] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. ### Change steps - steps to take to execute the change *Estimated Time to Complete (mins)* - 5 - [x] Set label ~"change::in-progress" - [x] Navigate to the GitLab Admin UI for the `gitlab-org` namespace audit event streaming settings - [x] Toggle audit event streaming back on - [x] Confirm audit event streaming is active and monitor dashboards for anomalies - [ ] Set label ~"change::complete" ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change *Estimated Time to Complete (mins)* - 5 - [ ] Navigate to the GitLab Admin UI for the `gitlab-org` namespace audit event streaming settings - [ ] Toggle audit event streaming back off - [ ] Notify the team and open a new incident if needed - [ ] Set label ~"change::aborted" ## Monitoring ### Key metrics to observe - Metric: Sidekiq apdex SLO (`catchall-b` shard) - Location: [Sidekiq Overview - Catchall-b](https://dashboards.gitlab.net/goto/afmiavudv3rb4b?orgId=1) - What changes to this metric should prompt a rollback: Apdex dropping below SLO threshold (as seen in [production#22114](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22114)) - Metric: Sidekiq queue depth - Location: [Queue Overview](https://dashboards.gitlab.net/goto/dfmiazgb4pmv4f?orgId=1) - What changes to this metric should prompt a rollback: Significant queue depth increase on `catchall-b` shard - Metric: Error rates - Location: [Error Rate](https://dashboards.gitlab.net/goto/ffmib1qmlo2kgd?orgId=1) - What changes to this metric should prompt a rollback: Elevated error rates following re-enabling of audit event streaming