# [#INC-10255: Redis primary CPU saturation on redis-sidekiq nodes approaching capacity limit](https://app.incident.io/gitlab/incidents/10255) Generated by John Jarvis on 20 May 2026 00:10. All timestamps are local to Etc/UTC # Key Information | Metric | Value | | ------ | ------ | | Customers Affected | All customers using audit event streaming — audit event records were dropped globally for ~3 hours | | Requests Affected | All `AuditEvents::AuditEventStreamingWorker` jobs globally dropped between 15:48–18:54 UTC | | Incident Severity | ~"severity::2" | | Impact Start Time | Tue, 19 May 2026 15:47:00 UTC| | Impact End Time | Tue, 19 May 2026 18:54:00 UTC | | Total Duration | 3 hours, 7 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22163 | # Summary **Problem**: The primary Redis node for the redis-sidekiq `catchall_b` shard exceeded CPU and memory limits due to a surge in `AuditEvents::AuditEventStreamingWorker` jobs. This was a near-miss repeat of the [S1 incident on 2026-05-12](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22103) where Redis Sidekiq memory exhaustion broke cluster quorum. **Impact**: For approximately 3 hours (15:48–18:54 UTC), audit event jobs were dropped globally. This resulted in missing audit event records for all customers using audit event streaming. No broader service degradation occurred because the team intervened before memory exhaustion could cause a Redis cluster failure. **Causes**: A surge in audit event jobs overwhelmed the redis-sidekiq `catchall_b` shard. Deferring the jobs (toggled at 12:47 UTC) appears to have created a feedback loop where `sidekiq` itself became the dominant source of job enqueues, particularly from the `catchall_b` shard. Redis memory grew from normal levels to 35 GiB before intervention. Redis's single-threaded architecture limited CPU scaling, compounding the resource bottleneck. The root cause is the same `AuditEvents::AuditEventStreamingWorker` that caused the [prior S1](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22103), and we are tracking corrective actions under the same [Feature Change Lock](https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/97). **Response strategy**: Switched from deferring to dropping audit event jobs globally, which quickly reduced load and resolved CPU and memory saturation. Audit event processing was resumed after approximately 3 hours, restoring normal operations. # What went well? 1. The team recognized the pattern from the [prior S1 incident](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22103) and acted quickly to prevent a repeat outage. Matt Smiley identified the memory growth trend early and escalated, enabling mitigation before Redis memory exhaustion could break cluster quorum. 2. The decision to switch from deferring to dropping audit event jobs was made decisively and was the correct call — it immediately halted memory growth and avoided a full Redis cluster failure that would have caused widespread S1-level impact. 3. Thorough post-incident analysis by Matt Smiley identified a potential feedback loop in the job deferral mechanism, where deferring `AuditEvents::AuditEventStreamingWorker` jobs caused sidekiq itself to re-enqueue more of them, amplifying the problem. This is a valuable finding for preventing recurrence. 4. Clear timeline documentation (by John Jarvis) and detailed metric analysis (by Matt Smiley) during the incident made it straightforward to understand the chain of events and root cause. # What was difficult? 1. The job deferral mechanism appears to have a feedback loop that amplifies the problem it is meant to mitigate. Deferring `AuditEvents::AuditEventStreamingWorker` jobs led to sidekiq itself becoming the dominant source of new enqueues for that same job class, causing Redis memory to grow continuously rather than stabilize. 2. Roughly half of the audit event jobs during the demand spike had `null` namespace metadata (jobs authenticated via tokens), making it difficult to attribute the source of the surge and determine whether disabling streaming for specific namespaces would be effective. 3. This is a near-miss repeat of the S1 incident from one week earlier — the same worker, the same Redis shard, the same failure mode. The corrective actions from the [FCL](https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/97) (circuit breaker, runbook, Redis streaming buffer) had not yet been fully deployed when this recurrence happened. 4. The initial triage set severity at S3, and it took over 3 hours before it was upgraded to S2 and broader escalation occurred. Earlier severity assessment could have brought more resources to bear sooner. **Capacity Planning considerations:** - The redis-sidekiq `catchall_b` shard is vulnerable to memory exhaustion from job backlogs. This incident and its S1 predecessor demonstrate that a single misbehaving job class can saturate the shard. - The [circuit breaker MR (!235349)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235349) and [Redis streaming buffer plan (#599701)](https://gitlab.com/gitlab-org/gitlab/-/work_items/599701) from the FCL are the planned mitigations. # Investigation Details <details> <summary> Timeline</summary> # Incident Timeline **2026-05-19** **06:30** Audit events toggled on for the `gitlab-org` namespace ([production change summary](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22125#note_3361913172)). **12:47** Audit event jobs deferred via feature flag ([feature flag log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/work_items/58935)) as part of INC-10250 where `catchall-b` had a large queue of jobs. This appears to have triggered a feedback loop where sidekiq itself became the dominant source of new `AuditEvents::AuditEventStreamingWorker` enqueues. **14:13** Audit events toggled off for the `gitlab-org` namespace. **14:36** Prometheus Alertmanager alert reported, incident opened in triage. Acknowledged by Silvester Wainaina. **15:42** Matt Smiley identified memory usage growth on the `catchall_b` Redis shard. **15:47** **Impact start** — CPU and memory saturation on the Redis primary node for `catchall_b`. **15:53** Matt Smiley switched from deferring to dropping `AuditEvents::AuditEventStreamingWorker` jobs globally via feature flag ([feature flag log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/work_items/58952)). **15:56** Memory usage began dropping. Matt Smiley noted: "I think we avoided another sev1." **16:23** Memory usage continuing to drop as the accumulated job queue shrank. Severity set to S3, status to Investigating. **16:30** CPU saturation resolved. Bob Van Landuyt confirmed the backlog was cycling out. **17:19** Status update posted noting this was the first time audit event jobs were dropped globally (vs. selectively for specific namespaces). **18:54** **Impact end** — Audit event processing resumed globally ([feature flag log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/work_items/58966)). **18:57** Severity upgraded to S2. Escalated to GitLab.com Production and IMOC. Acknowledged by Alex Ives and Matt Smiley. **19:15–23:00** Post-incident analysis by Matt Smiley identified the deferral feedback loop: enqueue rate from sidekiq itself jumped from ~20 jobs/sec to hundreds after the deferral FF was toggled, correlating with Redis memory growth. The `catchall-b` shard was the dominant source of self-enqueues. **23:39** Incident resolved, entered post-incident flow. </details> # Follow-ups | Follow-up | Owner | | --- | --- | | [Prevent job deferral of AuditEvents::AuditEventStreamingWorker](https://app.incident.io/gitlab/incidents/10255?tab=post-incident) | John Jarvis | | [Implement Redis streaming buffer for audit event streaming](https://app.incident.io/gitlab/incidents/10255?tab=post-incident) | Unassigned | This incident shares corrective actions with the [S1 incident (22103)](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22103) under [FCL #97](https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/97). Key in-progress items: - [Add per-destination circuit breaker (!235349)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235349) — @harsimarsandhu - [Create runbook for Redis Sidekiq memory saturation (#600089)](https://gitlab.com/gitlab-org/gitlab/-/work_items/600089) — @huzaifaiftikhar1 - [Redis streaming buffer plan (#599701)](https://gitlab.com/gitlab-org/gitlab/-/work_items/599701) — @harsimarsandhu - [Re-enable audit event streaming for gitlab-org (#29044)](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29044) — @nrosandich # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock). - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [ ] S1 incidents [require a public RCA within 7 days](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/#timeline-that-we-expect-for-reviews-to-be-completed) of the incident. If this review cannot be made public, [create a separate public RCA](https://handbook.gitlab.com/handbook/engineering/root-cause-analysis/#how-to-perform-an-rca). - [x] Close the review before the due date - [x] Go back to the incident channel or page and close out the remaining post-incident tasks