# [#INC-10096: Error rates violating SLO](https://app.incident.io/gitlab/incidents/10096) Generated by Joe Burnett on 13 May 2026 00:05. All timestamps are local to Etc/UTC # Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | |--------|-------| | Customers Affected | All GitLab.com customers active during the impact period | | Requests Affected | >1 million requests ([source](https://dashboards.gitlab.net/d/general-triage/general3a-platform-triage?from=2026-05-12T16:58:17.376Z&orgId=1&timezone=utc&to=2026-05-12T18:35:30.501Z&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&viewPanel=panel-3)); 12 million delayed Sidekiq jobs resulted in pipeline creation going near-zero for extended periods, with an estimated >10,000 pipelines impacted | | Incident Severity | Severity 1 | | Impact Start Time | Tue, 12 May 2026 17:12:00 UTC | | Impact End Time | Tue, 12 May 2026 18:43:44 UTC | | Impact Duration | 1 hour, 31 minutes | | Total Duration | 4 hours, 15 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22103 | # Summary **Problem**: A major Redis Sidekiq cluster outage led to widespread elevated error rates, authentication failures, and job processing delays for all major GitLab.com services, including web, CI runners, and Git. **Impact**: Many users experienced frequent 500 errors, CI pipeline failures, stuck or delayed pipelines, and unreliable Git access. Error rates peaked at 5% for CI runners and affected almost all services, leading to a high volume of support tickets. At its worst, the Sidekiq queue backlog reached over 12 million jobs, causing pipeline creation and job processing to stall. **Causes**: Earlier mitigation for a previous incident led to audit event jobs being deferred, which then accumulated and consumed all available memory on the Redis Sidekiq cluster. This memory exhaustion resulted in kernel errors and caused multiple Redis nodes to fail, breaking the cluster quorum. **Response strategy**: Engineers performed targeted restarts and forced resets on failed Redis nodes, then sequentially resized all Redis VMs to double their memory and CPU. As of 18:40 UTC, all three nodes are upgraded, error rates have dropped, and the system is recovering. The large Sidekiq queue backlog is now draining and being monitored. Long-term improvements are planned for the problematic audit event worker to prevent recurrence. **⚠️ Recurring Contributing Factor**: `AuditEvents::AuditEventStreamingWorker` has caused at least **4 prior incidents** since March 2025, escalating in severity each time: * [#19532](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19532) — INC-310, S2, Mar 24 2025 (PGBouncer saturation) * [#19543](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19543) — INC-311, S2, Mar 25 2025 (repeat next day) * [#19690](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19690) — INC-489, S2, Apr 15 2025 (bad streaming destination) * This incident — INC-10096, **S1**, May 12 2026 (Redis memory exhaustion) # What went well? - **Fast mobilization and coordination**: We quickly pulled the right responders into a shared Zoom and coordinated across multiple teams (Production/SRE, Sidekiq, Runners, Release/Deploy, and CMOC). - **Effective mitigation once the failure mode was confirmed**: After we identified the redis-sidekiq catchall-b nodes were unhealthy/unresponsive, responders executed a clear recovery plan (controlled restarts/resets and then VM resizing for headroom), which correlated with service error rates stabilizing. - **Good technical triage discipline**: We used dashboards and logs to rule out a traffic surge (RPS flat/declining), helping focus on shared dependency and infrastructure-level failure modes rather than chasing load-driven explanations. - **Risk-aware decision making under pressure**: We explicitly paused on irreversible options (like queue flushing) and prioritized safer mitigations first. # What was difficult? - **Multi-symptom, cross-service impact made diagnosis noisy**: The incident presented as broad degradation (web, git, auth, CI runners, Sidekiq, internal API), which made it harder to quickly converge on a single causal narrative and communicate it crisply. - **Observability and discoverability gaps for key customer symptoms**: “Pipelines stuck in created” was hard to quantify and confirm in real time; relevant dashboards/SLIs either weren’t obvious to on-call responders or didn’t directly answer the operational question. - **Host-level unresponsiveness slowed execution**: SSH/login hangs and generally degraded VM behavior increased time-to-diagnose and increased operational risk while making changes. - **Alerting/routing friction**: Some signals existed but didn’t reliably reach the right on-call audiences (e.g., labeling/routing gaps), and responders called out that tooling/dashboards weren’t easily discoverable during the event. - **Difficult reliability tradeoffs in the datastore failure mode**: Redis-sidekiq behavior under memory saturation raised hard choices (availability vs. data loss) that are challenging to navigate during an active incident and point to needed guardrails/design improvements. # Investigation Details <details> <summary>Timeline</summary> # Incident Timeline **2026-05-12** **17:12:00** **Impact started at** Custom timestamp "Impact started at" occurred **17:15:10** **Incident reported in triage by Prometheus Alertmanager alert** Prometheus Alertmanager alert reported the incidentSeverity: NoneStatus: Triage **17:15:24** **Escalation acknowledged by Stephanie Jackson** Stephanie Jackson acknowledged the escalation **17:16:59** **Incident accepted** Stephanie Jackson shared an update Severity: ~~None~~ → Severity 2 Status: ~~Triage~~ → Investigating **17:16:59** **Escalated to GitLab.com Production** Your workflow manually escalated the incident to the escalation path [GitLab.com Production](https://app.incident.io/gitlab/on-call/escalation-paths/01K5YWH36WZZX2YT9HNRR0WMB0) **17:16:59** **Escalated to Incident Manager Oncall (IMOC)** Your workflow manually escalated the incident to the escalation path [Incident Manager Oncall (IMOC)](https://app.incident.io/gitlab/on-call/escalation-paths/01K8VN9TTAJH6N3WKNCAMNXQ40) **17:17:04** **Escalation acknowledged by Steve Abrams** Steve Abrams acknowledged the escalation **17:17:11** **Escalation acknowledged by Stephanie Jackson** Stephanie Jackson acknowledged the escalation **17:19:55** **Image posted by Stephanie Jackson** Stephanie Jackson posted an image to the channel source (https://dashboards.gitlab.net/d/git-main/git3a-overview?from=2026-05-12T11:19:00.000Z&to=2026-05-12T17:18:59.999Z&var-environment=gprd&var-stage=cny&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd)  **17:20:27** **Image posted by Terri Chu** Terri Chu posted an image to the channel I saw this when I was looking at my correlation ID, but didn't look much further https://log.gprd.gitlab.net/app/discover#/?\_tab=h@64474a0&\_g=h@2294574&\_a=h@07ba51a  **17:21:15** **Message from Stephanie Jackson** Stephanie Jackson's message was pinned by Alex Hanselka https://dashboards.gitlab.net/d/general-triage/general3a-platform-triage?orgId=1&from=n\[…\]S_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main (https://dashboards.gitlab.net/d/general-triage/general3a-platform-triage?orgId=1&from=now-1h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main) -- many things show increased error rate. **17:22:38** **Image posted by Stephanie Jackson** Stephanie Jackson posted an image to the channelMany things erroring: source (https://dashboards.gitlab.net/d/general-triage/general3a-platform-triage?orgId=1&from=now-1h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&viewPanel=panel-3)  **17:26:58** **Escalated to tier1 - Runners Platform - Escalation** Steve Abrams manually escalated the incident to the escalation path [tier1 - Runners Platform - Escalation](https://app.incident.io/gitlab/on-call/escalation-paths/01K7HSQ433CMD61V4RNS70BJ47) **17:32:49** **Severity upgraded from Severity 2 → Severity 1** Steve Abrams shared an update Severity: ~~Severity 2~~ → Severity 1 **17:32:50** **Escalated to Infrastructure Leadership** Your workflow manually escalated the incident to the escalation path [Infrastructure Leadership](https://app.incident.io/gitlab/on-call/escalation-paths/01K7QAHP7P28668C5MHTHKJM34) **17:32:56** **Escalation acknowledged by Steve Abrams** Steve Abrams acknowledged the escalation **17:33:46** **Update shared** Steve Abrams shared an updateEngineers have confirmed widespread elevated error rates and authentication failures affecting the git, CI runners, and web services. CI runners are showing error rates up to 5%, with the git workhorse API and web service also impacted. Users are experiencing frequent 500 errors, CI pipeline failures, and unreliable access to Git features.The team initially investigated Redis connectivity as a possible cause, but recent evidence also points to potential DNS service discovery issues or a broader disruption in the US East region. There is no ongoing rollout in production that would explain the issue; the only recent change was a registry-related rollout in staging.Multiple on-call teams have been paged, and incident severity has been escalated to S1. Engineers are coordinating across teams and analyzing dashboards, error logs, and service dependencies to isolate the root cause. Additional support is being sought in for external DNS expertise. **17:43:05** **Escalation acknowledged by Davis Bickford** Davis Bickford acknowledged the escalation **17:44:22** **Image posted by Stephanie Jackson** Stephanie Jackson posted an image to the channel source (https://dashboards.gitlab.net/d/redis-sidekiq-main/redis-sidekiq3a-overview?orgId=1&from=2026-05-12T11:44:00.000Z&to=2026-05-12T17:44:59.999Z&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-shard=catchall_b&var-shard-2=$\__all)  **17:46:49** **Message from Stephanie Jackson** Stephanie Jackson's message was pinned by Alex HanselkaComputer broken: ``` May 12 17:06:07 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: Finished Exports ubuntu_advantage metrics for use with the node_exporter. May 12 17:06:25 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: Starting Timer-run service to periodically run Redis RDB backups, unless this node is currently the primary... May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd redis_backup.sh[634708]: 2026-05-12 17:06:26 UTC [634708] LOG Replication lag is acceptable: 0 seconds May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd redis_backup.sh[634708]: 2026-05-12 17:06:26 UTC [634708] SKIP_BACKUP Redis role is primary and at least one replica is healthy. May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: redis-rdb-backup.service: Main process exited, code=exited, status=1/FAILURE May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: redis-rdb-backup.service: Failed with result 'exit-code'. May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: Failed to start Timer-run service to periodically run Redis RDB backups, unless this node is currently the primary. [16861909.855206] INFO: task kcompactd0:67 blocked for more than 120 seconds. [16861909.862237] Tainted: G O K 5.15.0-1083-gcp #92~20.04.1-Ubuntu [16861909.870013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 12 17:36:17 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.855206] INFO: task kcompactd0:67 blocked for more than 120 seconds. May 12 17:38:14 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.862237] Tainted: G O K 5.15.0-1083-gcp #92~20.04.1-Ubuntu May 12 17:40:02 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.870013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 12 17:41:54 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878211] task:kcompactd0 state:D stack: 0 pid: 67 ppid: 2 flags:0x00004000 May 12 17:43:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878222] Call Trace: May 12 17:44:26 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878226] <TASK> May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878231] __schedule+0x2cd/0x890 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878238] schedule+0x69/0x110 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878240] io_schedule+0x16/0x40 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878242] wait_on_page_bit_common+0x16e/0x3b0 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878246] ? filemap_invalidate_unlock_two+0x50/0x50 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878249] __lock_page+0x4c/0x60 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878251] migrate_pages+0xaf6/0xda0 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878253] ? move_freelist_tail+0xc0/0xc0 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878257] ? fast_isolate_freepages+0x7b0/0x7b0 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878259] compact_zone+0x658/0xe90 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878262] ? __compaction_suitable+0xa6/0x100 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878264] kcompactd_do_work+0x10b/0x280 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878267] kcompactd+0x310/0x370 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878270] ? wait_woken+0x60/0x60 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878273] ? kcompactd_do_work+0x280/0x280 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878275] kthread+0x127/0x150 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878277] ? set_kthread_struct+0x50/0x50 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878279] ret_from_fork+0x1f/0x30 May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878284] </TASK> ``` **17:48:47** **Message from Stephanie Jackson** Stephanie Jackson's message was pinned by Alex HanselkaRedis broken: ``` May 12 17:21:46 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:44Z","level":"warning","caller":"lite/lite.go:946","message":"SLOW TRANSACTION","component":"sqlite","duration":1771360449,"stack":"goroutine 108 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1() (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1())\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x5c\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction)(0xc002e10bb0, {0x55b7268b06a0, 0xc003130b90}, 0xc004c3fe20)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997) +0x1fd\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:73 +0xbc\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:46 +0xff\ncreated by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig (http://github.com/gravitational/teleport/lib/backend/lite.NewWithConfig) in goroutine 1\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266) +0x7c7\n"} May 12 17:21:47 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: redis-server.service: Found left-over process 2516163 (redis-server) in control group while starting unit. Ignoring. May 12 17:21:47 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. May 12 17:21:47 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:44Z","level":"warning","caller":"service/connect.go:751","message":"Connection problem: sync rotation state.","component":"proc:1","pid":"3734543.1","error":"connection error: desc = \"transport: Error while dialing: failed to dial: context deadline exceeded\""} May 12 17:21:48 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Started Run a redis-server process.. May 12 17:21:48 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Stopped Login Service. May 12 17:21:48 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Condition check resulted in Load Kernel Module drm being skipped. May 12 17:21:51 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Starting Login Service... May 12 17:21:53 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:49Z","level":"warning","caller":"service/connect.go:661","message":"Sync rotation state cycle failed.","component":"proc:1","pid":"3734543.1","retry_interval":10000000000} May 12 17:21:58 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"caller":"srv/heartbeatv2.go:583","component":null,"level":"warning","message":"Failed to perform inventory heartbeat for ssh server: inventory control msg not sent: EOF","timestamp":"2026-05-12T17:21:42Z"} May 12 17:21:59 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:58Z","level":"warning","caller":"labels/cloud.go:180","message":"Failed to fetch cloud tags","component":"cloudlabels","error":"Get \"https://compute.googleapis.com/compute/v1/projects/gitlab-production/zones/us-east1-d/instances/redis-sidekiq-catchall-b-02-db-gprd\": EOF"} May 12 17:22:00 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:58Z","level":"warning","caller":"labels/cloud.go:180","message":"Failed to fetch cloud tags","component":"cloudlabels","error":"Get \"https://compute.googleapis.com/compute/v1/projects/gitlab-production/zones/us-east1-d/instances/redis-sidekiq-catchall-b-02-db-gprd\": net/http: TLS handshake timeout"} May 12 17:22:00 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:22:00Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)" May 12 17:22:01 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:22:01Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)" May 12 17:22:01 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:22:01Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)" May 12 17:22:01 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:01Z","level":"info","caller":"service/state.go:112","message":"Detected Teleport component is running in a degraded state.","component":"proc:1","pid":"3734543.1","component":"node"} May 12 17:22:03 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:01Z","level":"warning","caller":"lite/lite.go:946","message":"SLOW TRANSACTION","component":"sqlite","duration":12021385705,"stack":"goroutine 108 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1() (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1())\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x5c\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction)(0xc002e10bb0, {0x55b7268b06a0, 0xc003130b90}, 0xc004c3fe20)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997) +0x1fd\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:73 +0xbc\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:46 +0xff\ncreated by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig (http://github.com/gravitational/teleport/lib/backend/lite.NewWithConfig) in goroutine 1\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266) +0x7c7\n"} May 12 17:22:04 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:22:04Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)" May 12 17:22:04 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:01Z","level":"warning","caller":"inventory/inventory.go:213","message":"Failed to create inventory control stream","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to dial: context deadline exceeded\""} May 12 17:22:10 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:08Z","level":"warning","caller":"lite/lite.go:946","message":"SLOW TRANSACTION","component":"sqlite","duration":3396583005,"stack":"goroutine 108 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1() (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1())\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x5c\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction)(0xc002e10bb0, {0x55b7268b06a0, 0xc003130b90}, 0xc004c3fe20)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997) +0x1fd\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:73 +0xbc\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:46 +0xff\ncreated by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig (http://github.com/gravitational/teleport/lib/backend/lite.NewWithConfig) in goroutine 1\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266) +0x7c7\n"} May 12 17:22:11 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:09Z","level":"info","caller":"service/connect.go:666","message":"Initiating new sync rotation state cycle after backoff.","component":"proc:1","pid":"3734543.1","backoff_time":14308371920} May 12 17:22:11 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"caller":"cache/cache.go:1307","component":"node:1:cache","level":"warning","message":"Re-init the cache on error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to dial: context deadline exceeded\"","timestamp":"2026-05-12T17:22:04Z"} May 12 17:22:12 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Reloading. May 12 17:22:12 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:12Z","level":"warning","caller":"lite/lite.go:946","message":"SLOW TRANSACTION","component":"sqlite","duration":1816520807,"stack":"goroutine 108 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1() (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1())\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x5c\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction)(0xc002e10bb0, {0x55b7268b06a0, 0xc003130b90}, 0xc004c3fe20)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997) +0x1fd\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:73 +0xbc\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:46 +0xff\ncreated by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig (http://github.com/gravitational/teleport/lib/backend/lite.NewWithConfig) in goroutine 1\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266) +0x7c7\n"} May 12 17:22:21 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:05Z","level":"info","caller":"client/alpn_conn_upgrade.go:88","message":"ALPN connection upgrade test failed.","address":"production.teleport.gitlab.net:443 (http://production.teleport.gitlab.net:443)","error":"dial tcp: lookup production.teleport.gitlab.net (http://production.teleport.gitlab.net): i/o timeout"} May 12 17:22:22 redis-sidekiq-catchall-b-02-db-gprd snapd[853973]: SIGABRT: abort May 12 17:22:57 redis-sidekiq-catchall-b-02-db-gprd snapd[853973]: PC=0x55d3dab938e1 m=0 sigcode=0 May 12 17:23:09 redis-sidekiq-catchall-b-02-db-gprd dbus-daemon[673]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 33308ms) May 12 17:23:13 redis-sidekiq-catchall-b-02-db-gprd snapd[853973]: goroutine 0 gp=0x55d3dc8997a0 m=0 mp=0x55d3dc89aa00 [idle]: May 12 17:23:50 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:23:26Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)" May 12 17:37:32 redis-sidekiq-catchall-b-02-db-gprd dbus-daemon[673]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service' requested by ':1.1172883' (uid=0 pid=2413205 comm="sshd: ahmad [priv] " label="unconfined") ``` **17:50:55** **Message from Raynard Omongbale** Raynard Omongbale's message was pinned by Sarah Walker ``` May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: redis-rdb-backup.service: Failed with result 'exit-code'. Failed to start Timer-run service to periodically run Redis RDB backups, unless this node is currently the primary. [16861909.855206] INFO: task kcompactd0:67 blocked for more than 120 seconds. ``` **17:59:17** **Image posted by Cameron McFarland** Cameron McFarland posted an image to the channel FYI for later: https://dashboards.gitlab.net/goto/eflv2dgnkl5ogc?orgId=1  **18:02:47** **Update shared** Steve Abrams shared an updateThe Redis Sidekiq catch-all cluster experienced a major outage due to multiple node failures, resulting in widespread elevated error rates, pipeline failures, and degraded service across git, CI runners, and web services. The outage was traced to resource exhaustion and kernel errors on the Redis hosts, which left only one node healthy and unable to maintain cluster quorum.Engineers performed targeted restarts and a forced reset of the two unresponsive Redis nodes to restore cluster capacity. As of 18:00 UTC, all three Redis nodes (O1, O2, O3) are up, with O2 acting as the current primary, restoring service redundancy.Teams are monitoring following the reset, but initially the nodes are immediately throwing errors again. **18:10:12** **Message from Rachel Nienaber** Rachel Nienaber's message was pinned by Sarah Walker Noting which one is stopping - Cameron is stopping `redis-sidekiq-catchall-b-01-db-gprd` **18:12:32** **Message from Rachel Nienaber** Rachel Nienaber's message was pinned by Sarah Walker Cameron is going up to the next machine type to `c2-standard-16` **18:16:16** **Message from Sarah Walker** Sarah Walker pinned their own messagecatchall-b-02 has been stopped - waiting... :waiting: **18:19:36** **Message from Sarah Walker** Sarah Walker pinned their own messageStopped - machine is being upsized - started **18:20:14** **Update shared** Steve Abrams shared an updateThe root cause was linked to audit event jobs that consumed all available memory after being deferred earlier in the day in inc-10084-sidekiq-queueing-apdex-slo-violation-on-shard-catchall-b-apdex-at. **18:25:51** **Message from Sarah Walker** Sarah Walker pinned their own messagecatchall-b-03 is now stopped **18:29:07** **Monitoring at** Custom timestamp "Monitoring at" occurred **18:29:07** **Identified at** Custom timestamp "Identified at" occurred **18:29:07** **Status changed from Investigating → Monitoring** Steve Abrams shared an update Status: ~~Investigating~~ → Monitoring **18:30:58** **Status changed from Monitoring → Fixing** Steve Abrams shared an update Status: ~~Monitoring~~ → Fixing **18:32:06** **Message from Raynard Omongbale** Raynard Omongbale's message was pinned by Sarah Walker https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq3a-shard-detail?orgId=1&fro\[…\]-gprd&var-environment=gprd&var-stage=main&var-shard=$\__all (https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq3a-shard-detail?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-shard=$\__all) https://dashboards.gitlab.net/d/alerts-sat_redis_primary_cpu/alerts3a-redis-primary-cpu\[…\]=&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd (https://dashboards.gitlab.net/d/alerts-sat_redis_primary_cpu/alerts3a-redis-primary-cpu3a-saturation-detail?from=now-12h&to=now&var-environment=gprd&var-type=redis-sidekiq&var-stage=&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd) **18:35:42** **Image posted by Stan Hu** Stan Hu posted an image to the channel FYI https://log.gprd.gitlab.net/app/r/s/mbKw6 shows in the last 15 minutes about 14% of these workers came from `gitlab-org` projects:   **18:36:07** **Message from Rachel Nienaber** Rachel Nienaber's message was pinned by Sarah Walker @Steve Abrams This is the second S2 incident involving this worker this month. The previous one was INC-8169. There was also an S3 in December. As part of the incident review, the team who owns this worker needs to suggest alternatives to reduce the pressure that this worker places on the system that should be then done as part of the FCL for them. **18:37:38** **Image posted by Raynard Omongbale** Raynard Omongbale posted an image to the channel Source (https://dashboards.gitlab.net/d/alerts-sat_redis_primary_cpu/alerts3a-redis-primary-cpu3a-saturation-detail?from=2026-05-12T15:36:17.330Z&to=2026-05-12T18:36:17.330Z&var-environment=gprd&var-type=redis-sidekiq&var-stage=&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&refresh=5m&viewPanel=panel-2)  **18:38:08** **Image posted by Raynard Omongbale** Raynard Omongbale posted an image to the channel  **18:38:48** **Message from Stephanie Jackson** Stephanie Jackson's message was pinned by Sarah Walker @kam https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?orgId=1&from=now-1h&to=\[…\]d&var-environment=gprd&var-stage=main&var-shard=catchall-b (https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?orgId=1&from=now-1h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-shard=catchall-b) **18:41:08** **Image posted by Stephanie Jackson** Stephanie Jackson posted an image to the channelCurrent queue: source (https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?orgId=1&from=2026-05-12T17:40:42.773Z&to=2026-05-12T18:40:42.773Z&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-shard=catchall-b)  **18:43:44** **Fixed at** Custom timestamp "Fixed at" occurred **18:43:44** **Status changed from Fixing → Monitoring** Steve Abrams shared an update Status: ~~Fixing~~ → Monitoring **18:58:53** **Image posted by Raynard Omongbale** Raynard Omongbale posted an image to the channel @stejacks Just to confirm based on https://gitlab.slack.com/archives/C101F3796/p1778610325539689 We were able to re-enabled `AuditEventStreamingWorker` and not deferred it. My reasons being that the `AuditEventStreamingWorker` numbers are climbing. might be due to the earlier deferrals but it is worth shouting out that that queue is climbing. I will check back on it. source (https://dashboards.gitlab.net/goto/eflv7ql311rswf?orgId=1)  **19:10:21** **Image posted by Stan Hu** Stan Hu posted an image to the channel Job queue has dropped from a peak of 12.2 million jobs to under 7 million: https://dashboards.gitlab.net/goto/eflv8rgw4gnpcf?orgId=1  2h later **21:30:57** **Incident resolved and entered the post-incident flow** Joe Burnett shared an update Status: ~~Monitoring~~ → Documenting # Investigation Notes </details> Any details you may want to add about the investigation can go here. # Follow-ups **Follow-up** - **Owner** - [Investigate why redis-sidekiq memory alert did not page](https://gitlab.com/gitlab-com/gl-infra/observability/team/-/work_items/4597) - @lmcandrew - [WebHookWorker poison pill: root cause, failure mode, and remediation status](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29039) - @thiagocsf - [Process improvement: explicit ownership and re-evaluation deadlines for mitigations that disable safety limits ](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29040)- @sabrams - [Remediate AuditEvents::AuditEventStreamingWorker from causing recurring incidents](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29033) - @nrosandich - [Create runbook for AuditEventStreamingWorker](https://gitlab.com/gitlab-org/gitlab/-/work_items/600089) - @nrosandich - [PG connection pool saturation](https://gitlab.com/gitlab-org/database-team/team-tasks/-/work_items/625) - @jarv - https://gitlab.com/gitlab-org/gitlab/-/work_items/597950+ - @jarv - https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/1069+ - @jarv # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock). - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [x] S1 incidents [require a public RCA within 7 days](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/#timeline-that-we-expect-for-reviews-to-be-completed) of the incident. If this review cannot be made public, [create a separate public RCA](https://handbook.gitlab.com/handbook/engineering/root-cause-analysis/#how-to-perform-an-rca). - [x] Close the review before the due date - [x] Go back to the incident channel or page and close out the remaining post-incident tasks