Incident Review: Error rates violating SLO
#INC-10096: Error rates violating SLO
Generated by Joe Burnett on 13 May 2026 00:05. All timestamps are local to Etc/UTC
Key Information
| Metric | Value |
|---|---|
| Customers Affected | All GitLab.com customers active during the impact period |
| Requests Affected | >1 million requests (source); 12 million delayed Sidekiq jobs resulted in pipeline creation going near-zero for extended periods, with an estimated >10,000 pipelines impacted |
| Incident Severity | Severity 1 |
| Impact Start Time | Tue, 12 May 2026 17:12:00 UTC |
| Impact End Time | Tue, 12 May 2026 18:43:44 UTC |
| Impact Duration | 1 hour, 31 minutes |
| Total Duration | 4 hours, 15 minutes |
| Link to Incident Issue | #22103 (closed) |
Summary
Problem: A major Redis Sidekiq cluster outage led to widespread elevated error rates, authentication failures, and job processing delays for all major GitLab.com services, including web, CI runners, and Git.
Impact: Many users experienced frequent 500 errors, CI pipeline failures, stuck or delayed pipelines, and unreliable Git access. Error rates peaked at 5% for CI runners and affected almost all services, leading to a high volume of support tickets. At its worst, the Sidekiq queue backlog reached over 12 million jobs, causing pipeline creation and job processing to stall.
Causes: Earlier mitigation for a previous incident led to audit event jobs being deferred, which then accumulated and consumed all available memory on the Redis Sidekiq cluster. This memory exhaustion resulted in kernel errors and caused multiple Redis nodes to fail, breaking the cluster quorum.
Response strategy: Engineers performed targeted restarts and forced resets on failed Redis nodes, then sequentially resized all Redis VMs to double their memory and CPU. As of 18:40 UTC, all three nodes are upgraded, error rates have dropped, and the system is recovering. The large Sidekiq queue backlog is now draining and being monitored. Long-term improvements are planned for the problematic audit event worker to prevent recurrence.
AuditEvents::AuditEventStreamingWorker has caused at least 4 prior incidents since March 2025, escalating in severity each time:
- #19532 — INC-310, S2, Mar 24 2025 (PGBouncer saturation)
- #19543 — INC-311, S2, Mar 25 2025 (repeat next day)
- #19690 — INC-489, S2, Apr 15 2025 (bad streaming destination)
- This incident — INC-10096, S1, May 12 2026 (Redis memory exhaustion)
What went well?
- Fast mobilization and coordination: We quickly pulled the right responders into a shared Zoom and coordinated across multiple teams (Production/SRE, Sidekiq, Runners, Release/Deploy, and CMOC).
- Effective mitigation once the failure mode was confirmed: After we identified the redis-sidekiq catchall-b nodes were unhealthy/unresponsive, responders executed a clear recovery plan (controlled restarts/resets and then VM resizing for headroom), which correlated with service error rates stabilizing.
- Good technical triage discipline: We used dashboards and logs to rule out a traffic surge (RPS flat/declining), helping focus on shared dependency and infrastructure-level failure modes rather than chasing load-driven explanations.
- Risk-aware decision making under pressure: We explicitly paused on irreversible options (like queue flushing) and prioritized safer mitigations first.
What was difficult?
- Multi-symptom, cross-service impact made diagnosis noisy: The incident presented as broad degradation (web, git, auth, CI runners, Sidekiq, internal API), which made it harder to quickly converge on a single causal narrative and communicate it crisply.
- Observability and discoverability gaps for key customer symptoms: “Pipelines stuck in created” was hard to quantify and confirm in real time; relevant dashboards/SLIs either weren’t obvious to on-call responders or didn’t directly answer the operational question.
- Host-level unresponsiveness slowed execution: SSH/login hangs and generally degraded VM behavior increased time-to-diagnose and increased operational risk while making changes.
- Alerting/routing friction: Some signals existed but didn’t reliably reach the right on-call audiences (e.g., labeling/routing gaps), and responders called out that tooling/dashboards weren’t easily discoverable during the event.
- Difficult reliability tradeoffs in the datastore failure mode: Redis-sidekiq behavior under memory saturation raised hard choices (availability vs. data loss) that are challenging to navigate during an active incident and point to needed guardrails/design improvements.
Investigation Details
Timeline
Incident Timeline
2026-05-12
17:12:00 Impact started at
Custom timestamp "Impact started at" occurred
17:15:10 Incident reported in triage by Prometheus Alertmanager alert
Prometheus Alertmanager alert reported the incidentSeverity: NoneStatus: Triage
17:15:24 Escalation acknowledged by Stephanie Jackson
Stephanie Jackson acknowledged the escalation
17:16:59 Incident accepted
Stephanie Jackson shared an update
Severity: None → Severity 2
Status: Triage → Investigating
17:16:59 Escalated to GitLab.com Production
Your workflow manually escalated the incident to the escalation path GitLab.com Production
17:16:59 Escalated to Incident Manager Oncall (IMOC)
Your workflow manually escalated the incident to the escalation path Incident Manager Oncall (IMOC)
17:17:04 Escalation acknowledged by Steve Abrams
Steve Abrams acknowledged the escalation
17:17:11 Escalation acknowledged by Stephanie Jackson
Stephanie Jackson acknowledged the escalation
17:19:55 Image posted by Stephanie Jackson
Stephanie Jackson posted an image to the channel
17:20:27 Image posted by Terri Chu
Terri Chu posted an image to the channel
I saw this when I was looking at my correlation ID, but didn't look much further https://log.gprd.gitlab.net/app/discover#/?\_tab=h@64474a0&\_g=h@2294574&\_a=h@07ba51a
17:21:15 Message from Stephanie Jackson
Stephanie Jackson's message was pinned by Alex Hanselka
https://dashboards.gitlab.net/d/general-triage/general3a-platform-triage?orgId=1&from=n\[…\]S_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main (https://dashboards.gitlab.net/d/general-triage/general3a-platform-triage?orgId=1&from=now-1h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main) -- many things show increased error rate.
17:22:38 Image posted by Stephanie Jackson
Stephanie Jackson posted an image to the channelMany things erroring:
17:26:58 Escalated to tier1 - Runners Platform - Escalation
Steve Abrams manually escalated the incident to the escalation path tier1 - Runners Platform - Escalation
17:32:49 Severity upgraded from Severity 2 → Severity 1
Steve Abrams shared an update
Severity: Severity 2 → Severity 1
17:32:50 Escalated to Infrastructure Leadership
Your workflow manually escalated the incident to the escalation path Infrastructure Leadership
17:32:56 Escalation acknowledged by Steve Abrams
Steve Abrams acknowledged the escalation
17:33:46 Update shared
Steve Abrams shared an updateEngineers have confirmed widespread elevated error rates and authentication failures affecting the git, CI runners, and web services. CI runners are showing error rates up to 5%, with the git workhorse API and web service also impacted. Users are experiencing frequent 500 errors, CI pipeline failures, and unreliable access to Git features.The team initially investigated Redis connectivity as a possible cause, but recent evidence also points to potential DNS service discovery issues or a broader disruption in the US East region. There is no ongoing rollout in production that would explain the issue; the only recent change was a registry-related rollout in staging.Multiple on-call teams have been paged, and incident severity has been escalated to S1. Engineers are coordinating across teams and analyzing dashboards, error logs, and service dependencies to isolate the root cause. Additional support is being sought in for external DNS expertise.
17:43:05 Escalation acknowledged by Davis Bickford
Davis Bickford acknowledged the escalation
17:44:22 Image posted by Stephanie Jackson
Stephanie Jackson posted an image to the channel
17:46:49 Message from Stephanie Jackson
Stephanie Jackson's message was pinned by Alex HanselkaComputer broken:
May 12 17:06:07 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: Finished Exports ubuntu_advantage metrics for use with the node_exporter.
May 12 17:06:25 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: Starting Timer-run service to periodically run Redis RDB backups, unless this node is currently the primary...
May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd redis_backup.sh[634708]: 2026-05-12 17:06:26 UTC [634708] LOG Replication lag is acceptable: 0 seconds
May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd redis_backup.sh[634708]: 2026-05-12 17:06:26 UTC [634708] SKIP_BACKUP Redis role is primary and at least one replica is healthy.
May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: redis-rdb-backup.service: Main process exited, code=exited, status=1/FAILURE
May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: redis-rdb-backup.service: Failed with result 'exit-code'.
May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: Failed to start Timer-run service to periodically run Redis RDB backups, unless this node is currently the primary.
[16861909.855206] INFO: task kcompactd0:67 blocked for more than 120 seconds.
[16861909.862237] Tainted: G O K 5.15.0-1083-gcp #92~20.04.1-Ubuntu
[16861909.870013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 12 17:36:17 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.855206] INFO: task kcompactd0:67 blocked for more than 120 seconds.
May 12 17:38:14 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.862237] Tainted: G O K 5.15.0-1083-gcp #92~20.04.1-Ubuntu
May 12 17:40:02 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.870013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 12 17:41:54 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878211] task:kcompactd0 state:D stack: 0 pid: 67 ppid: 2 flags:0x00004000
May 12 17:43:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878222] Call Trace:
May 12 17:44:26 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878226] <TASK>
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878231] __schedule+0x2cd/0x890
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878238] schedule+0x69/0x110
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878240] io_schedule+0x16/0x40
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878242] wait_on_page_bit_common+0x16e/0x3b0
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878246] ? filemap_invalidate_unlock_two+0x50/0x50
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878249] __lock_page+0x4c/0x60
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878251] migrate_pages+0xaf6/0xda0
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878253] ? move_freelist_tail+0xc0/0xc0
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878257] ? fast_isolate_freepages+0x7b0/0x7b0
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878259] compact_zone+0x658/0xe90
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878262] ? __compaction_suitable+0xa6/0x100
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878264] kcompactd_do_work+0x10b/0x280
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878267] kcompactd+0x310/0x370
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878270] ? wait_woken+0x60/0x60
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878273] ? kcompactd_do_work+0x280/0x280
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878275] kthread+0x127/0x150
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878277] ? set_kthread_struct+0x50/0x50
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878279] ret_from_fork+0x1f/0x30
May 12 17:44:37 redis-sidekiq-catchall-b-01-db-gprd kernel: [16861909.878284] </TASK>17:48:47 Message from Stephanie Jackson
Stephanie Jackson's message was pinned by Alex HanselkaRedis broken:
May 12 17:21:46 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:44Z","level":"warning","caller":"lite/lite.go:946","message":"SLOW TRANSACTION","component":"sqlite","duration":1771360449,"stack":"goroutine 108 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1() (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1())\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x5c\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction)(0xc002e10bb0, {0x55b7268b06a0, 0xc003130b90}, 0xc004c3fe20)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997) +0x1fd\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:73 +0xbc\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:46 +0xff\ncreated by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig (http://github.com/gravitational/teleport/lib/backend/lite.NewWithConfig) in goroutine 1\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266) +0x7c7\n"}
May 12 17:21:47 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: redis-server.service: Found left-over process 2516163 (redis-server) in control group while starting unit. Ignoring.
May 12 17:21:47 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 12 17:21:47 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:44Z","level":"warning","caller":"service/connect.go:751","message":"Connection problem: sync rotation state.","component":"proc:1","pid":"3734543.1","error":"connection error: desc = \"transport: Error while dialing: failed to dial: context deadline exceeded\""}
May 12 17:21:48 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Started Run a redis-server process..
May 12 17:21:48 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Stopped Login Service.
May 12 17:21:48 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Condition check resulted in Load Kernel Module drm being skipped.
May 12 17:21:51 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Starting Login Service...
May 12 17:21:53 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:49Z","level":"warning","caller":"service/connect.go:661","message":"Sync rotation state cycle failed.","component":"proc:1","pid":"3734543.1","retry_interval":10000000000}
May 12 17:21:58 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"caller":"srv/heartbeatv2.go:583","component":null,"level":"warning","message":"Failed to perform inventory heartbeat for ssh server: inventory control msg not sent: EOF","timestamp":"2026-05-12T17:21:42Z"}
May 12 17:21:59 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:58Z","level":"warning","caller":"labels/cloud.go:180","message":"Failed to fetch cloud tags","component":"cloudlabels","error":"Get \"https://compute.googleapis.com/compute/v1/projects/gitlab-production/zones/us-east1-d/instances/redis-sidekiq-catchall-b-02-db-gprd\": EOF"}
May 12 17:22:00 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:21:58Z","level":"warning","caller":"labels/cloud.go:180","message":"Failed to fetch cloud tags","component":"cloudlabels","error":"Get \"https://compute.googleapis.com/compute/v1/projects/gitlab-production/zones/us-east1-d/instances/redis-sidekiq-catchall-b-02-db-gprd\": net/http: TLS handshake timeout"}
May 12 17:22:00 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:22:00Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)"
May 12 17:22:01 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:22:01Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)"
May 12 17:22:01 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:22:01Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)"
May 12 17:22:01 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:01Z","level":"info","caller":"service/state.go:112","message":"Detected Teleport component is running in a degraded state.","component":"proc:1","pid":"3734543.1","component":"node"}
May 12 17:22:03 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:01Z","level":"warning","caller":"lite/lite.go:946","message":"SLOW TRANSACTION","component":"sqlite","duration":12021385705,"stack":"goroutine 108 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1() (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1())\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x5c\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction)(0xc002e10bb0, {0x55b7268b06a0, 0xc003130b90}, 0xc004c3fe20)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997) +0x1fd\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:73 +0xbc\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:46 +0xff\ncreated by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig (http://github.com/gravitational/teleport/lib/backend/lite.NewWithConfig) in goroutine 1\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266) +0x7c7\n"}
May 12 17:22:04 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:22:04Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)"
May 12 17:22:04 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:01Z","level":"warning","caller":"inventory/inventory.go:213","message":"Failed to create inventory control stream","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to dial: context deadline exceeded\""}
May 12 17:22:10 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:08Z","level":"warning","caller":"lite/lite.go:946","message":"SLOW TRANSACTION","component":"sqlite","duration":3396583005,"stack":"goroutine 108 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1() (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1())\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x5c\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction)(0xc002e10bb0, {0x55b7268b06a0, 0xc003130b90}, 0xc004c3fe20)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997) +0x1fd\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:73 +0xbc\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:46 +0xff\ncreated by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig (http://github.com/gravitational/teleport/lib/backend/lite.NewWithConfig) in goroutine 1\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266) +0x7c7\n"}
May 12 17:22:11 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:09Z","level":"info","caller":"service/connect.go:666","message":"Initiating new sync rotation state cycle after backoff.","component":"proc:1","pid":"3734543.1","backoff_time":14308371920}
May 12 17:22:11 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"caller":"cache/cache.go:1307","component":"node:1:cache","level":"warning","message":"Re-init the cache on error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to dial: context deadline exceeded\"","timestamp":"2026-05-12T17:22:04Z"}
May 12 17:22:12 redis-sidekiq-catchall-b-02-db-gprd systemd[1]: Reloading.
May 12 17:22:12 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:12Z","level":"warning","caller":"lite/lite.go:946","message":"SLOW TRANSACTION","component":"sqlite","duration":1816520807,"stack":"goroutine 108 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1() (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction.func1())\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x5c\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).inTransaction)(0xc002e10bb0, {0x55b7268b06a0, 0xc003130b90}, 0xc004c3fe20)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997) +0x1fd\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).removeExpiredKeys(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:73 +0xbc\ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0) (http://ngithub.com/gravitational/teleport/lib/backend/lite.(Backend).runPeriodicOperations(0xc002e10bb0))\n\tgithub.com/gravitational/teleport/lib/backend/lite/periodic.go:46 +0xff\ncreated by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig (http://github.com/gravitational/teleport/lib/backend/lite.NewWithConfig) in goroutine 1\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266 (http://tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:266) +0x7c7\n"}
May 12 17:22:21 redis-sidekiq-catchall-b-02-db-gprd teleport[3734543]: {"timestamp":"2026-05-12T17:22:05Z","level":"info","caller":"client/alpn_conn_upgrade.go:88","message":"ALPN connection upgrade test failed.","address":"production.teleport.gitlab.net:443 (http://production.teleport.gitlab.net:443)","error":"dial tcp: lookup production.teleport.gitlab.net (http://production.teleport.gitlab.net): i/o timeout"}
May 12 17:22:22 redis-sidekiq-catchall-b-02-db-gprd snapd[853973]: SIGABRT: abort
May 12 17:22:57 redis-sidekiq-catchall-b-02-db-gprd snapd[853973]: PC=0x55d3dab938e1 m=0 sigcode=0
May 12 17:23:09 redis-sidekiq-catchall-b-02-db-gprd dbus-daemon[673]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 33308ms)
May 12 17:23:13 redis-sidekiq-catchall-b-02-db-gprd snapd[853973]: goroutine 0 gp=0x55d3dc8997a0 m=0 mp=0x55d3dc89aa00 [idle]:
May 12 17:23:50 redis-sidekiq-catchall-b-02-db-gprd redis_exporter[3578979]: time="2026-05-12T17:23:26Z" level=error msg="Couldn't connect to redis instance (redis://localhost:6379)"
May 12 17:37:32 redis-sidekiq-catchall-b-02-db-gprd dbus-daemon[673]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service' requested by ':1.1172883' (uid=0 pid=2413205 comm="sshd: ahmad [priv] " label="unconfined")17:50:55 Message from Raynard Omongbale
Raynard Omongbale's message was pinned by Sarah Walker
May 12 17:06:26 redis-sidekiq-catchall-b-01-db-gprd systemd[1]: redis-rdb-backup.service: Failed with result 'exit-code'.
Failed to start Timer-run service to periodically run Redis RDB backups, unless this node is currently the primary.
[16861909.855206] INFO: task kcompactd0:67 blocked for more than 120 seconds.17:59:17 Image posted by Cameron McFarland
Cameron McFarland posted an image to the channel
FYI for later: https://dashboards.gitlab.net/goto/eflv2dgnkl5ogc?orgId=1
18:02:47 Update shared
Steve Abrams shared an updateThe Redis Sidekiq catch-all cluster experienced a major outage due to multiple node failures, resulting in widespread elevated error rates, pipeline failures, and degraded service across git, CI runners, and web services. The outage was traced to resource exhaustion and kernel errors on the Redis hosts, which left only one node healthy and unable to maintain cluster quorum.Engineers performed targeted restarts and a forced reset of the two unresponsive Redis nodes to restore cluster capacity. As of 18:00 UTC, all three Redis nodes (O1, O2, O3) are up, with O2 acting as the current primary, restoring service redundancy.Teams are monitoring following the reset, but initially the nodes are immediately throwing errors again.
18:10:12 Message from Rachel Nienaber
Rachel Nienaber's message was pinned by Sarah Walker
Noting which one is stopping - Cameron is stopping redis-sidekiq-catchall-b-01-db-gprd
18:12:32 Message from Rachel Nienaber
Rachel Nienaber's message was pinned by Sarah Walker
Cameron is going up to the next machine type to c2-standard-16
18:16:16 Message from Sarah Walker
Sarah Walker pinned their own messagecatchall-b-02 has been stopped - waiting... :waiting:
18:19:36 Message from Sarah Walker
Sarah Walker pinned their own messageStopped - machine is being upsized - started
18:20:14 Update shared
Steve Abrams shared an updateThe root cause was linked to audit event jobs that consumed all available memory after being deferred earlier in the day in inc-10084-sidekiq-queueing-apdex-slo-violation-on-shard-catchall-b-apdex-at.
18:25:51 Message from Sarah Walker
Sarah Walker pinned their own messagecatchall-b-03 is now stopped
18:29:07 Monitoring at
Custom timestamp "Monitoring at" occurred
18:29:07 Identified at
Custom timestamp "Identified at" occurred
18:29:07 Status changed from Investigating → Monitoring
Steve Abrams shared an update
Status: Investigating → Monitoring
18:30:58 Status changed from Monitoring → Fixing
Steve Abrams shared an update
Status: Monitoring → Fixing
18:32:06 Message from Raynard Omongbale
Raynard Omongbale's message was pinned by Sarah Walker
https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq3a-shard-detail?orgId=1&fro\[…\]-gprd&var-environment=gprd&var-stage=main&var-shard=$\__all (https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq3a-shard-detail?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-shard=$\__all) https://dashboards.gitlab.net/d/alerts-sat_redis_primary_cpu/alerts3a-redis-primary-cpu\[…\]=&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd (https://dashboards.gitlab.net/d/alerts-sat_redis_primary_cpu/alerts3a-redis-primary-cpu3a-saturation-detail?from=now-12h&to=now&var-environment=gprd&var-type=redis-sidekiq&var-stage=&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd)
18:35:42 Image posted by Stan Hu
Stan Hu posted an image to the channel
FYI https://log.gprd.gitlab.net/app/r/s/mbKw6 shows in the last 15 minutes about 14% of these workers came from gitlab-org projects:
18:36:07 Message from Rachel Nienaber
Rachel Nienaber's message was pinned by Sarah Walker
@steve Abrams This is the second S2 incident involving this worker this month. The previous one was INC-8169. There was also an S3 in December. As part of the incident review, the team who owns this worker needs to suggest alternatives to reduce the pressure that this worker places on the system that should be then done as part of the FCL for them.
18:37:38 Image posted by Raynard Omongbale
Raynard Omongbale posted an image to the channel
18:38:08 Image posted by Raynard Omongbale
Raynard Omongbale posted an image to the channel
18:38:48 Message from Stephanie Jackson
Stephanie Jackson's message was pinned by Sarah Walker
@Kam https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?orgId=1&from=now-1h&to=\[…\]d&var-environment=gprd&var-stage=main&var-shard=catchall-b (https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?orgId=1&from=now-1h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-shard=catchall-b)
18:41:08 Image posted by Stephanie Jackson
Stephanie Jackson posted an image to the channelCurrent queue:
18:43:44 Fixed at
Custom timestamp "Fixed at" occurred
18:43:44 Status changed from Fixing → Monitoring
Steve Abrams shared an update
Status: Fixing → Monitoring
18:58:53 Image posted by Raynard Omongbale
Raynard Omongbale posted an image to the channel
@stejacks Just to confirm based on https://gitlab.slack.com/archives/C101F3796/p1778610325539689 We were able to re-enabled AuditEventStreamingWorker and not deferred it. My reasons being that the AuditEventStreamingWorker numbers are climbing. might be due to the earlier deferrals but it is worth shouting out that that queue is climbing. I will check back on it. source (https://dashboards.gitlab.net/goto/eflv7ql311rswf?orgId=1)
19:10:21 Image posted by Stan Hu
Stan Hu posted an image to the channel
Job queue has dropped from a peak of 12.2 million jobs to under 7 million: https://dashboards.gitlab.net/goto/eflv8rgw4gnpcf?orgId=1
2h later
21:30:57 Incident resolved and entered the post-incident flow
Joe Burnett shared an update
Status: Monitoring → Documenting
Investigation Notes
Any details you may want to add about the investigation can go here.
Follow-ups
Follow-up - Owner
- Investigate why redis-sidekiq memory alert did not page - @lmcandrew
- WebHookWorker poison pill: root cause, failure mode, and remediation status - @thiagocsf
- Process improvement: explicit ownership and re-evaluation deadlines for mitigations that disable safety limits - @sabrams
- Remediate AuditEvents::AuditEventStreamingWorker from causing recurring incidents - @nrosandich
- Create runbook for AuditEventStreamingWorker - @nrosandich
- PG connection pool saturation - @jarv
- Investigate active queue management for Sidekiq (gitlab-org/gitlab#597950 - closed) - @jarv
- Add Ruby thread scheduler priority patch (gitlab-org/gitlab-build-images!1069 - merged) - @jarv
Review Guidelines
This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI.
For the person opening the Incident Review
- Set the title to
Incident Review: (Incident issue name) - Assign a
Service::*label (most likely matching the one on the incident issue) - Set a
Severity::*label which matches the incident - In the
Key Informationsection, make sure to include a link to the incident issue - Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) The DRI for the incident review is the issue assignee.
For the assigned DRI
- Fill in the remaining fields in the
Key Informationsection, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - If there are metrics showing
Customers AffectedorRequests Affected, link those metrics in those fields - For all S1 and S2 incidents, begin the Feature Change Lock (FCL) process and open an issue in the FCL project.
- Create a few short sentences in the Summary section summarizing what happened (TL;DR)
- Link any corrective actions and describe any other actions or outcomes from the incident
- Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported?
- Once discussion wraps up in the comments, summarize any takeaways in the details section
- If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue.
- S1 incidents require a public RCA within 7 days of the incident. If this review cannot be made public, create a separate public RCA.
- Close the review before the due date
- Go back to the incident channel or page and close out the remaining post-incident tasks