On-Call Handover 2026-04-05 15:00 UTC

woodhouse-handover
2026-04-05T14:03:31Z

# On-Call Handover *Brought to you by [woodhouse](https://gitlab.com/gitlab-com/gl-infra/woodhouse/-/blob/master/pkg/handover/generator.go) for EOC shift changes only (not IMOC or CMOC)* ## 👥 Shift Transitions - **EOC egress:** @knottos - **EOC ingress:** @poneill - **IMOC egress:** @mrincon - **IMOC ingress:** @cheryl.li - **CMOC egress:** @dserafin-gitlab - **CMOC ingress:** twilliams @dserafin-gitlab **Previous handover:** [On-Call Handover 2026-04-05 07:00 UTC](https://gitlab.com/gitlab-com/gl-infra/on-call-handovers/-/work_items/6998) ## 📊 Shift Statistics - 🔥 **2** ongoing incidents - 📝 **42** incidents in workflow - 🔵 **3** mitigated incidents - ✅ **0** resolved incidents - 📟 **0** PagerDuty incidents - 🔧 **1** change requests ## :book: Summary #### What (if any) time-critical work is being handed over? #### What contextual info may be useful for the next few on-call shifts? ## 🚨 Critical: Ongoing Incidents ### GitLab Incidents | Incident | Status | Severity | Created | GitLab | Summary | |----------|--------|----------|---------|--------|---------| | [500 error on GitLab repo](https://app.incident.io/gitlab/incidents/01KN7EERCQ1BEQPHS5QCC7J9Y7) | Investigating | ~"severity::3" | 2026-04-02 | [#21705](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21705) | Problem**: A Gitaly node experienced a sudden failure of its git process, result... | | [Chef failures on patroni VMs](https://app.incident.io/gitlab/incidents/01KKKGPVD5BZESPGG12D982ENQ) | Fixing | ~"severity::3" | 2026-03-13 | [#21543](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21543) | Problem**: Chef failed on Patroni VMs due to errors with the Python environment,... | ## 🔵 Monitoring: Mitigated Incidents

👀 Under Monitoring (3 incidents)
| Incident | Status | Severity | Created | GitLab | Summary | |----------|--------|----------|---------|--------|---------| | [Loadbalancer error rate for we...](https://app.incident.io/gitlab/incidents/01KN323271NFXPKZP0FCFF7JZE) | Monitoring | ~"severity::3" | 2026-03-31 | [#21680](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21680) | Problem**: The loadbalancer for the web service in the main stage had an error r... | | [Accidential Deletion of Group ...](https://app.incident.io/gitlab/incidents/01KMJW0X18Y6W4G3Q3FAV28N7D) | Monitoring | ~"severity::1" | 2026-03-25 | [#21638](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21638) | Problem**: A GitLab.com customer group was accidentally deleted during migration... | | [No credit remain error due to ...](https://app.incident.io/gitlab/incidents/01KJF0K2BQTZK9C29TTV87PBNA) | Monitoring | ~"severity::3" | 2026-02-27 | [#21404](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21404) | Problem**: Self-Managed customers see a 'no credit remain' error in the GitLab D... |
## ✅ Recently Resolved No incidents resolved during this shift. ## 📝 Incidents in Workflow

📋 Documenting/Reviewing Phase (42 incidents)
| Incident | Status | Severity | Created | GitLab | Summary | |----------|--------|----------|---------|--------|---------| | [Loadbalancer 5xx error rate vi...](https://app.incident.io/gitlab/incidents/01KND5KP931PCYCNJB17NB3NQ5) | Documenting | ~"severity::4" | 2026-04-04 | [#21711](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21711) | The load balancer for the web service in the main stage has an aggregated HTTP 5... | | [Blackbox probe availability be...](https://app.incident.io/gitlab/incidents/01KND1DY1KV1JP2EY84VBF7QCK) | Documenting | ~"severity::4" | 2026-04-04 | [#21710](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21710) | Problem**: Availability for blackbox probes to... | | [Users can't login with 2FA via...](https://app.incident.io/gitlab/incidents/01KN3WJXA4XB5YXXZFAW12F1ZA) | Documenting | ~"severity::2" | 2026-04-01 | [#21683](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21683) | Problem**: Free-tier users could not log in with email-based two-factor authenti... | | [High Backlog](https://app.incident.io/gitlab/incidents/01KN26YCM5JWQ2ZC8E7TV0A6BG) | Documenting | ~"severity::2" | 2026-03-31 | [#21676](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21676) | Several team members reported "Service Unavailibile: High Backlog" | | [Blackbox probe availability be...](https://app.incident.io/gitlab/incidents/01KN26NQYX3XVKSVB49ZJGQR3J) | Documenting | ~"severity::2" | 2026-03-31 | [#21677](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21677) | Availability checks for https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1... | | [pre environment is down](https://app.incident.io/gitlab/incidents/01KN1CW5QAZ3XTHKNMRJQFZAEH) | Documenting | ~"severity::3" | 2026-03-31 | [#21670](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21670) | Problem**: GitLab's pre environment is down, with the website returning a 503 er... | | [smoke test failing; merge requ...](https://app.incident.io/gitlab/incidents/01KMZYCK3MJDCN919X14R5HR0F) | Documenting | ~"severity::3" | 2026-03-30 | [#21666](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21666) | Problem**: Automated smoke tests on gstg-canary were failing because comments su... | | [Increased PermissionDenied err...](https://app.incident.io/gitlab/incidents/01KMDFJJPG6KX8GS52Y0JZFWRP) | Documenting | ~"severity::3" | 2026-03-23 | [#21615](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21615) | The `gitaly-09-stor-gprd.c.gitlab-gitaly-gprd-83fd.internal` Gitaly node has suf... | | [CNG Security Mirror Master Bro...](https://app.incident.io/gitlab/incidents/01KMC6CDH8VAMWXA202R53YM80) | Documenting | ~"severity::3" | 2026-03-23 | [#21609](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21609) | Problem**: CI pipelines are failing on the default branch of the CNG repository'... | | [Error rate for server_code_gen...](https://app.incident.io/gitlab/incidents/01KKXT4WD9XNQAVX9RQ37X3TPD) | Documenting | ~"severity::4" | 2026-03-17 | [#21574](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21574) | Problem**: Server_code_generations in the AI Gateway (europe-west2) is experienc... | | [Production QA failure](https://app.incident.io/gitlab/incidents/01KKKYP1CDQK471RSG21J2J2X5) | Documenting | ~"severity::3" | 2026-03-13 | [#21548](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21548) | Problem**: QA pipelines have failed for two consecutive days on a specific test,... | | [Sidekiq jobs queueing duration...](https://app.incident.io/gitlab/incidents/01KKH3XHHNDCBK3M8ZYXVES8RA) | Documenting | ~"severity::2" | 2026-03-12 | [#21528](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21528) | The Sidekiq queueing apdex on the low-urgency-cpu-bound shard is currently 99.36... | | [Sidekiq queueing SLO violation...](https://app.incident.io/gitlab/incidents/01KKGTF8F38D5HYN59SZBQKW6W) | Documenting | ~"severity::4" | 2026-03-12 | [#21527](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21527) | On the urgent-other shard, 96.77% of Sidekiq jobs met their queueing duration ta... | | [Duo Agents cannot be selected ...](https://app.incident.io/gitlab/incidents/01KKGTCPEV73QC0313HHTZ1BCA) | Documenting | ~"severity::3" | 2026-03-12 | [#21526](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21526) | Problem**: Users with valid credits were unable to select Duo Agents due to the ... | | [Sidekiq queueing apdex SLO vio...](https://app.incident.io/gitlab/incidents/01KKE9F50R76KXEFXDD7SD2DSP) | Documenting | ~"severity::4" | 2026-03-11 | [#21513](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21513) | Problem**: The sidekiq_queueing component on the elasticsearch shard had a lower... | | [Sidekiq queueing apdex SLO vio...](https://app.incident.io/gitlab/incidents/01KKBGBRKYACJCDXN78KQARHKP) | Documenting | ~"severity::4" | 2026-03-10 | [#21495](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21495) | Problem**: Sidekiq jobs on the elasticsearch shard repeatedly failed to meet que... | | [2026-03-09 \| Daily Data Team S...](https://app.incident.io/gitlab/incidents/01KK9Z80VYHB3422ZB9PQHQPFB) | Documenting | ~"severity::4" | 2026-03-09 | [#21492](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21492) | resolved by itself, however, a longer term fix is still needed as this has been ... | | [SQL transaction apdex SLO viol...](https://app.incident.io/gitlab/incidents/01KK9C9XD6RC2CB87NKRN7D3TH) | Documenting | ~"severity::4" | 2026-03-09 | [#21483](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21483) | The apdex value for SQL transactions from the Rails application to the Postgres ... | | [The polling SLI of the ci-runn...](https://app.incident.io/gitlab/incidents/01KK96QNAJ3M3ZVKMSPD7NWNJY) | Documenting | ~"severity::4" | 2026-03-09 | [#21481](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21481) | Problem**: The ci-runners service experienced a spike in 5xx errors on job polli... | | [Apdex SLO violation for anthro...](https://app.incident.io/gitlab/incidents/01KK676C0KJTYMS3JC1P64W2XG) | Documenting | ~"severity::4" | 2026-03-08 | [#21474](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21474) | The apdex score for non-streaming inferences to the anthropic model in the europ... | | [Prod deployment failed because...](https://app.incident.io/gitlab/incidents/01KJWK1MT35GYRMYZEX3HMZWYS) | Documenting | ~"severity::4" | 2026-03-04 | [#21446](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21446) | Problem**: Production deployment failed due to a missing secret (gitlab-clickhou... | | [Intemrittent 400 errors when l...](https://app.incident.io/gitlab/incidents/01KJT8MS9CYH0APHZMCA2ND0JK) | Documenting | ~"severity::3" | 2026-03-03 | [#21436](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21436) | Problem**: Intermittent 400 Bad Request errors occur when users attempt to log i... | | [CustomersDot usage quota traff...](https://app.incident.io/gitlab/incidents/01KJT3MDEVARAGT2GBTF0WH0F3) | Documenting | ~"severity::4" | 2026-03-03 | [#21435](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21435) | Problem**: Usage quota traffic for CustomersDot dropped to near zero in monitori... | | [Flows failing with In Memory E...](https://app.incident.io/gitlab/incidents/01KJSCG6WQQ85GZAZWGR37GZ2Q) | Documenting | ~"severity::2" | 2026-03-03 | [#21429](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21429) | Problem**: Remote flows, including Duo Developer, failed due to an InMemoryPromp... | | [Apdex SLO violation for Rails ...](https://app.incident.io/gitlab/incidents/01KJQBXPETGV3W9F18N50X2NE7) | Documenting | ~"severity::3" | 2026-03-02 | [#21423](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21423) | Problem**: The apdex score for SQL transactions from the Rails monolith to the P... | | [PostgreSQL LockManager content...](https://app.incident.io/gitlab/incidents/01KJA6Q75ABBXXDYWG3RASZHHD) | Documenting | ~"severity::4" | 2026-02-25 | [#21383](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21383) | PostgreSQL's LockManager LWLock contention for the patroni-ci database in the ma... | | [Pipelines in for gitlab-org/gi...](https://app.incident.io/gitlab/incidents/01KJA2T0R5R8NW6XEDT9PR2YAB) | Documenting | ~"severity::3" | 2026-02-25 | [#21381](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21381) | Problem**: Pipelines in gitlab-org/gitlab stopped running, with jobs stuck in a ... | | [Red master branch on dev (buil...](https://app.incident.io/gitlab/incidents/01KJ8H0EGEKSJ807C18FRD863B) | Documenting | ~"severity::3" | 2026-02-24 | [#21379](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21379) | Problem**: The master branch failed due to a recurring test failure. **Impact**:... | | [Elevated inference error rate ...](https://app.incident.io/gitlab/incidents/01KJ845B6W2B6MVCM1K2C3Q3K3) | Documenting | ~"severity::4" | 2026-02-24 | [#21374](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21374) | Inferences sent to the vertex-ai engines through the ai-gateway in europe-west2 ... | | [Sidekiq queueing apdex SLO vio...](https://app.incident.io/gitlab/incidents/01KJ57CBW0ZC4T9K3ZMHFTYZJ3) | Documenting | ~"severity::3" | 2026-02-23 | [#21361](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21361) | Problem**: Sidekiq jobs failed to meet queueing duration targets across the `cat... | | [qa-smoke 5/8 smoke tests are f...](https://app.incident.io/gitlab/incidents/01KHRPCYKQX25YTFCX5SE6PZ37) | Documenting | ~"severity::3" | 2026-02-18 | [#21324](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21324) | Summary** qa-smoke 5/8 are failing on `gprd-cny` due to merge request preparatio... | | [E2E Test Failure: Work Items O...](https://app.incident.io/gitlab/incidents/01KH6NBJQ62ZHWRKPA2SVJBA05) | Documenting | ~"severity::3" | 2026-02-11 | [#21259](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21259) | An end-to-end test failure occurred in the staging environment due to a Work Ite... | | [E2E Test Failures Blocking 18....](https://app.incident.io/gitlab/incidents/01KGST6FNGKAA62TX3ZW4CDKRQ) | Documenting | ~"severity::3" | 2026-02-06 | [#21227](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21227) | Problem**: Persistent failures in the GitLab QA test suite prevented progress on... | | [The polling SLI of the ci-runn...](https://app.incident.io/gitlab/incidents/01KGPBVF46PPC5XJ5C4ECC0CR1) | Documenting | ~"severity::4" | 2026-02-05 | [#21213](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21213) | Problem**: The error rate for job polling by ci-runners in the main stage has ex... | | [No traffic received by pulp_ap...](https://app.incident.io/gitlab/incidents/01KGK3ZF8W9Q9XTNKRS82Q254R) | Documenting | ~"severity::4" | 2026-02-04 | [#21204](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21204) | The pulp_app_tasks component in the main stage has not received any traffic for ... | | [Incorrect binding of composite...](https://app.incident.io/gitlab/incidents/01KG7RPBBWM45RTEW765RB8NHE) | Documenting | ~"severity::3" | 2026-01-30 | [#21172](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21172) | It seems a revert has already been merged. Subsequent actions should be tracked ... | | [Jobs stuck in pending state wi...](https://app.incident.io/gitlab/incidents/01KCA5ECVCREANJVQ7EP6CC3X8) | Documenting | ~"severity::2" | 2025-12-12 | [#20996](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20996) | Problem**: CI jobs were stuck in the pending state with no corresponding entry i... | | [Gitaly goserver apdex SLO viol...](https://app.incident.io/gitlab/incidents/01KK9SBE1ZRYGN2FT424C5AJ0Z) | Reviewing | ~"severity::2" | 2026-03-09 | [#21487](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21487) | Problem**: The gitaly goserver component in the cny stage experienced degraded p... | | [Gitaly ongoing performance iss...](https://app.incident.io/gitlab/incidents/01KHQRGFM6D9SEZKARK7XFS3FN) | Reviewing | ~"severity::2" | 2026-02-18 | [#21315](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21315) | Problem**: Some Gitaly nodes experienced intermittent resource exhaustion, causi... | | [https://gitlab.com/gitlab-org/...](https://app.incident.io/gitlab/incidents/01KHC701167E74JJX70QRW93FQ) | Reviewing | ~"severity::2" | 2026-02-13 | [#21289](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21289) | Problem**: https://gitlab.com/gitlab-org/gitlab returned HTTP 500 errors for a... | | [Web service error rate violati...](https://app.incident.io/gitlab/incidents/01KNBZ8XVCV0GVYCGQ5V6SE1WS) | Paused | ~"severity::3" | 2026-04-04 | [#21709](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21709) | Problem**: The web service is experiencing HTTP 5xx error rates above the define... | | [Elevated count of Claude Opus ...](https://app.incident.io/gitlab/incidents/01KN1HMEEYJ17K6G6YTEAGJ7PZ) | Paused | ~"severity::3" | 2026-03-31 | [#21672](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21672) | Problem**: A surge of rate limit errors affected users accessing Claude Opus 4.6... |
## 🔧 Change Management ### 🔄 In Progress No change requests currently in progress. ### ✅ Completed During Shift

📋 Closed Changes (1 requests)
| Change Request | Closed | GitLab | Status | |----------------|--------|--------|--------| | [Confidential Change](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21217) | 2026-04-05 | [GitLab](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21217) | ✅ Closed |
--- *Generated via [woodhouse](https://gitlab.com/gitlab-com/gl-infra/woodhouse) on 2026-04-05T15:00:00Z*