# [INC-8712: Slow Gitaly operations slow on gitlab.com and gitlab-org 500 errors](https://app.incident.io/gitlab/incidents/8712) Generated by Steve Abrams on 24 Mar 2026 19:00. All timestamps are local to Etc/UTC # Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | | ------ | ------ | | Customers Affected | Unknown| | Requests Affected | All requests within `gitlab-org` namespace| | Incident Severity | Severity 2 | | Impact Start Time | Tue, 24 Mar 2026 13:30:00 UTC| | Impact End Time | Tue, 24 Mar 2026 17:45:00 UTC | | Total Duration | 3 hours, 28 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21624 / https://app.incident.io/gitlab/incidents/8712 | # Summary **Problem**: Gitaly operations on [gitlab.com](http://gitlab.com "gitlab.com"), especially for the gitlab-org group, have been running slower than normal, impacting work on merge requests, comments, and merge request reviews. The gitlab-org namespace became unavailable due to a Gitaly storage node reboot failure, resulting in 500 errors and outages. **Impact**: Users working in the gitlab-org group on gitlab.com experienced significant slowdowns, widespread 500 errors, and outages for merge requests, comments, and reviews. Any customer or user with pipelines or integrations accessing this group also saw errors. Service has now been restored, but integrity checks are ongoing. **Causes**: A single Gitaly storage node (gitaly-cny-01-stor-gprd.c.gitlab-production.internal) experienced a sharp spike in anonymous memory usage, leading to degraded performance. After rebooting the node as mitigation, it failed to mount its data disk, causing further outages until filesystem repair and a successful reboot restored service. **Response strategy**: We identified the affected storage node and attempted a reboot. When the node failed to mount its data disk, we used console access to investigate and repair the filesystem. After a successful reboot, service was restored. # What went well? - We quickly identified a single Gitaly storage node as the primary driver, which helped focus the response. - The team escalated appropriately and coordinated effectively across SRE and Gitaly stakeholders. - We restored service relatively quickly once the failure mode was clear, using console access and filesystem repair. - We captured clear follow-ups to validate repository integrity and investigate the underlying cause. # What was difficult? - Early signals were ambiguous (slowness vs outage), which made it harder to quickly assess scope and severity. - A mitigation step (reboot) increased impact when the node failed to remount its data disk, requiring deeper recovery work. - Recovery decisions carried risk (fsck vs snapshot restore) with limited confidence until the node was back. - Post-recovery validation at scale was challenging, and our existing tooling/paths for bulk integrity checks weren’t immediately frictionless. # Investigation Details <details> <summary> Timeline</summary> # Incident Timeline **2026-03-24** **13:30:00** **Impact started at** Custom timestamp "Impact started at" occurred **15:27:18** **Incident reported by Terri Chu** Terri Chu reported the incident Severity: Severity 3 Status: Investigating **15:30:19** **Message from Terri Chu** Terri Chu pinned their own message From Heinrich: Tested with https://gitlab.com/issue-reproduce/rtse/-/merge_requests/19 and seems okay **15:31:25** **Image posted by Brian Williams** Brian Williams posted an image to the channel 95th percentile of `json.gitaly_duration_s`, last 3 hours.  **15:33:41** **Escalated to GitLab.com Production** Your workflow manually escalated the incident to the escalation path [GitLab.com Production](https://app.incident.io/gitlab/on-call/escalation-paths/01K5YWH36WZZX2YT9HNRR0WMB0) **15:34:02** **Escalation acknowledged by Dan Ryan** Dan Ryan acknowledged the escalation **15:41:34** **Image posted by Brian Williams** Brian Williams posted an image to the channel gitlab-org looks definitively slow  **15:42:49** **Image posted by Brian Williams** Brian Williams posted an image to the channel That one was `duration_s`. Here is `gitaly_duration_s`.  **15:44:50** **Image posted by Dan Ryan** Dan Ryan posted an image to the channel https://dashboards.gitlab.net/goto/cfgzvnjz39nuof?orgId=1 similiar rails_request sli performance as last time as well. https://dashboards.gitlab.net/goto/afgzvreub1wxsd?orgId=1 rails_request Latency - per endpoint_id is showing a similar pattern, albeit much less of a degradation as last time. (https://gitlab.slack.com/archives/C0AMQ7KMLBD/p1774030513423699) similarly, we have a notable apdex degradation, though not severe enough to automatically page. https://dashboards.gitlab.net/goto/ffgzw2ipkhfcwd?orgId=1 given the frequency of incidents lately, I would consider any performance degradation to be higher severity.  **15:49:14** **Message from Dan Ryan** Dan Ryan pinned their own message metrics spikes approximately 1330 UTC **15:54:20** **Image posted by Dan Ryan** Dan Ryan posted an image to the channel also noting that we see similar patterns in metrics to #inc-7515-gitaly-ongoing-performance-issues. namely an increase in anonymous pages on gitaly-cny-01-stor-gprd.c.gitlab-production.internal https://dashboards.gitlab.net/goto/dfgzwv0jzeku8d?orgId=1  **15:56:18** **Escalated to tier2 - Gitaly Escalation** Dan Ryan manually escalated the incident to the escalation path [tier2 - Gitaly Escalation](https://app.incident.io/gitlab/on-call/escalation-paths/01JJWB07RXAG02RXYR4QR47J9E) **15:58:57** **Severity upgraded from Severity 3 → Severity 2** Dan Ryan shared an update Severity: ~~Severity 3~~ → Severity 2 Performance monitoring confirms a degradation in web Apdex and elevated Gitaly durations, particularly for the gitlab-org group. Metrics show a spike in latency and Apdex drop around 13:30 UTC, with performance remaining below typical levels since then but still within SLO limits. Memory usage on gitaly-cny-01-stor-gprd.c.gitlab-production.internal is elevated, with a notable increase in certain memory types after 13:15 UTC, suggesting increased memory pressure. These patterns are similar to a previous Gitaly performance incident inc-7515-gitaly-ongoing-performance-issues Given recurring performance issues and recent frequency, the incident is being escalated to ensure appropriate response. Assistance from CMOC and IMOC has been requested. **15:58:57** **Escalated to GitLab.com Production** Your workflow manually escalated the incident to the escalation path [GitLab.com Production](https://app.incident.io/gitlab/on-call/escalation-paths/01K5YWH36WZZX2YT9HNRR0WMB0) **15:58:57** **Escalated to Incident Manager Oncall (IMOC)** Your workflow manually escalated the incident to the escalation path [Incident Manager Oncall (IMOC)](https://app.incident.io/gitlab/on-call/escalation-paths/01K8VN9TTAJH6N3WKNCAMNXQ40) **15:59:04** **Escalation acknowledged by Dan Ryan** Dan Ryan acknowledged the escalation **15:59:10** **Escalation acknowledged by Steve Abrams** Steve Abrams acknowledged the escalation **16:03:24** **Escalation acknowledged by Karthik Nayak** Karthik Nayak acknowledged the escalation **16:54:22** **Message from Dan Ryan** Dan Ryan pinned their own message ``` root@gitaly-cny-01-stor-gprd.c.gitlab-production.internal:~# uptime 16:54:12 up 25 days, 4:38, 6 users, load average: 223.32, 174.90, 162.48 ``` **16:55:14** **Message from Dan Ryan** Dan Ryan pinned their own message rebooted `gitaly-cny-01-stor-gprd.c.gitlab-production.internal`at this timestamp **17:02:03** **Update shared** Steve Abrams shared an update We've identified that the gitaly node `gitaly-cny-01-stor-gprd.c.gitlab-production.internal` was a major outlier in memory usage compared to other nodes, with a significant spike in anonymous memory pages after 13:00 UTC. See the [Grafana dashboard](https://dashboards.gitlab.net/goto/efh017bxpof7ke?orgId=1 "Grafana dashboard") for details. As a mitigation, the team has rebooted this node to address both elevated memory usage and ongoing slowness affecting merge requests, comments, and reviews for the gitlab-org group. During the reboot, users of gitlab-org/gitlab may have seen 500 errors for affected projects, which is expected while the node is restarting. We're monitoring the node as it comes back online and will assess whether this resolves the degraded performance for gitlab-org. **17:12:46** **Escalation acknowledged by Dan Ryan** Dan Ryan acknowledged the escalation **17:16:53** **Message from Raynard Omongbale** Raynard Omongbale's message was pinned by Cameron McFarland Looking inti the cny serial output after the reboot, it is failing to mount/unmount the /var/opt/gitlab ``` [FAILED] Failed to start File Syste…c-a41d-4f70-b082-c7e88629c016. See 'systemctl status "systemd-fsck@dev…\x2dc7e88629c016.service"' for details. [DEPEND] Dependency failed for /var/opt/gitlab. [DEPEND] Dependency failed for Local File Systems. ``` **17:19:28** **Message from John Skarbek** John Skarbek's message was pinned by Cameron McFarland ``` [[0;1;38;5;185mDEPEND[0m] Dependency failed for [0;1;39m/var/opt/gitlab[0m. [[0;1;38;5;185mDEPEND[0m] Dependency failed for [0;1;39mLocal File Systems[0m. [[0;32m OK [0m] Started [0;1;39mEmergency Shell[0m. [[0;32m OK [0m] Reached target [0;1;39mEmergency Mode[0m. You are in emergency mode. After logging in, typPress Enter for maintenance (or press Control-D to continue): ``` **17:33:46** **Update shared** Steve Abrams shared an update Users accessing the `gitlab-org` group on gitlab.com are seeing widespread 500 errors and outages due to a failure on a specific Gitaly storage node (`gitaly-cny-01-stor-gprd.c.gitlab-production.internal`). After an attempted reboot, the node failed to mount the data disk (`/var/opt/gitlab`) and entered maintenance mode for reasons not understood. The team has gained root access to the node via serial console and is investigating the disk state, including considering file system checks and recovery options. This incident is primarily impacting the `gitlab-org` group, but any customers or users whose pipelines or integrations access this group may also experience errors. We are actively working to restore service and will provide further updates as recovery progresses. **17:42:34** **Message from John Skarbek** John Skarbek pinned their own message Just documenting what I did: ``` 35s projects % gcloud compute instances add-metadata gitaly-cny-01-stor-gprd --metadata serial-port-enable=1 No zone specified. Using zone [us-east1-c] for instance: [gitaly-cny-01-stor-gprd]. Updated [https://www.googleapis.com/compute/v1/projects/gitlab-production/zones/us-east1-c/instances/gitaly-cny-01-stor-gprd]. WARNING: Some requests generated warnings: - Your VM is using global DNS. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns 13s projects % gcloud compute connect-to-serial-port gitaly-cny-01-stor-gprd No zone specified. Using zone [us-east1-c] for instance: [gitaly-cny-01-stor-gprd]. Updating instance ssh metadata...⠏Updated [https://www.googleapis.com/compute/v1/projects/gitlab-production/zones/us-east1-c/instances/gitaly-cny-01-stor-gprd]. Updating instance ssh metadata...⠛WARNING: Some requests generated warnings: - Your VM is using global DNS. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns Updating instance ssh metadata...done. [FAILED] Failed to start File Syste…c-a41d-4f70-b082-c7e88629c016. See 'systemctl status "systemd-fsck@dev…\x2dc7e88629c016.service"' for details. [DEPEND] Dependency failed for /var/opt/gitlab. [DEPEND] Dependency failed for Local File Systems. [ OK ] Started Emergency Shell. [ OK ] Reached target Emergency Mode. You are in emergency mode. After logging in, typPress Enter for maintenance serialport: Connected to gitlab-production.us-east1-c.gitaly-cny-01-stor-gprd port 1 (session ID: 446e819c126507d39e15052d8d335d68ab0f811389975e768aa197077ae3dce7, active connections: 1). root@gitaly-cny-01-stor-gprd.c.gitlab-production.internal:~# ``` What we saw: ``` root@gitaly-cny-01-stor-gprd.c.gitlab-production.internal:~# mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,relatime,size=185598668k,nr_inodes=46399667,mode=755,inode64) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=37124156k,mode=755,inode64) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) /dev/nvme0n1p1 on / type ext4 (rw,relatime,discard,errors=remount-ro) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10347) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) /var/lib/snapd/snaps/canonical-livepatch_384.snap on /snap/canonical-livepatch/384 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/canonical-livepatch_378.snap on /snap/canonical-livepatch/378 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/core20_2717.snap on /snap/core20/2717 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/core20_2769.snap on /snap/core20/2769 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/core22_2339.snap on /snap/core22/2339 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/core22_2411.snap on /snap/core22/2411 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/google-cloud-cli_436.snap on /snap/google-cloud-cli/436 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/lxd_38331.snap on /snap/lxd/38331 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/lxd_38469.snap on /snap/lxd/38469 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/google-cloud-cli_438.snap on /snap/google-cloud-cli/438 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/snapd_26382.snap on /snap/snapd/26382 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /var/lib/snapd/snaps/snapd_25935.snap on /snap/snapd/25935 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide) /dev/nvme0n1p15 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro) /dev/nvme0n3 on /var/log type ext4 (rw,relatime,discard) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime) root@gitaly-cny-01-stor-gprd.c.gitlab-production.internal:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 13M 1 loop /snap/canonical-livepatch/378 loop1 7:1 0 13.4M 1 loop /snap/canonical-livepatch/384 loop2 7:2 0 63.8M 1 loop /snap/core20/2717 loop3 7:3 0 63.8M 1 loop /snap/core20/2769 loop4 7:4 0 74M 1 loop /snap/core22/2339 loop5 7:5 0 74M 1 loop /snap/core22/2411 loop6 7:6 0 435.2M 1 loop /snap/google-cloud-cli/436 loop7 7:7 0 435.6M 1 loop /snap/google-cloud-cli/438 loop8 7:8 0 91.6M 1 loop /snap/lxd/38331 loop9 7:9 0 91.7M 1 loop /snap/lxd/38469 loop10 7:10 0 48.1M 1 loop /snap/snapd/25935 loop11 7:11 0 48.4M 1 loop /snap/snapd/26382 nvme0n1 259:0 0 40G 0 disk ├─nvme0n1p1 259:1 0 39.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0 part /boot/efi nvme0n2 259:4 0 15.6T 0 disk nvme0n3 259:5 0 50G 0 disk /var/log root@gitaly-cny-01-stor-gprd.c.gitlab-production.internal:~# root@gitaly-cny-01-stor-gprd.c.gitlab-production.internal:~# nvme0n2 Disk /dev/nvme0n2: 15.63 TiB, 17179869184000 bytes, 33554432000 sectorsdev/nvme0n2 Disk model: nvme_card-pd Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes ``` What we did: ``` fsck -y /dev/nvme0p2 reboot ``` **17:45:00** **Fixed at** Custom timestamp "Fixed at" occurred **17:56:11** **Monitoring at** Custom timestamp "Monitoring at" occurred **17:56:11** **Identified at** Custom timestamp "Identified at" occurred **17:56:11** **Status changed from Investigating → Monitoring** Steve Abrams shared an update Status: ~~Investigating~~ → Monitoring **18:13:26** **Message from Dan Ryan** Dan Ryan pinned their own message we need to run git-fsck against every project on gitaly-cny-01-stor-gprd node **18:19:14** **Message from Dan Ryan** Dan Ryan pinned their own message determine what led to the data disk filesystem becoming corrupted as part of a reboot **18:55:50** **Incident resolved and entered the post-incident flow** Steve Abrams shared an update Status: ~~Monitoring~~ → Documenting # Investigation Notes </details> Any details you may want to add about the investigation can go here. # Follow-ups **Follow-up** 1. [Run git-fsck against every project on gitaly-cny-01-stor-gprd](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28614) - This work is scheduled in the ~"group::tenant services" Q1 KTLO 2. [Determine what led to the data disk filesystem becoming corrupted after a Gitaly node reboot](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28615) - Discovered that there were disk errors for over a month on canary 3. Created https://gitlab.com/gitlab-cookbooks/gitlab-exporters/-/merge_requests/373 to add metrics for ext4 errors, until they are added to prometheus node exporter with https://github.com/prometheus/node_exporter/pull/3577 4. Created https://gitlab.com/gitlab-com/runbooks/-/merge_requests/10349 for alerting on filesystem errors. # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [ ] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock). - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [~] S1 incidents [require a public RCA within 7 days](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/#timeline-that-we-expect-for-reviews-to-be-completed) of the incident. If this review cannot be made public, [create a separate public RCA](https://handbook.gitlab.com/handbook/engineering/root-cause-analysis/#how-to-perform-an-rca). - [ ] Close the review before the due date - [ ] Go back to the incident channel or page and close out the remaining post-incident tasks