# [INC-6113: Gitlab.com is down (500)](https://app.incident.io/gitlab/incidents/6113) Generated by Vladimir Glafirov on 5 Dec 2025 10:11. All timestamps are local to Etc/UTC # Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | | ------ | ------ | | Customers Affected | All SaaS users| | Requests Affected | All public API's, [~155 million requests](https://dash.cloudflare.com/852e9d53d0f8adbd9205389356f2303d/gitlab.com/analytics/traffic?date-from=2025.12.05-0846&date-to=2025.12.05-0912) | | Incident Severity | Severity 1 | | Impact Start Time | Fri, 05 Dec 2025 08:40:00 UTC| | Impact End Time | Fri, 05 Dec 2025 09:26:00 UTC | | Total Duration | 25 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20939 | # Summary **Problem**: GitLab.com and all related services were down, returning HTTP 500 errors for all major endpoints. **Impact**: All users were unable to access GitLab.com and core functionalities—including deployments, feature flags, and internal tools like runbooks, dashboards, and operational platforms—from 08:45 to 09:12 UTC. 13 related incident alerts were triggered. Services have now been restored. **Causes**: A widespread [Cloudflare outage](https://blog.cloudflare.com/5-december-2025-outage/) prevented access to GitLab.com and related services. Cloudflare confirmed and resolved the issue at 09:20 UTC. **Response strategy**: We escalated the issue with Cloudflare support. Cloudflare implemented a fix at 09:12 UTC, and confirmed full resolution at 09:20 UTC. We are monitoring recovery and taking follow-up actions to reduce Cloudflare dependency and improve outage documentation. # What went well? Use this section to highlight what went well during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. 1. Quick identification it was Cloudflare (within 2 minutes) 2. Team coordination and communication was strong - the right people from the right teams showed up and engaged quickly given the high severity. 3. Blackbox monitoring and immediate incident escalation 4. Identifying downstream impacts (for example, sidekiq queues) # What was difficult? Use this section to highlight opportunities for improvement discovered during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. If the improvement seems like a simplest change, consider adding it as a corrective action above instead. Think about how to improve response next time, and consider any patterns pointing to broader issues, like “key person risk.” 1. Cloudflare is a known point of failure, but that knowledge did not make the situation easier to manage. 1. Loss of critical operational tools (runbooks, dashboards, ops.gitlab.net), some of which were a surprise, leaving us questioning what actions we could take. 1. Some users were unable to access incident.io itself during the incident 1. No ability to take direct action - only wait for Cloudflare 1. Difficulty assessing full impact without dashboards # Investigation Details <details> <summary> Timeline</summary> # Incident Timeline **2025-12-05** **08:40:00** **Impact started at** Custom timestamp "Impact started at" occurred **08:50:40** **Incident reported by François Rosé** François Rosé reported the incident Severity: Severity 1 Status: Investigating **08:50:45** **Escalated to GitLab.com Production** Your workflow manually escalated the incident to the escalation path [GitLab.com Production](https://app.incident.io/gitlab/on-call/escalation-paths/01K5YWH36WZZX2YT9HNRR0WMB0) **08:50:45** **Escalated to Incident Manager Oncall (IMOC)** Your workflow manually escalated the incident to the escalation path [Incident Manager Oncall (IMOC)](https://app.incident.io/gitlab/on-call/escalation-paths/01K8VN9TTAJH6N3WKNCAMNXQ40) **08:52:21** **Message from Andrew Newdigate** Andrew Newdigate's message was pinned by Donna Alexandra Cloudflare is down **08:52:24** **Message from Gonzalo Servat** Gonzalo Servat's message was pinned by Donna Alexandra looks like a pretty big Cloudflare outage **08:54:39** **Image posted by Rehab Hassanein** Rehab Hassanein posted an image to the channel Documenting for the future  **08:54:59** **Message from Vishal Tak** Vishal Tak's message was pinned by Donna Alexandra IMOC here. Starting a google meet - https://meet.google.com/uyt-chcx-qvq **08:56:37** **Message from Donna Alexandra** Donna Alexandra pinned their own message This is impacting everything we have that uses Cloudflare (including CustomersDot) **08:58:13** **Message from Elliot Forbes** Elliot Forbes's message was pinned by Donna Alexandra Cloudflare have updated their status page - https://www.cloudflarestatus.com/ - but it's not fully representative **09:01:07** **Message from Donna Alexandra** Donna Alexandra pinned their own message I've gone for the noisy approach with Cloudflare channel to try and get attention from someone in Cloudflare Support who is online. **09:04:00** **Image posted by Pravar Gauba** Pravar Gauba posted an image to the channel  **09:06:23** **Image posted by Rehab Hassanein** Rehab Hassanein posted an image to the channel Impact seems to have started around `08:45 UTC` ref: https://log.gprd.gitlab.net/app/r/s/41dCx  **09:06:50** **Message from Vishal Tak** Vishal Tak's message was pinned by Donna Alexandra Starting a thread here to document all our systems which are affected by Cloudflare 🧵 **09:07:09** **Message from Timo Furrer** Timo Furrer's message was pinned by Rehab Hassanein Pretty much aligns with updog information: https://updog.ai/status/cloudflare **09:07:37** **Message from Vishal Tak** Vishal Tak's message was pinned by Donna Alexandra https://runbooks.gitlab.com/ **09:07:41** **Message from Vishal Tak** Vishal Tak's message was pinned by Donna Alexandra https://dashboards.gitlab.net/ **09:08:21** **Message from Pierre Guinoiseau** Pierre Guinoiseau's message was pinned by Donna Alexandra https://ops.gitlab.net/ **09:08:26** **Message from Pierre Guinoiseau** Pierre Guinoiseau's message was pinned by Donna Alexandra https://argocd.gitlab.net/ **09:08:44** **Message from Pierre Guinoiseau** Pierre Guinoiseau's message was pinned by Donna Alexandra https://vault.gitlab.net/ **09:11:42** **Update shared** Rehab Hassanein shared an update GitLab.com and all related services remain inaccessible due to a widespread Cloudflare outage. The incident began around 08:45 UTC based on log data, and impacts all users and core functionalities, including deployments and feature flags. Multiple related incidents and monitoring alerts have been merged for unified response. The outage also affects internal tools such as runbooks, dashboards, and operational platforms. Status updates have been posted to the GitLab status page, and communications have been escalated with Cloudflare support for urgent assistance. A corrective action to migrate dashboards.gitlab.net off Cloudflare has been proposed, and a follow-up to create a runbook for Cloudflare outages is underway. Cloudflare has acknowledged issues with their Dashboard and API services during scheduled maintenance in the Detroit datacenter. Investigation and coordination are ongoing across all affected systems. **09:12:15** **Message from François Rosé** François Rosé's message was pinned by Donna Alexandra Looks like we're back online? **09:13:16** **Image posted by Rehab Hassanein** Rehab Hassanein posted an image to the channel A fix by cloudflare at `09:12 UTC`  **09:14:00** **Identified at** Custom timestamp "Identified at" occurred **09:14:19** **Status changed from Investigating → Fixing** Rehab Hassanein shared an update Status: ~~Investigating~~ → Fixing Cloudflare has implemented a fix for their dashboard and API services at 09:12 UTC, and GitLab.com is now back online. Some users have reported restored access to GitLab.com and related services. The outage began around 08:45 UTC and affected all users and core functionality, including deployments, feature flags, and internal tools. We will continue monitoring to ensure full recovery. Cloudflare's root cause was linked to issues during scheduled maintenance in the Detroit datacenter. **09:16:36** **Image posted by Rehab Hassanein** Rehab Hassanein posted an image to the channel ref: https://log.gprd.gitlab.net/app/r/s/6Yzh8  **09:17:03** **Image posted by Vladimir Glafirov** Vladimir Glafirov posted an image to the channel APDEX is climbing up  **09:17:30** **Image posted by Vladimir Glafirov** Vladimir Glafirov posted an image to the channel Error rate is getting down  **09:21:48** **Image posted by Igor Wiedler** Igor Wiedler posted an image to the channel sidekiq catchall is under pressure, queues are going up, we may see some impact on the sidekiq_queueing SLI (source (https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?orgId=1&from=now-1h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-shard=catchall))   **09:22:50** **Message from Vishal Tak** Vishal Tak's message was pinned by Donna Alexandra The following systems are all impacted by Cloudflare. Some of these are used by us in production incidents. Thus, we need to investigate if we need to move them off Cloudflare. - https://runbooks.gitlab.com/ - https://dashboards.gitlab.net/ - https://ops.gitlab.net/ - https://argocd.gitlab.net/ - https://vault.gitlab.net/ - packages.gitlab.com (http://packages.gitlab.com) - CustomersDot - incident.io (http://incident.io) itself as well(users can't login if Cloudflare is down) \cc @Rachel **09:23:47** **Image posted by Rehab Hassanein** Rehab Hassanein posted an image to the channel Wee seem to be back to pre-incident log levels: ref: https://log.gprd.gitlab.net/app/r/s/x2MpN  **09:26:00** **Monitoring at** Custom timestamp "Monitoring at" occurred **09:26:00** **Fixed at** Custom timestamp "Fixed at" occurred **09:26:01** **Status changed from Fixing → Monitoring** Rehab Hassanein shared an update Status: ~~Fixing~~ → Monitoring GitLab.com and all related services have recovered following Cloudflare's fix at 09:12 UTC. Users report restored access, and internal monitoring shows that error rates and request volumes are returning to pre-incident levels. APDEX scores and service metrics are also recovering across key components. All 13 attached incident alerts have been resolved. Internal systems such as runbooks, dashboards, and operational tools are accessible again, and CustomersDot is operational. We are monitoring resource saturation and Sidekiq queues, which showed transient spikes but are now stabilizing. We have confirmed the incident was caused by a widespread Cloudflare outage. Follow-up actions are underway to reduce Cloudflare dependency for critical services and improve documentation for similar future events. **09:30:33** **Image posted by Rehab Hassanein** Rehab Hassanein posted an image to the channel At `09:20 UTC` cloudflare have resolved their incident: https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q  **09:40:56** **Message from Vladimir Glafirov** Vladimir Glafirov's message was pinned by Rehab Hassanein #inc-6127-sidekiq-queueing-sli-apdex-slo-violation-on-catchall-b-shard-apdex as a consequence of this one I believe. Monitoring it **09:43:00** **Message from Donna Alexandra** Donna Alexandra's message was pinned by Rehab Hassanein FYI, we received a response from Cloudflare. **09:51:57** **Incident resolved and entered the post-incident flow** Vladimir Glafirov shared an update Status: ~~Monitoring~~ → Documenting **10:00:39** **Message from Rehab Hassanein** Rehab Hassanein pinned their own message Regarding dedicated - it seems like there was no impact for them :green-circle-check: # Investigation Notes </details> Any details you may want to add about the investigation can go here. # Follow-ups **Follow-up** **Owner** [Review all systems dependent on Cloudflare if they are used for incident management](https://app.incident.io/gitlab/incidents/6113?tab=post-incident) Steve Abrams [Move https://dashboards.gitlab.net/ off Cloudflare](https://app.incident.io/gitlab/incidents/6113?tab=post-incident) Andrew Newdigate [Corrective action: we need to put pressure on Incident.io (http://Incident.io)...](https://app.incident.io/gitlab/incidents/6113?tab=post-incident) Andrew Newdigate [Write a runbook for what to do with a cloudflare outage](https://app.incident.io/gitlab/incidents/6113?tab=post-incident) Rachel Nienaber [Move https://dashboards.gitlab.net/off cloudflare](https://app.incident.io/gitlab/incidents/6113?tab=post-incident) Heinrich Lee Yu # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [x] Close the review before the due date - [x] Go back to the incident channel or page and close out the remaining post-incident tasks