Incident Review: INC-6665 Pipelines are failing to upload artifacts due to 502 Bad Gateway

INC-6665: Pipelines are failing to upload artifacts due to 502 Bad Gateway

Generated by Amar Patel on 16 Jan 2026 20:51. All timestamps are local to Etc/UTC

Key Information

Metric	Value
Customers Affected	At least 12 customers affected
Requests Affected	Artifact uploads 763000 requests of this type with a sampling rate of 0.1% out of approximately 19151300 representing slightly less than 5% error rate
Incident Severity	Severity 2
Impact Start Time	Fri, 16 Jan 2026 09:00:00 UTC
Impact End Time	Fri, 16 Jan 2026 20:00:00 UTC
Total Duration	10 hours, 22 minutes
Link to Incident Issue	2026-01-16: Pipelines are failing to upload art... (#21096 - closed)

Summary

Problem: Pipelines were failing to upload artifacts due to 502 Bad Gateway errors, primarily when uploading large files.

Impact: Artifact uploads failed globally for both internal and customer CI/CD pipelines, with at least 12 customers affected. These failures disrupted pipeline execution and blocked feature flag rollouts for impacted jobs. Deployments were not blocked.

Causes: A recent deployment of a Cloudflare Worker in the HTTP router caused large artifact uploads to exceed the worker's memory buffer, triggering 'ReadableStream.tee() buffer limit exceeded' errors. This led Cloudflare to return 502 Bad Gateway errors for affected requests.

Response strategy: We rolled back the Cloudflare HTTP router deployment in both staging and production. After the rollback, Cloudflare dashboards showed a sharp drop to zero 502 errors for artifact endpoints. A test job was retried to confirm successful artifact uploads post-rollback. A follow-up was opened to investigate missing error reporting in Sentry.

What went well?

Use this section to highlight what went well during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared.

Rapid rollback capability - Once the root cause was identified as the HTTP router deployment, the team was able to quickly revert using an automated rollback button in the deployment pipeline. This enabled immediate recovery, with Cloudflare dashboards showing a sharp drop to zero 502 errors post-rollback and confirming successful artifact uploads on retry.
Effective cross-team collaboration and escalation - The team successfully escalated to Cloudflare support when internal logs showed no issues, and Cloudflare's trace tooling ultimately proved essential to identifying the ReadableStream.tee() buffer limit exceeded error in the Worker logs.

What was difficult?

Use this section to highlight opportunities for improvement discovered during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. If the improvement seems like a simplest change, consider adding it as a corrective action above instead. Think about how to improve response next time, and consider any patterns pointing to broader issues, like “key person risk.”

Lack of visibility into the HTTP router component - There were no alerts or obvious dashboard indicators that the router was the culprit. The 502 errors only became apparent after heavy filtering in Cloudflare dashboards, and because requests failed at the Cloudflare Workers layer, they never reached internal logs (Rails, Workhorse, HAProxy), creating a complete observability gap that significantly delayed root cause identification.
Inability to reproduce the issue in test environments - The team couldn't reproduce the ReadableStream.tee() buffer limit exceeded error in staging or manual tests, making it difficult to understand the specific conditions that triggered the failure. This suggests environmental or configuration factors remain unclear and warrant further investigation.
Deployment visibility and process gaps - The HTTP router deploys separately from the standard deployment process with no staging-to-production visibility, no deployment annotations in monitoring dashboards, and no clear awareness across the on-call team about where the router code lives or how it's deployed. This lack of integration with standard tooling made it harder to correlate the deployment with the incident.

Investigation Details

Timeline

Incident Timeline

2026-01-16

09:00:00 Impact started at

Custom timestamp "Impact started at" occurred

10:00:39 Incident reported by Ségolène Delaby Bouly

Ségolène Delaby Bouly reported the incident

Severity: Severity 2

Status: Investigating

10:00:44 Escalated to GitLab.com Production

Your workflow manually escalated the incident to the escalation path GitLab.com Production

10:00:45 Escalated to Incident Manager Oncall (IMOC)

Your workflow manually escalated the incident to the escalation path Incident Manager Oncall (IMOC)

11:09:59 Image posted by Pravar Gauba

Pravar Gauba posted an image to the channel

src: https://log.gprd.gitlab.net/app/r/s/Qqe9k

runner failed logs.

12:38:39 Update shared

Pavel Shutsin (via @incident) shared an update

We're continuing to investigate pipeline failures caused by 502 Bad Gateway errors during artifact uploads. Sample jobs show that artifact upload requests are not reaching backend services like Workhorse or Rails, and there are no matching logs on the runners.

Evidence suggests these requests may be hitting a rate limit before reaching our infrastructure. The rate limit configuration is managed in cloudflare-general.tf. The team is also reviewing recent changes to ClusterIssuer resources, but so far, these changes do not appear to be the root cause.

Users continue to experience intermittent 502 errors when running pipelines. Investigation into the rate limiting and related configuration changes is ongoing.

13:17:53 Image posted by John Jarvis

John Jarvis posted an image to the channel

Workhorse logs for the artifacts endpoint in case this wasn't posted yet https://log.gprd.gitlab.net/app/r/s/UQ5X8

no significant change here, no recent spike in 5xx or decrease in 2xx.

13:29:26 Image posted by John Jarvis

John Jarvis posted an image to the channel

we see an uptick for all countries so it isn't region specific

14:07:05 Update shared

Pavel Shutsin (via @incident) shared an update

We've confirmed that the 502 Bad Gateway errors affecting artifact uploads are being generated by Cloudflare when it fails to get a timely response from the origin (gitlab.com). Cloudflare logs show the edge waiting over 18 seconds without a response from the backend, resulting in a timeout and 502 error.

This issue appears to primarily affect POST requests from runners to the artifacts endpoint, with no clear region-specific pattern. Cloudflare analytics indicate a global increase in 502s for these requests, while backend logs (Workhorse/Rails) do not show corresponding errors or elevated failure rates. We have not identified any rate limiting or runner misconfiguration as the cause.

We are continuing to investigate possible backend performance bottlenecks and other upstream factors that could be causing these delayed responses.

14:22:21 Message from John Jarvis

John Jarvis's message was pinned by Shreya Shah

I picked a 502 reported by cloudflare and cannot find the corresponding request in the workhorse logs. I am pretty sure that narrows it down to nginx

15:04:50 Message from Maina Ng'ang'a

Maina Ng'ang'a's message was pinned by Pavel Shutsin

This is also blocking releases and we have an internal release that needs to go out before we can start next weeks patch release, both releases will require passing pipelines for the omnibus project (https://gitlab.com/gitlab-org/security/omnibus-gitlab/-/pipelines) which is failing with the same error.

15:15:24 Update shared

John Jarvis (via @incident) shared an update

We continue to see 502 Bad Gateway errors affecting artifact uploads in CI/CD pipelines. Cloudflare is returning these errors after waiting over 18 seconds for a response from the origin (gitlab.com), resulting in artifact upload failures for both internal and customer pipelines.

The issue is global and primarily impacts POST requests from runners to the artifacts endpoint. Cloudflare analytics confirm a spike in these 502 errors, while backend logs (Workhorse and Rails) do not show matching failures. Nginx logs are being reviewed for further clues, but no clear pattern has emerged.

The artifact upload failures are disrupting pipeline execution and blocking deployments and feature flag rollouts for affected jobs. Investigation is ongoing to identify the root cause between Cloudflare and the backend services.

15:44:27 Message from Cameron McFarland

Cameron McFarland pinned their own message

This is the current Cloudflare dashboard that I think shows the problem: https://dash.cloudflare.com/852e9d53d0f8adbd9205389356f2303d/gitlab.com/security/analy[…]gitlab.com&http-method=POST&user-agent~starts=gitlab-runner (https://dash.cloudflare.com/852e9d53d0f8adbd9205389356f2303d/gitlab.com/security/analytics?status-code=502&host=gitlab.com&http-method=POST&user-agent~starts=gitlab-runner)

16:07:04 Update shared

Donald Cook (via @incident) shared an update

We've identified that some artifact upload requests are taking over 18 seconds in Workhorse before returning a 201 status in HAProxy, while Cloudflare times out and reports a 502 error. This suggests most of the delay occurs within Workhorse, particularly during Google Cloud Storage (GCS) uploads, though we currently lack detailed timing information for these GCS steps.

No recent changes or clear signs of GCS slowdowns have been found on dashboards so far. The team is continuing to investigate for any backend bottlenecks or changes that could explain the sudden onset of these delays. Users continue to experience intermittent 502 errors during pipeline artifact uploads.

Relevant logs can be found for Rails and Workhorse, with another example here: Rails, Workhorse.

16:48:27 Image posted by Cameron McFarland

Cameron McFarland posted an image to the channel

https://dash.cloudflare.com/852e9d53d0f8adbd9205389356f2303d/gitlab.com/security/analy[…]%2F&path~contains=artifacts&user-agent~starts=gitlab-runner (https://dash.cloudflare.com/852e9d53d0f8adbd9205389356f2303d/gitlab.com/security/analytics?host=gitlab.com&path~starts=%2Fapi%2Fv4%2Fjobs%2F&path~contains=artifacts&user-agent~starts=gitlab-runner)

This graph shows the problem clearly. There is a large increase in 502 responses for artifacts.

17:07:43 Update shared

Donald Cook (via @incident) shared an update

Several customers (escalating from 6 to 12) are reporting 502 errors when uploading pipeline artifacts, causing partial disruption to CI/CD operations. Deployments are not currently blocked, but artifact upload failures continue to impact both internal and customer pipelines globally.

The team is focused on investigating the artifact upload flow, particularly delays in GitLab Workhorse and potential resource saturation. Logs from runners, Workhorse, and Rails are being analyzed to determine the root cause. Recent analysis shows most of the delay occurs in Workhorse, especially during uploads to Google Cloud Storage, but detailed timing from GCS is not available. Additional Workhorse log samples are being reviewed (latest logs).

A corrective action has been created to add timing information to GCS-related logs in Workhorse. The incident has been communicated on the status page, and the deployment block label was removed based on current understanding of impact. The team will continue to monitor customer reports, artifact upload metrics, and will reapply deployment blocks if evidence suggests further impact.

18:09:38 Update shared

Donald Cook (via @incident) shared an update

We are continuing to investigate the 502 Bad Gateway errors affecting artifact uploads in CI/CD pipelines. Cloudflare support has confirmed that the majority of these errors are POST requests and has started enabling tracing to help diagnose the issue. However, their tracing is limited by a 200 requests-per-second cap, and they have requested either a specific path or a separate test environment to proceed.

We have also explored whether artifact upload failures are related to large payload sizes, since many failing jobs involve sizable artifacts (1GB+), but this remains under investigation. Manual test uploads with large files have produced some "request entity too large" errors, which may be influenced by project artifact size limits or compression.

No changes have been observed after toggling the duo_access_through_namespaces feature flag, and there is no evidence of recent GitLab SaaS deploys or runner version changes correlating with the incident window. GCS dashboards related to Workhorse uploads are currently reporting no data, which may indicate a lack of telemetry or a data collection issue.

We are coordinating with Cloudflare and continuing to review logs, monitor impact, and test hypotheses about request size and upstream filtering. The incident remains visible on the status page.

19:07:25 Update shared

Donald Cook (via @incident) shared an update

We're investigating why 502 Bad Gateway errors are affecting artifact uploads and some job-related API requests in CI/CD pipelines. Analysis of recent failures shows these requests are consistently not reaching HAProxy or backend services, suggesting the errors are happening before our infrastructure handles them.

Cloudflare support is tracing the affected endpoints, and engineers are collaborating to capture a reproducible failing job for deeper analysis. No evidence of backend or runner configuration changes aligns with the incident's timing, and payload size does not appear to be a primary factor based on manual testing and Cloudflare feedback.

Some metrics—such as increased CI job trace archiving errors and queuing latency for certain runner types—show suspicious timing, but so far do not correlate directly with the artifact upload failures. The team is focused on collecting more data via Cloudflare traces and reviewing HAProxy logs to rule out any backend involvement.

19:24:43 Message from Hordur Yngvason

Hordur Yngvason's message was pinned by Rehab Hassanein

https://gitlab.com/gitlab-org/cells/http-router

19:39:31 Message from Hordur Yngvason

Hordur Yngvason pinned their own message

Link to the potentially problematic change identified by duo gitlab-org/cells/http-router!1030 (comment 3013264330)

20:04:50 Message from Cameron McFarland

Cameron McFarland pinned their own message

This was the error from Cloudflare worker logs:

ReadableStream.tee() buffer limit exceeded. This error usually occurs when a Request or Response with a large body is cloned, then only one of the clones is read, forcing the Workers runtime to buffer the entire body in memory. To fix this issue, remove unnecessary calls to Request/Response.clone() and ReadableStream.tee(), and always read clones/tees in parallel.

20:07:02 Update shared

Amar Patel (via @incident) shared an update

We've identified that the recent spike in 502 Bad Gateway errors during artifact uploads was linked to a Cloudflare Worker deployment in the HTTP router. The error, "ReadableStream.tee() buffer limit exceeded", appeared in Cloudflare worker logs and was triggered when large artifact uploads overwhelmed the worker's memory buffer. This affected both internal and customer pipelines globally.

A rollback of the HTTP router deployment has been initiated in both staging and production environments. Early data from Cloudflare dashboards shows a sharp decline in 502 errors for artifact endpoints following the rollback, with recent intervals reporting zero errors. A test job is being retried to confirm that artifact uploads now succeed post-rollback. You can view the Cloudflare dashboard here.

We've also opened a follow-up action to investigate why these worker errors were not reported in Sentry, as expected. Further monitoring is underway to confirm resolution.

20:16:43 Image posted by Cameron McFarland

Cameron McFarland posted an image to the channel

The 502s have dropped off quite a bit. I think we're ok. https://dash.cloudflare.com/852e9d53d0f8adbd9205389356f2303d/gitlab.com/security/analy[…]%2F&path~contains=artifacts&user-agent~starts=gitlab-runner (https://dash.cloudflare.com/852e9d53d0f8adbd9205389356f2303d/gitlab.com/security/analytics?path~starts=%2Fapi%2Fv4%2Fjobs%2F&path~contains=artifacts&user-agent~starts=gitlab-runner)

20:17:23 Message from Sarah Walker

Sarah Walker pinned their own message

Posterity: The error we saw was

ReadableStream.tee() buffer limit exceeded. This error usually occurs when a Request or Response with a large body is cloned, then only one of the clones is read, forcing the Workers runtime to buffer the entire body in memory. To fix this issue, remove unnecessary calls to Request/Response.clone() and ReadableStream.tee(), and always read clones/tees in parallel.

20:23:10 Identified at

Custom timestamp "Identified at" occurred

20:23:10 Incident resolved and entered the post-incident flow

Amar Patel shared an update

Status: ~~Investigating~~ → Documenting

Investigation Notes

Any details you may want to add about the investigation can go here.

Actions

Action

Owner

One thing I found is requests to the artifacts endpoint taking over 18 seconds are reported to return 201 in HAProxy. Cross-referencing those requests against Rails and Workhorse, we see that the request is quick inside Rails and almost all the time is spent in Workhorse. The workhorse logs (grouped by correlation ID) also show entries related to uploading to GCS, but unfortunately those don't contain timing information.

Unassigned

Follow-ups

Follow-up

Owner

Conduct an incident review for the artifact upload 502 outage

Amar Patel

Cell cloudflare worker needs more observability/alerting

Sarah Walker

Investigate http-router Sentry ingration

Sarah Walker

Corrective action: Add timing information to GCS-related logs in Workhorse

Ahmad Sherif

Review Guidelines

This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI.

For the person opening the Incident Review

Set the title to Incident Review: (Incident issue name)
Assign a Service::* label (most likely matching the one on the incident issue)
Set a Severity::* label which matches the incident
In the Key Information section, make sure to include a link to the incident issue
Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) The DRI for the incident review is the issue assignee.

For the assigned DRI

Edited Jan 23, 2026 by David Leach