## Summary Some requests to gitlab.com occasionally stall for tens of seconds (seen up to ~2 minutes) before the client receives the first byte, sometimes past the client's own timeout. When it does return, the response is normal. First reported as ~15s hangs on `gitlab.com/jwt/auth` container-registry token requests, the same pattern shows up on other endpoints and on reused as well as fresh connections. Stopped reproducing on 2026-07-02 and closed as no longer reproducible, see Status. ## Runner impact (the original symptom) A stalled `/jwt/auth` token fetch during a CI image pull. The Docker daemon waits ~15s for response headers, gets none, and the job ends as `runner_system_failure` before its script runs. gitlab-runner now retries this failure class on a fresh connection, which usually succeeds, so the user-visible failure is mostly masked: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6873. That mitigates but does not explain it. ## Leading hypothesis: the Cloudflare Worker (Cells http-router) gitlab.com runs through the Cells `http-router` Cloudflare Worker (`production-gitlab-com-cells-http-router`), which reads a Flagship feature flag and makes one upstream call to the origin. In every request we have correlated, the origin serves quickly (Rails, Workhorse, and HAProxy all sub-second and `200`) and the stall is on the way back to the client. On a stalled request the Worker's wall time is tens of seconds (one organic example was 142s) on a few milliseconds of CPU. A staging request captured at every layer places that time inside Cloudflare: its own `edgeWorkerFetch` record shows the origin response arriving at the edge in about 80ms, and a packet capture on the origin LB confirms the full response acknowledged within ~150ms, yet the Worker does not receive it for tens of seconds. High wall time with near-zero CPU looks like the isolate being parked rather than doing work, so the delay is in the edge-to-worker handoff after the fetch completes, inside Cloudflare's Workers runtime. It is per request, not the connection or the origin: the same origin connection served hundreds of nearby requests without stalling. It reproduces on staging independent of production traffic and origin protocol. Production Worker observability is head-sampled at 1%, so organic stalls are rarely captured. http-router!1266 (https://gitlab.com/gitlab-org/cells/http-router/-/merge_requests/1266) closed that gap by logging upstream response-header timing from the Worker. Deployed to staging, it caught one organic stall from inside the Worker before the stalls stopped: 30.7s to response headers on a 200 (ray `a14d0d18cd75453e`, 2026-07-02 10:44 UTC). ## Correlation note The `Cf-Ray` the client sees is not the one HAProxy logs: Cloudflare assigns a separate ray to the Worker's origin-pull subrequest, linked to the eyeball request by `ParentRayID`. Join the Worker to Rails and Workhorse on `correlation_id` (the eyeball ray plus colo). HAProxy's logged client IP (`%ci`) is a per-request GCP translation, not the socket peer, so it is not a reliable connection key. The origin-side packet capture is. ## Status Closed as no longer reproducible. The last observed stall was 2026-07-02 10:44 UTC. Since then roughly 30 hours of soak against staging and production (~10M requests), including full business days, caught nothing. Nothing changed on our end in that window (the http-router change only logs), which suggests a fix on Cloudflare's side, unconfirmed. The evidence stands as-is: the delay sat between Cloudflare's edge completing the origin fetch and the Worker receiving the response, inside the Workers runtime. If it recurs: the reproducer soak is one pipeline trigger away (branches `diag/staging-soak-*` and `diag/prod-soak-3h`), the Worker now logs upstream header timing on every request, and the end-to-end capture note documents the full correlation method. ## References - End-to-end capture of one stalled request: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29267#note_3513902361 - http-router upstream-timing observability: https://gitlab.com/gitlab-org/cells/http-router/-/merge_requests/1266 - Runner retry mitigation: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6873 - CI reproducer (probe plus pipeline config): https://gitlab.com/igorwwwwwwwwwwwwwwwwwwww/hello-world/-/tree/diag/jwt-hang-probe