Enable Topology Service client for gitlab-shell on gprd (Cells SSH routing)
Production Change
Change Summary
Enable the Topology Service gRPC client (Cells SSH routing) for gitlab-shell on gprd, so that GitLab Shell resolves the owning cell for each SSH Git operation through the Topology Service and reaches that cell's internal API over https://gitlab.com:443 instead of the in-cluster webservice-internal-api service.
This is the production counterpart of the staging enablement in k8s-workloads/gitlab-com!5836 and completes the rollout planned in gitlab-org/gitlab-shell#874, validated on staging in gitlab-org/gitlab-shell#864 (closed).
The configuration change itself is four lines per cluster:
gitlab:
gitlab-shell:
config:
topologyService:
enabled: true
cellEndpoint:
scheme: https
port: 443Warning
This change moves the SSH authentication critical path off in-cluster networking and onto the public Cloudflare/HAProxy edge for 100% of gitlab.com SSH Git traffic. It was reverted three times on staging before the root cause was found. Read Risk assessment before reviewing.
Status (updated 2026-11-06 12:30 UTC)
Important
Phase 1 attempt 4 (gprd-cny, !5966) went live at 11:13 UTC and was soaked for ~70 min. Revert !5967 merged at 12:20 UTC, but its deploy is on hold at the EOC's request (a gprd-cny Helm upgrade starts the canary migrations pod, see INC-16026 and production#23436). Routing stays on in canary until the next k8s-workloads pipeline from master at or after 667f73546.
- Fixed and confirmed in production: SSH certificates (790 routed certificate sessions, 0 cell 403s), deploy keys and IP restrictions, transport (every pack POST 200, 0
cannot rewind body), Topology Service (0 Classify failures). - Two edge-related failure modes block Phase 2 (details in Phase 1 attempt 4 outcome):
- Idle protocol-v2 streams reset at ~15 s (
PROTOCOL_ERROR): 0.044% of routed sessions. The cause is clients pausing afterls-refswhile gitlab-shell holds the HTTP request open. New prerequisite P14. - SSH logins failing on a ~10 s edge latency tail for
authorized_keys: 0.053% of connections, ~14× the canary baseline. New prerequisite P15.
- Idle protocol-v2 streams reset at ~15 s (
- Rollback triggers #1 (closed) and #2 (closed) ("any occurrence") can't be met as written, because the direct path logs
Internal API unreachabletoo. Rate-based replacements are proposed in P16.
Note
Before attempt 4 (2026-11-06, morning). Phase 1 attempt 4 was ready to schedule: enable !5966 (draft), revert !5967.
- P13 done. gitlab-shell v14.61.0 carries the SSH certificate namespace in the Shell JWT, and gitlab-org/gitlab!267101 (merged) (
6705929e4c2e) honours it under the existingcells_shell_jwt_key_claimsflag (on in gstg and gprd). Both are deployed to every gstg and gprd cluster. - P12 (e) passes on staging (2026-11-06 09:59 UTC, routing on, group with Enforce SSH certificates): certificate clone and push succeed through the cell, a plain key is denied by
/allowedexactly as on the direct path, and a deploy key still clones. See P12. - P1 re-verified 2026-11-06: ruleset
ad55efc9…at version 16, Disable Gitlab-shell SSH buffering enabled. - Remaining before
change::scheduled: EOC and release-manager acknowledgement, and a change window outside a gprd auto-deploy.
Note
Attempt 3 (2026-10-30). Phase 1 attempt 3 (gprd-cny, !5950, live from 17:57 UTC on 2026-10-30) is being rolled back via !5951.
- Transport and authorization for SSH keys are fixed. All 3,393 routed pack POSTs returned 200 (p50 0.43 s, p99 0.58 s), with no
PROTOCOL_ERRORorcannot rewind body. Key-authenticated routed sessions: 3,352 OK and 2 failed (0.06%). The deploy-key and client-IP fix (P11) works. Attempt 2 failed ~8.8% here. - SSH-certificate users fail. In groups that enforce SSH certificates, the cell answers 403 on
ssh-upload-packafter/allowedreturned 200: 3 of 7 certificate sessions, across 2 groups. The cell's access check gets no certificate namespace, because the Shell JWT doesn't carry it. That's P12 case (e), which wasn't tested on staging. A new prerequisite, P13, covers the fix. Details: Phase 1 attempt 3 outcome. - P1 had silently regressed. The Cloudflare buffering rule on
gitlab.comwas deleted on 2026-10-27 by a provider v4 Terraform apply (config-mgmt!15703). It's now restored, and managed in Terraform on provider v5 (config-mgmt!15834, config-mgmt!15835, config-mgmt!15837, config-mgmt!15838). See P1.
Note
Earlier attempts. Phase 1 (gprd-cny) was executed on 2026-10-12 14:32–15:20 UTC and rolled back. Under production load (~50 SSH ops/s) about 2.3% of routed pulls failed with Internal API unreachable: net/http: cannot rewind body after connection loss (rollback trigger #2 (closed)). Root cause and fix: gitlab-shell!1605, validated on gprd-cny with a custom image at 0.00% over ~4,450 routed operations. Details in Phase 1 outcome.
Blocked again. Phase 1 attempt 2 (!5909, 2026-10-14 11:15 UTC) fixed the transport problem (0 cannot rewind across 5,481 routed operations), but ~8.8% of routed pulls failed with 404/403 from the cell: command success was 89.7% vs 97.3% without routing. Rollback !5910 is with the EOC. Root cause: the cell re-authorizes the routed request with only the user from the Shell JWT and the request's own IP, so deploy keys act as their owner (or the Ghost user) and group IP restrictions are checked against the Cloud NAT IP. Needs a Rails fix plus a gitlab-shell change (P11) and new staging tests (P12). Details: Phase 1 attempt 2 outcome.
Full staging evidence: gitlab-org/gitlab-shell#874 (comment).
| # | Failure mode | Fix | Staging | gprd |
|---|---|---|---|---|
| 1 | Idle pack-body streams reset by the Cloudflare Workers runtime at ~15s | Cloudflare Configuration Rule disabling body buffering on */ssh-upload-pack and */ssh-receive-pack (P1) |
cloudflare-cfgrules-gstg) |
cloudflare-cfgrules-gprd). It had been deleted 2026-10-27 by config-mgmt!15703 |
| 2 | git push stream left idle while Git's LFS pre-push hook runs, then reset |
gitlab-org/gitlab-shell!1561 (merged): advertise refs with one request, open the push request only once the client starts sending. Shipped in v14.57.4 | ||
| 3 | Git LFS PUT gets HTTP 524 after exactly 400s, and every retry restarts from byte 0 |
Cloudflare proxy_read_timeout 400s → 1800s: gstg config-mgmt!15473, gprd config-mgmt!15474 (P9) |
||
| 4 | remote: ERROR: cells routing: receive-pack response: Not found when Git's pre-push hook runs longer than ~60s |
gitlab-org/gitlab-shell!1570 (merged): sign the 1-minute Shell JWT for each request instead of once per session (P6) | v14.58.0-20261005.083629, which includes gitlab-shell!1570) |
|
| 5 | Cell authorizes routed deploy keys as their owner or the Ghost user, and checks group IP restrictions against the Cloud NAT IP (404/403) | gitlab-org/gitlab-shell!1619 (merged) (JWT claims gl_key_id, gl_key_type, check_ip; v14.60.0) plus gitlab-org/gitlab!265396 (merged) behind cells_shell_jwt_key_claims (P11) |
||
| 6 | Cell returns 403 for routed SSH-certificate users in groups that enforce SSH certificates (no certificate namespace in the Shell JWT) | gitlab-shell 0a180921 (v14.61.0) plus gitlab-org/gitlab!267101 (merged), same flag (P13) |
||
| 7 | Routed protocol-v2 upload-pack stream reset after ~15 s of idleness (stream error: … PROTOCOL_ERROR; received from peer) when the client pauses after ls-refs |
Request-body framing with 5 s keepalives: gitlab-org/gitlab-shell!1663 (merged) and gitlab-org/gitlab!269512 (merged), behind cells_ssh_body_framing (P14); a Cloudflare-side change isn't possible |
not reproduced (staging traffic too low) | |
| 8 | SSH login fails when /api/v4/internal/authorized_keys over the edge hits gitlab-shell's 10 s timeout (Cloudflare ATL latency tail, no origin response) |
Hedged retry for authorized_keys, or keep the internal API in-cluster (P15) |
not reproduced |
Rollout MRs (P7): all eight are pre-created as drafts, one enable and one revert MR per phase.
| Phase | Environment | Enable | Revert |
|---|---|---|---|
| 1 | gprd-cny |
!5950 (attempt 3, merged 2026-10-30 17:32 UTC); attempt 2 !5909 and attempt 1 !5877 were merged and reverted. Attempt 4: !5966 (merged 2026-11-06 11:08 UTC, live from 11:13). | Attempt 4: !5967 (merged 12:20 UTC; deploy held, see Status); !5951 merged for attempt 3; !5910 merged for attempt 2; !5881 closed as stale |
| 2 | gprd-us-east1-b |
!5878 | !5882 |
| 3 | gprd-us-east1-c |
!5879 | !5883 |
| 4 | gprd-us-east1-d |
!5880 | !5884 |
Change Details
- Services Impacted - ~"Service::GitLab Shell" ServiceGit ServiceCloudflare ServiceHAProxy ServiceTopology Service
- Change Technician - @vglafirov
- Change Reviewer - {+TBD: spin the Reviewer Roulette scoped to
production; must not be the current EOC+} - Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM) - {+TBD: to be scheduled once all Prerequisites are complete. The original
2026-09-17 09:00slot was dropped.+} - Time tracking - 6.5h (5.5h execution across four phases with soak windows, 1h reserved for rollback)
- Downtime Component - None expected. SSH Git availability is at risk throughout, but no planned downtime and no maintenance window required.
Risk assessment
What actually changes at runtime
Today on gprd, every SSH Git operation authenticates in-cluster:
msg: "Finished HTTP request"
url: http://gitlab-webservice-internal-api.gitlab.svc.cluster.local:8181/api/v4/internal/allowed
duration_s: 0.084 status: 200With the Topology Service client enabled, the same call leaves the cluster. Measured on gstg after !5836:
msg: "Finished HTTP request"
url: https://staging.gitlab.com:443/api/v4/internal/allowed
duration_s: 0.17 - 0.34 status: 200So this change does two things, not one:
- The Git pack transfer (
ssh-upload-pack/ssh-receive-pack) is routed to the cell endpoint returned by the Topology Service. /api/v4/internal/allowedand/api/v4/internal/authorized_keys— the authentication critical path for every SSH connection — also traverse the public edge.
Blast radius
| Value | |
|---|---|
gitlab-shell pods, gprd-us-east1-{b,c,d} |
100 each (KEDA HPA min 100 / max 150) |
gitlab-shell pods, gprd-cny |
8 |
| Git SSH ops observed, 3 pods over 66s | 829 git-upload-pack, 25 git-receive-pack |
Fleet-wide SSH rate (measured 2026-10-12, haproxy_backend_sessions_total{env="gprd",backend="ssh"}) |
~1,600 sessions/s now, ~1,990/s 7-day peak |
gitlab-shell → internal API today (gitlab_shell_http_requests_total, in-cluster) |
~4,400 req/s (~2.75 per SSH session) |
| Added edge load when fully rolled out | ~6k req/s, ~7.5k at SSH peak (+13% on the HAProxy https frontend). See P5. |
| Staging validation volume for comparison | 134 ops over 33 minutes (~0.07/s) |
Production carries roughly four orders of magnitude more SSH traffic than the window in which this was validated on staging. That gap is the single largest source of residual risk and it cannot be closed by more staging testing.
SLIs directly threatened
From metrics-catalog/services/git.jsonnet:
gitlab_sshd— apdex ongitlab_shell_sshd_session_established_duration_seconds_bucket, satisfied 1s / tolerated 5s. The added edge latency lands squarely inside this budget.workhorse_auth_api— apdex satisfied threshold 1s, with the in-repo comment "1s is too slow for an auth request". Note this SLI is emitted byinternal-apiand will see traffic shift away from it, which is itself a signal: a sharp drop ingitlab_workhorse_http_requests_total{type="internal-api"}is the confirmation the change took effect.gitlab_shell—haproxy_backend_sessions_total{backend=~"ssh|altssh"}.gitservice error ratio, monitoring threshold 0.9999.
Why this failed three times on staging
Attempts !5810, !5823 and the gstg-cny variant were all reverted. Every Git pack transfer failed, clustered at 15.04–15.07s:
http2: Transport: cannot retry err [stream error: stream ID 1;
PROTOCOL_ERROR; received from peer] after Request.Body was writtenThree hypotheses were raised and are now retracted: Cloudflare terminating long-lived streams, a ~60s intermediary idle client-body timeout, and the pull-body fix in gitlab-shell!1546 (shipped in 14.57.3). Pods were verified running v14.57.3-20260907.103559 and still failed at 15s.
Isolation testing identified the real cause:
| Target | Result |
|---|---|
internal-gateway.gstg:11443 |
held idle request body 60s |
kas.staging.gitlab.com — same CF zone, no Worker route |
survived 60s |
| Any Worker on the production runtime, including a 25-line probe that never reads the body | reset at ~15.1s |
It is a Cloudflare Workers runtime reset of the idle inbound HTTP/2 request body at ~15s. Not http-router code, not HAProxy, not Workhorse. The mitigation is to disable request and response body buffering on the pack paths, which is Prerequisite P1 below and is the reason this CR cannot proceed without a Cloudflare change on the production zone.
git push with Git LFS: failure modes found on 2026-09-23
Pulls and small pushes were fine, but a normal single-command git push that uploads LFS objects still failed. Git opens git-receive-pack and receives the ref advertisement before running the LFS pre-push hook. The hook then uploads the objects over HTTPS, which can take many minutes, and only after that does Git send its commands and pack.
- Idle push stream. Before gitlab-shell v14.57.4, the Cells push path held one streaming
POST /ssh-receive-packopen throughout the hook. Cloudflare cancelled it after ~15s (Worker outcomecanceledat 14,995 ms, Raya3c6b5edfbc3b574, HAProxyterm_state=CD--). The pack then went into a dead stream:PROTOCOL_ERROR,unexpected disconnect. Fix: gitlab-org/gitlab-shell!1561 (merged) (v14.57.4). It fetches the advertisement with a short0000request, then opens the real push request only once the client starts sending. - Expired Shell JWT. With gitlab-shell!1561, the second request can start minutes after the first, but it reused the Shell JWT (1-minute TTL) signed at session start. Rails rejects the expired token and answers
404by design:remote: ERROR: cells routing: receive-pack response: Not found. On staging, the secondssh-receive-packreturned 404 in 174 ms, 10m44s after the advertisement. Fix: gitlab-org/gitlab-shell!1570 (merged). It signs the JWT for each request, for push and pull. Pull wasn't affected in practice, because its single request starts immediately and the JWT is only checked when a request starts. - Cloudflare 400s read timeout on LFS uploads (not Cells-specific). A LFS object upload is a single
PUTthat gets no response until the whole body has arrived, soproxy_read_timeout(400s on both zones since 2020) covered the whole upload. The result was 524 at 400,220 ms (Raya3fbfa2bc8280cb6), with only 268,953,929 bytes received. Thebasicadapter can't resume uploads, so every retry restarted from byte 0. Fix: raiseproxy_read_timeoutto 1800s on both zones (config-mgmt!15473 / config-mgmt!15474, now applied).
The object transfer itself never touches gitlab-shell. Only git-lfs-authenticate is routed through the Topology Service; objects go directly over HTTPS to Workhorse.
Staging evidence
Observed on gstg on 2026-09-14 after !5836, window 09:46:40–10:19:25 UTC:
| Metric | gstg zonal b/c/d | gstg-cny |
|---|---|---|
| Commands | 108 pull / 26 push | 5 pull / 2 push |
command executed successfully |
133 / 134 | 7 / 7 |
| Pack requests non-200 | 0 / 134 | 0 / 7 |
PROTOCOL_ERROR |
0 | 0 |
Internal API unreachable |
0 | 0 |
| Session p50 / p95 / max | 1.68s / 9.85s / 431.23s | 1.77s / 2.98s / 2.98s |
The decisive data point is a complete gitlab-org/gitlab.git clone:
10:09:49.942 .../gitlab-org/gitlab.git/ssh-upload-pack 200 0.38s <- headers
10:17:00.418 session: handleShell: command executed successfully
10:17:00.418 connection: handleRequests: done duration_s: 431.23430.5 seconds of continuous pack streaming, completed, zero PROTOCOL_ERROR — 28x past the wall that killed the three prior attempts. One earlier 210s session ended in a clean client-side EOF with no upstream error; the same clone from the same client succeeded nine minutes later, so it is treated as an operator-aborted attempt rather than an infrastructure limit.
Git LFS evidence (2026-09-22 → 2026-09-23, gitlab-com/gl-infra/cmbr on gstg)
| Test | Configuration | Result |
|---|---|---|
Standalone git lfs push, 1 GiB |
before the timeout raise | upload_finalize 200 |
git lfs fetch of 8 objects, including 524 MB and 105 MB |
— | git lfs fsck --objects OK |
| Single-command push, 1 GiB | 400s timeout, v14.57.4 | |
| Single-command push, 2 GiB | 1800s timeout, v14.57.4 | |
| Single-command push, 2 GiB | 1800s timeout, custom image with gitlab-shell!1570 on all gstg clusters (b, c, d, cny), originals scaled to 0 | ssh-receive-pack 200 at +9m07s. 1a063cc..f3d401d HEAD -> vg/test-lfs-rule |
The custom image was gitlab-shell:dev-vg-cells-jwt-3d03b493 (sha256:2b794e3c…): the current staging image with only the binaries rebuilt from gitlab-shell!1570. Staging has since been restored to its normal deployments.
Known gaps carried into production
- Git LFS: validated on staging with the gitlab-shell!1570 fix, pending re-validation on the standard image. See Git LFS evidence and P4/P10. The earlier
F4 … LFS content differsresult from the 2026-09-13 matrix wasn't reproduced: every downloaded object passedgit lfs fsck --objects. - Pure-SSH LFS (
git-lfs-transfer) is not enabled (lfs.pure_ssh_protocol: false). Git LFS 3.x tries it first, getsUnknown command, and falls back to HTTPS, adding ~1.5s per LFS operation. This isn't a regression, but users will see the error text in verbose output. - LFS uploads are still limited to 1800s per object. At 1800s, a 5 GiB object (the zone's
max_upload) needs more than ~2.9 MB/s. Slower uploads still get a 524 and restart from zero. This applies to gitlab.com as a whole and isn't specific to Cells. - Disabling body buffering disables Cloudflare WAF body inspection on
*/ssh-upload-packand*/ssh-receive-pack. Requires security sign-off for the production zone. - Routing the internal API over the public edge is a registered risk: StORM #61 and prod-eng#13744. Long term the Topology Service needs to advertise an internally-routable endpoint; its
addressdoubles as the mTLS certificate CN, so it cannot simply be changed to an internal host today. - No feature flag. The only control is Helm values per cluster, so rollback is a deploy, not a toggle. See Rollback.
- Terraform drift: resolved 2026-10-30. This risk came true: the
gprdprovider v4 apply of config-mgmt!15703 deleted both hand-added rules fromgitlab.comon 2026-10-27 at 03:05 UTC (rulesetad55efc9…version 14 → 15), and its plan didn't show it. Both zones' Configuration Rules now live in provider v5 environments,cloudflare-cfgrules-gstgandcloudflare-cfgrules-gprd, with the buffering rules codified (config-mgmt!15834, config-mgmt!15835, config-mgmt!15837, config-mgmt!15838). The rest of this bullet is kept for history. On staging, two rules added by hand live outside Terraform: Disable Gitlab-shell SSH buffering (pack paths) and a test rule disabling buffering on/gitlab-lfs/objects/and/info/lfs/. gstg applies since 2026-09-14 haven't removed them, but every config-mgmt plan touching the zone's config ruleset must be checked. The LFS-path rule is not required for this change, because the524was caused by the read timeout, not by buffering. It is a leftover from testing and can be removed from staging.
Phase 1 outcome (2026-10-12)
| UTC | Event |
|---|---|
| 14:32 | !5877 merged; rollout complete ~14:50. All 8 pods routed via https://gitlab.com:443, v14.58.0. |
| 14:49–14:53 | 11,102 SSH commands, 10,518 routed pulls, 272 routed pushes. /allowed and authorized_keys over the edge: 22k requests, all 200. Classify p50 well under baseline. |
| 14:50–14:53 | 241 routed pulls failed (2.3%) with cannot rewind body after connection loss, in bursts of up to 7/s, each failing in ~40 ms. 7 PROTOCOL_ERROR, 11 upload-pack: EOF. Zonal clusters without routing: 0 in the same window. Trigger #2 (closed) fired. |
| 15:20 | Reverted via 9e8ea73f5 (direct revert; !5881 left open). Pods back on in-cluster routing at 16:04. |
Root cause. Each gitlab-shell pod multiplexes its pack requests over a shared HTTP/2 connection to the edge. When Cloudflare closed that connection (idle close or GOAWAY) with requests in flight, Go could not retry them: the pack body streamed SSH stdin through an io.Pipe, which has no Request.GetBody and cannot be replayed. In-memory requests (/allowed, authorized_keys) retry transparently and were unaffected. Staging never showed it because at ~0.07 ops/s every request got a fresh connection.
Fix. gitlab-shell!1605: a replayable pack body (git.ReplayBody, 1 MiB limit, streaming preserved) so the transport resends on a new connection; plus Topology Service NotFound moved from a WARN and status="fail" to debug and status="not_found" (it produced 2,041 warnings in 3.5 min and would have tripped trigger #11 (closed) on healthy traffic).
Validation on gprd-cny (16:02–16:13 UTC). Custom image from 6eb5ea87 as 2 extra pods in the canary Service (~20% of canary SSH), routing on: 4,319 pulls + 129 pushes, 0 cannot rewind failures (baseline would be ~100), 0 NotFound warnings; PROTOCOL_ERROR/EOF at the same low rate as before (unrelated: clients idle >15s mid-negotiation). Test pods and image removed afterwards. Evidence: gitlab-shell!1605 (comment).
Also learned. Classify is issued on every SSH op; Topology Service and edge handled ~50 ops/s without visible latency impact. P5 measurements for canary are in the comment above.
Phase 1 attempt 2 outcome (2026-10-14)
Transport and classification: fixed and healthy.
| Result | |
|---|---|
| gitlab-shell | v14.59.0 on all 8 canary pods (includes gitlab-shell!1605); rollout complete 11:15 UTC |
cannot rewind body failures |
0 across 5,481 routed operations (attempt 1: 2.29%) |
| Topology Service Classify | ~107/s ok, ~10/s not_found (matches unknown SSH keys: in-cluster authorized_keys 404 at ~8.9/s), 0 fail, p50 18 ms / p99 34 ms |
| Routing | 100% of routed pack requests to https://gitlab.com:443, which is correct with only cell-1 in production |
Authorization on the cell: broken. Routed operations, gitlab-shell logs, 122 s on 8 pods:
| Total | OK | 404 | 403 | Other (EOF, disconnects) | |
|---|---|---|---|---|---|
| Pulls | 5,329 | 4,780 (89.7%) | 383 (7.2%) | 84 (1.6%) | 82 (1.5%) |
| Pushes | 148 | 143 (96.6%) | 2 (1.4%) | 0 | 3 |
Without routing (zonal b, same period): 97.3% of SSH commands succeeded, with no pack-level 403/404. In every failing session /api/v4/internal/allowed returned 200. Then the routed POST /<project>.git/ssh-upload-pack was rejected by the cell, and the session exited with status 1.
Root cause. On the direct path, gitlab-shell authorizes once (/allowed with key_id and the client's check_ip) and then talks to Gitaly. On the routed path, the cell's Repositories::GitHttpController re-authorizes using only the Shell JWT gl_id and the request's own IP:
- Deploy keys (403).
/allowedreturnsgl_id: Gitlab::GlId.gl_id(actor.user). For a deploy key that's the key's owner, or the Ghost user when there is none (DeployKey#user). The cell authenticates as that user, so the deploy key's own permissions are never checked. Every 403 in a 47-request Rails sample wasmeta.user: ghost1, across 9 root namespaces. - Client IP (404). The cell sees
remote_ip= the Cloud NAT IP (34.74.226.x), not the client's IP (check_ipin/allowed). Group IP restrictions therefore deny the read. 82% of the 404s were in Premium/Ultimate groups (against 58% of successful routed pulls), and 8% were free (against 34%). The 3 free-plan 404s fit deploy keys whose owner has lost access.
Also affected, even when requests succeed: routed requests are attributed to the deploy key's owner; protected-branch deploy-key push rules don't apply; and audit IPs and IP restrictions use GitLab's NAT range. IP restrictions fail closed today, but fail open for any group allowlisting 34.74.226.0/24.
Why staging missed it: gstg Rails logged only 3 routed SSH requests in 24 h, all gitlab-qa with a personal key. The whole validation matrix (P4, P10) used personal keys, with no deploy keys and no IP-restricted groups.
Projected at full rollout: ~140 failed pulls/s (~8k/min), mostly paying customers' CI/CD and GitOps automation. The failure is deterministic, so retries don't help.
Phase 1 attempt 4 outcome (2026-11-06)
Summary. Attempt 4 confirms every earlier fix in production, including the P13 SSH-certificate fix that attempt 3 was rolled back for. Two failure modes are left. Both come from the Cloudflare edge path, both run at about 0.05% of routed sessions, and neither is visible in apdex. Neither breaks the SLO at canary scale, so canary was kept on to soak, but at full rollout they would add up to about 120 failed SSH operations per minute. They block Phase 2.
Timeline (UTC)
| Time | Event |
|---|---|
| 09:59 | P12 (e) passes on staging (routed SSH certificates in a group that enforces them). |
| 11:00–11:08 | #production thread with @sre-oncall. EOC (Nidhey Indurkar) approves !5966 and adds eoc_approved. |
| 11:08 | !5966 merged. Pipeline 6525192; gprd-cny Helm revision 13853 at 11:13:24. |
| 11:13:53–11:15:32 | All 8 canary pods replaced: gitlab-sshd v14.61.0-20261106.083153, topology_service in config.yml, TLS files present. |
| 11:15 | Classify from canary steady at ~145 req/s ok, ~5 req/s not_found. |
| 11:18–11:28 | Canary gitlab_sshd error ratio 0.0011 → 0.0019 (baseline 0.0001–0.0003, trigger #3 (closed) at 0.002). Apdex unchanged. |
| 11:20–12:15 | Investigation: shell, Workhorse, Gitaly, Rails and Sidekiq logs; Grafana; Cloudflare analytics. Results below. |
| 12:16 | Decision to revert, to stop the exposure while the fixes are built. |
| 12:20 | !5967 merged. |
| 12:21 | Revert pipeline 6525575 cancelled at the EOC's request (Shreya Shah): a gprd-cny Helm upgrade starts the canary migrations job, which set off INC-16026 earlier today. It waits for production#23436 (delete ci_partitions row 118). |
| 12:22 | Release-manager auto-deploy 6525472 upgrades gprd-cny (Helm revision 13854). It was cut at 8dd5204bc (attempt 4 enable), so routing stays on. The revert ships with the next pipeline cut from master at or after 667f73546. |
Results, soak window 11:15–12:25 UTC (70 min)
Compared with the same canary before routing (09:15–11:13) and with gprd-us-east1-b and -c in the same window. Sources: esquery on pubsub-shell-inf-gprd*, Grafana mimir-gitlab-gprd and mimir-runway.
| Metric | Canary, routing on | Canary baseline | Zonal b + c |
|---|---|---|---|
| SSH commands | 292,855 | 502,756 | 3,938,867 |
| Routed pack sessions | 276,237 upload-pack + 7,331 receive-pack | n/a | n/a |
| Non-zero exit status | 2.54% | 2.40% | 2.36% |
PROTOCOL_ERROR (failure mode A) |
125 (0.044% of routed) | 0 | 0 |
cells routing: … EOF (client hung up, not a failure) |
230 (0.081% of routed) | n/a | n/a |
authorized_keys 10 s timeouts (failure mode B) |
224 (0.053% of 425,656 connections) | 28 (0.004% of 738,442) | 0 |
/allowed cancelled by the client |
68 | 1 | 2 |
| SSH-certificate routed sessions | 790, 3 non-zero, 0 pack 403s | n/a | n/a |
| Pack POSTs | all 200; p50 0.42 s, p99 0.56 s | n/a | n/a |
| Topology Service Classify | ~145/s ok, ~5/s not_found, 0 fail, 1 retried timeout; p99 44–46 ms; server apdex 1.000, error ratio 0 |
none | none |
gitlab_sshd apdex |
0.9996–0.9997 | 0.9994–0.9998 | 0.9996–0.9998 |
gitlab_sshd error ratio |
0.0009–0.0019 | 0.0001–0.0003 | 0.0001–0.0003 |
workhorse_auth_api apdex (cny) |
0.9947–0.9978 | 0.9979–0.9987 | n/a |
Overall, routing costs about 0.14 percentage points of SSH sessions, against the canary's own baseline.
Confirmed working
- SSH certificates (P13): 790 routed certificate sessions with 0 cell 403s. Attempt 3 had 3 of 7 fail.
- Deploy keys and client-IP restrictions (P11): no repeatable 403/404 class on routed pack requests.
- Transport (P6, !1605): 0
cannot rewind body. Every pack POST returned 200, and routed fetches that keep moving data ran for 16.5 s to over 120 s. - Cloudflare buffering rule (P1): active; streams that carry data are not cut.
- Topology Service: no client or server errors, and no measurable latency impact.
Failure mode A: idle protocol-v2 streams reset after ~15 s (PROTOCOL_ERROR)
68 sessions traced end to end, from gitlab-shell (canary) through Workhorse (gitlab.com, all via Cloudflare ATL) to Gitaly:
- The client authenticates and sends its first request. The routed
POST …/ssh-upload-packgets 200 in ~0.4 s. - gitlab.com answers in full. In 39 of 67 sessions the request was only
ls-refs(Gitaly logspackfile negotiation complete, command: ls-refs, wants: 0, 173–418 response bytes). Some had already received the whole pack (up to 9.4 MB). - The client then sends nothing: no next command, flush or EOF. Protocol v2 lets the client wait before its next command, so both directions of the HTTP/2 stream are idle.
- After 14.6–16.0 s the stream is reset. gitlab-shell gets
stream error: stream ID 1; PROTOCOL_ERROR; received from peerand exits 1. At the same instant Workhorse ends the request (duration_ms≈ 15,000, status 200) and Gitaly cancels upload-pack (Canceled … signal: terminated: context canceled,time_ms≈ 15,000).
- The clients are ordinary git CLI users: protocol v2 in 57 of 68; macOS, Windows and Linux; git 2.30 to 2.56; 62 different repositories; spread evenly over all pods.
- gitlab-shell keeps the stream open by design:
readUploadPackRequest(internal/command/githttp/util.go) closes the HTTP request body only when it seesdoneor SSH stdin reaches EOF.ls-refsnever sendsdone, and neither does a fetch the server ends early withready.
Proof that the failure is the idle limit on the routed path. Gitaly SSHUploadPackWithSidechannel, 11:20–11:35 UTC, all gprd:
| Session length | Direct (gitlab-shell client) |
Routed (gitlab-workhorse via edge) |
|---|---|---|
| ≥ 14.5 s, all kinds | 1,012 of 1.19M (0.085%) | 45 of 60k (0.075%) |
ls-refs only, 14.5–16.5 s |
27 | 18 |
ls-refs only, 16.5–30 s |
102 | 3 |
ls-refs only, 30–120 s |
164 | 0 |
ls-refs only, > 120 s |
13 | 0 |
The same share of clients pause on both paths. Directly, they continue after 30–120 s and succeed. Routed, every idle stream ends at ~15 s, while routed streams that carry data live well past 15 s.
Cause: an idle-stream limit of ~15 s (no bytes in either direction) on the Cloudflare edge path through the production-gitlab-com-cells-http-router Worker. It's the same ~15 s limit found during the staging attempts; the P1 rule removes buffering but not idleness. It isn't configurable on our side; to be confirmed with Cloudflare.
Fix (P14, updated 2026-11-11): request-body framing with keepalives, gitlab-org/gitlab-shell!1663 (merged) and gitlab-org/gitlab!269512 (merged). gitlab-shell sends a keepalive frame after 5 idle seconds and Workhorse strips the frames, so a request is never silent for long enough to be reset. That's better than the per-command requests first proposed here: it also covers the idle-after-ready case and pushes, and works with every protocol version. Changing Cloudflare isn't an option; see P14.
Failure mode B: SSH logins failing on a ~10 s edge latency tail
- 224
GET /api/v4/internal/authorized_keysrequests over the edge hit gitlab-shell's 10 s timeout (context deadline exceeded). 73 of the 74 sampled sessions never authenticated, so the client sees a failed login. Another 68/allowedcalls ended when the client gave up (~66 s). - The timeouts are isolated single requests (79 of 82 alone, evenly over the 8 pods), so it's not a stalled shared connection.
- Ruled out:
- Cloud NAT
gitlab-gke: 0 dropped packets, 0 allocation failures. - HAProxy
api: queue 0, 0 connection errors, 5xx flat at 0.17%. - Workhorse: 4 of 68,586 canary edge requests took > 5 s, none ended 499.
- Rails: the matched requests answered in ~5 ms.
- Cloud NAT
- Cloudflare: every stall is at ATL, with no origin response (499 when gitlab-shell gives up). The
cells-http-routerWorker has 0 script errors. Other clients'GET /api/v4/*through ATL show the same tail: 0.04–0.12% take > 9.5 s or are abandoned, against 0.07% for gitlab-shell. So the tail already existed; routing puts SSH authentication behind it.
Fix (P15): do one of these:
- send a second
authorized_keysrequest after 2–3 s and take the first answer (it's an idempotent GET), or use a shorter timeout with a retry; - or keep
authorized_keysand/allowedin-cluster and route only pack traffic. That also addresses StORM #61.
Not a failure: cells routing: upload-pack: EOF
There were 230. The error comes 0–0.3 s after the POST, at the moment the client disconnects, and 35 of 40 sampled are protocol v0. These are clients that close once they have the advertisement (ls-remote style). The direct path doesn't log this as an error, but on the routed path it makes up more than half of the gitlab_sshd session errors. gitlab-shell should treat a client EOF after a complete response as success (P16).
Projected impact at full rollout
Canary carried ~4,050 routed sessions/min. The fleet runs ~1,600 SSH sessions/s, about 24× that. Without P14 and P15, that means about 45 failed fetches/min (mode A) and about 75 failed SSH logins/min (mode B), for clients that did nothing wrong.
Rollback trigger review
- #1 (closed) "any
PROTOCOL_ERROR" and #2 (closed) "anyInternal API unreachable" can't be met as written. The direct path logsInternal API unreachableall the time: canary baseline 0.013% of commands, and 1.27% in gprd-us-east1-d today from an unrelated fault (below). Proposed rate-based triggers are in P16. - #3 (closed) (
gitlab_sshderror ratio > 0.002 for 5 min): came within 0.0001. More than half of that ratio is the EOF accounting above. - #13 (closed) (repeatable
/allowed200 then cell 403/404): did not fire. This is the trigger that attempts 2 and 3 were rolled back on.
Unrelated finding (direct path)
Two pods in gprd-us-east1-d (gitlab-gitlab-shell-…58zct, …ds68r) fail 1.27% of that zone's SSH commands with dial tcp 10.69.134.87:8181: connect: connection timed out and kube-dns lookup timeouts for gitlab-webservice-internal-api, since at least 09:13 UTC (about 36k log lines in 2 h). Routing is off in that cluster. To be raised with the EOC separately.
Phase 1 attempt 3 outcome (2026-10-30)
| UTC | Event |
|---|---|
| 14:28 | cells_shell_jwt_key_claims enabled on gprd (P11). No impact, since routing was off. |
| 14:57 | Pre-flight: an isolated gprd-cny test pod (not in the Service) with routing on fails its routed ssh-upload-pack, reset after 15 s with PROTOCOL_ERROR. |
| 15:30–17:25 | Cause found: the P1 rule was deleted on 2026-10-27 by config-mgmt!15703. Restored through Terraform on provider v5 (config-mgmt!15834, config-mgmt!15835, config-mgmt!15837, config-mgmt!15838). Ruleset ad55efc9… is now at version 16. |
| 17:28 | The same isolated pod passes: routed ls-remote, clone and fetch, with ssh-upload-pack returning 200 in 0.43–0.48 s. |
| 17:32 | !5950 merged. Its deploy pipeline queued behind an auto-deploy and started the gprd-cny upgrade at 17:54. New pods from 17:57. |
| 17:58–17:59 | Routed 403s for SSH-certificate users. Rollback MR !5951 created. |
Routed sessions on the new pods, 17:57–17:59:33 UTC (gitlab-shell logs, completed sessions):
| Auth | OK | Failed | Notes |
|---|---|---|---|
| SSH keys (incl. deploy keys) | 3,352 | 2 (0.06%) | Both pack POSTs returned 200. One push ended with cells routing: receive-pack response: EOF. |
| Group-scoped SSH certificates | 4 | 3 (403) | Phaidra/… once, mxtechnologies/… twice. /allowed returned 200, then the cell returned 403 on ssh-upload-pack. |
- Transport: 3,238
ssh-upload-packand 155ssh-receive-packPOSTs, all 200. p50 0.43 s, p99 0.58 s, max 1.62 s. 0PROTOCOL_ERROR, 0cannot rewind body, 0Internal API unreachable. - Topology Service Classify (canary): ~120–145/s
ok, ~5/snot_found, 0fail. p50 18 ms, p99 50 ms. Server-side Classify apdex 1.000 at ~140 req/s. gitlab_sshdsession error ratio (canary): rose from ~0.0001 to 0.0015 at 18:05 UTC, against the 0.002 trigger.- Sessions that
/allowedrejected before routing (no access, not found): 98 of 201 non-routed sessions. That's normal user error, unrelated to routing. - Topology Service warm-up surge: each gprd deploy wave makes new Rails processes send
ListLeases(ClaimService#warmup!) andGetCell, up to ~55 req/s. It's unrelated to routing, but it overlapped this rollout.
Root cause of the certificate 403s. ee/lib/ee/gitlab/git_access_project.rb, allowed_access_namespace?: if a request has no allowed_namespace_path, access is allowed only when the root group does not enforce SSH certificates (return !enforced_ssh_certificates? if allowed_namespace_path.blank?). On the direct path, gitlab-shell passes the certificate's namespace to /allowed. On the routed path, the cell's Repositories::GitHttpController re-runs the access check for ssh-upload-pack, and the Shell JWT has no namespace claim. Every certificate user in a group with Enforce SSH certificates is therefore denied with namespace_forbidden. The failure is deterministic. Certificate users in groups that don't enforce certificates are unaffected (4 OK).
Prerequisites
These are blocking and must all be complete before the ~"change::scheduled" label is applied.
- P1 — Cloudflare Configuration Rule on the production
gitlab.comzone (zone ID5b1bc06af128a829167e3a1212d86c28) settingrequest_body_buffering: noneandresponse_body_buffering: none, scoped to*/ssh-upload-packand*/ssh-receive-pack. Use the staging rule Disable Gitlab-shell SSH buffering on zone7e9c74d7c62f2f734c1d1b048b0844beas the template. Present (created 2026-10-12 14:13 UTC, re-verified 2026-10-14): the rule Disable Gitlab-shell SSH buffering is enabled with the expression and parameters below. A second rule, Disable buffering for LFS objects (14:15 UTC), is also present; it is not required by this change (see Known gaps). Without the pack-path rule, every clone of any non-trivial repository fails at ~15s.⚠️ 2026-10-30: found deleted, now restored in Terraform. The provider v4gprdapply of config-mgmt!15703 (2026-10-27 03:05 UTC) replaced the whole ruleset and dropped both hand-added rules without showing them in the plan. The ruleset is now owned byenvironments/cloudflare-cfgrules-gprdon provider v5. The rules Disable Gitlab-shell SSH buffering (ref = disable_gitlab_shell_ssh_buffering) and Disable buffering for LFS objects were restored as in version 14 by config-mgmt!15838 (applied 17:25 UTC, ruleset version 16). Staging was moved the same way (cloudflare-cfgrules-gstg, ruleset version 29 unchanged). The drift risk below no longer applies.- Expression:
(http.host eq "gitlab.com" and (ends_with(http.request.uri.path, "/ssh-upload-pack") or ends_with(http.request.uri.path, "/ssh-receive-pack"))) - Action parameters:
{"request_body_buffering": "none", "response_body_buffering": "none"} ⚠️ Terraform drift risk is higher than on staging.cloudflare_ruleset.gitlab_com_config_ruleinenvironments/gprd/cloudflare-configrules.tfmanages the whole entrypoint ruleset, so any gprd Atlantis apply that touches it would delete a rule added by hand. Before adding the rule, agree on one of: codifying it (needs Cloudflare provider v5), addinglifecycle { ignore_changes = [rules] }to that resource, or a documented manual rule with a pre-flight check (below).
- P2 — Security sign-off on disabling WAF body inspection for those two paths on the production zone, referencing StORM #61 (closed) and prod-eng#13744.
- P3 —
cell-1-production-mtls-certis valid (checked 2026-10-05). It is present ingitlabongprd-us-east1-{b,c,d}and ingitlab-cnyongprd-gitlab-gke,CN=gitlab.com, OU=cells. Current certificates expire 2026-11-04 (zonal) and 2026-11-06 (gprd-gitlab-gke). They are issued and renewed automatically by cert-manager (Certificate/cell-1-production-mtls-cert,Ready=True,ClusterIssuer/vault-cells-legacy-production). Re-checknotAfterin pre-flight if the window falls after 2026-11-04. - P4 — Git LFS validated on
gstg. Standalone LFS push and pull,fsckOK, and single-command push with 1–2 GiB objects. Three failure modes were found and fixed (Status). TheLFS content differsresult wasn't reproduced. Evidence: gitlab-org/gitlab-shell#874 (comment). - P5 — Edge capacity for the added request volume. Gates Phase 2 onward; Phase 1 (cny, 8 pods, ~+150 req/s) may proceed and provides the real per-operation numbers.
- Load estimate (measured on gprd 2026-10-12):
- SSH sessions: ~1,600/s now, ~1,990/s 7-day peak.
- gitlab-shell → internal API: ~4,400 req/s today (2.7k POST, 1.4k GET, 0.3k 404), all in-cluster. With the Topology Service client on, all of it moves to the public edge, plus 1 HTTPS pack request per pull and 2 per push (~1.5k req/s).
- Total added: ~6k req/s, ~7.5k at SSH peak. That is +13% on the HAProxy
httpsfrontend (46.9k now, 62k 7-day peak), +16% onapi(28k), +15% onhttps_git(9.6k). - Per phase: Phase 1 ~+150 req/s, Phases 2–4 ~+2k req/s each (~+2.5k at peak).
- HAProxy (
haproxy-main, 60 nodes): confirm headroom for +13%. Today: fleet CPU saturation 44%, busiest single node 67% (a single-node 7-day peak touched 97%),httpsfrontend sessions at 25% of the limit. - HAProxy per-IP API rate limit: the chef role
gprd-base-haproxy-main-configsetsapi.rate_limit_http_rate_per_minute: 2000. Spread over the 130 Cloud NAT egress IPs, 4.4k req/s is about 2,000 req/min per IP, right at that limit. The attribute is not referenced in the currentgitlab-haproxycookbook templates, so it looks stale, but HAProxy owners must confirm no per-IP limit applies to the NAT ranges. - Cloudflare WAF and rate limiting: gitlab-shell egresses through Cloud NAT
gitlab-gke(34.74.90.64/28,34.74.226.0/24), which matchesmodule.common_ips.gitlab_ips, part ofrate_limit_bypass_ipsinenvironments/gprd/cloudflare-general.tf. Confirm this bypass covers every rate-limit and WAF phase these requests hit. - Cloudflare Worker (
production-gitlab-com-cells-http-routerongitlab.com/*): about 13% more invocations at peak. Confirm Worker limits, cost and zone request entitlement are fine. The Worker also adds latency to every internal-API call. - API fleet: the ~4.5k req/s moves from
webservice-internal-api(476 pods) to the publicwebservice-apifleet (1,245 pods, busiest pod at ~73% of its CPU request). Confirm headroom, including NGINX ingress. - Connections and Cloud NAT: gitlab-shell uses Go's default HTTP transport (2 idle connections per host), so expect more new TLS connections and NAT port use. In Phase 1, measure HTTPS requests per SSH operation, new connections per second, and Cloud NAT port usage and drops for the
gitlab-gkeNAT (min_ports_per_vm4096, max 32768), and extrapolate to 100 pods before Phase 2. - Sign-off from the ServiceHAProxy and ServiceCloudflare owners, as a comment on this issue, using the Phase 1 measurements.
- Load estimate (measured on gprd 2026-10-12):
- P6 —
gitlab-shellongprdincludes !1561, !1570 and !1605.- !1605 (replayable pack bodies; found in Phase 1) shipped in
v14.59.0(gitlab-shell!1610, gitlab-org/gitlab!261825 (merged), merged 2026-10-13 15:50 UTC).✅ - Verified on 2026-10-14 with
gitlab-sshd -versionon every cluster:gprd-cnyv14.59.0-20261014.061858(19-5-202610140606-1dfbab7e88e, since ~2026-10-13 21:25 UTC);gprd-us-east1-b/c/dv14.59.0-20261014.041722(19-5-202610140407-44b913bf4df, 100/100 each, since ~08:40 UTC). Topology Service routing is off on all of them, as expected after the Phase 1 rollback. - gitlab-shell!1561 shipped in
v14.57.4, and gprd runsv14.57.4-20260923.153251(gitlab-shell:19-5-202609231516-123ea848f7c)✅ . - gitlab-shell!1570 shipped in
v14.57.6(gitlab-shell!1572, gitlab-org/gitlab!257493 (merged)). As of 2026-10-05, every gprd cluster (us-east1-b/c/d,cny) runsv14.58.0-20261005.083629, andgit merge-base --is-ancestor 572faf78 v14.58.0is true✅ . - Note the exact build in a comment. Without gitlab-shell!1570, any push whose LFS pre-push hook takes longer than ~60s fails with
Not found.
- !1605 (replayable pack bodies; found in Phase 1) shipped in
- P7 — Revert MRs pre-created, one per phase, so rollback is a single merge: !5910 (cny, attempt 2), !5882 (b), !5883 (c), !5884 (d). Each targets its phase's branch until that phase merges, then GitLab retargets it to
master. The enable MRs are !5909 (cny), !5878, !5879, !5880. !5881 (revert for the first cny attempt) was closed as stale. - P8 — Confirm the change window does not fall in a Production Change Lock period.
- P9 — Cloudflare
proxy_read_timeout= 1800s on the production zone. config-mgmt!15474 was applied by Atlantis on 2026-09-23 21:58 UTC (commit18af620d). Re-verified on 2026-10-05: the API returns1800. The staging zone was set by config-mgmt!15473 and validated with LFS uploads of 539s and 637s. - P10 — Staging re-validated on the standard image (2026-09-24). With all gstg clusters on
v14.57.6and no custom pods, a single-commandgit pushuploaded a new 2 GiB LFS object (PUT→200after 8m09s), then completedok refs/heads/vg/test-lfs-rule(f3d401d..a4bbac6). Details are in a comment on this issue. - P11 — The cell authorizes routed requests as the original actor, from the original IP. Done 2026-10-30. gitlab-org/gitlab-shell!1619 (merged) sends
gl_key_id,gl_key_typeandcheck_ipin the Shell JWT (released in v14.60.0, on every gstg and gprd cluster). gitlab-org/gitlab!265396 (merged) (b840c238b57d) honours them behind the feature flagcells_shell_jwt_key_claims(rollout gitlab-org/gitlab#636655 (closed)), which is enabled on gstg and gprd. Attempt 3 confirmed it in production: 0.06% key-authenticated routed failures, down from ~8.8%. The cell logs the client IP inmeta.remote_ip; the top-levelremote_ipstill shows the NAT address by design. Found in Phase 1 attempt 2. Needs a gitlab-shell change (put the key identity and client IP in the Shell JWT) and a Rails change (Repositories::GitHttpControllerSSH-over-HTTP actions authorize theDeployKeyactor, and use the signed client IP forGitlab::IpAddressState). Both must be released and deployed to gprd. Tracking issue: TBD. - P12 — Staging regression tests for routed SSH (a–d pass, 2026-10-30 14:21 UTC, flag on; (e) failed in production, see attempt 3 and P13):
- (a) deploy key with no owner (Ghost): clones.
- (b) deploy key whose owner lost access: clones.
- (c) push with a write deploy key to a branch where only that key may push: accepted. A read-only deploy key push is rejected with "This deploy key does not have write access".
- (d) IP-allowlisted Premium group, non-admin user, from the allowed IP: clones. From an IP outside the allowlist: "project not found", the same as direct SSH.
- (e) SSH-certificate user, passes 2026-11-06 09:59 UTC (routing on, gitlab-shell v14.61.0, flag on; Premium group with a group SSH CA and Enforce SSH certificates on). Each case on its own SSH connection:
Case Result Evidence Certificate user: clone ✅ exit 0/allowed200 withnamespace_path; cellssh-upload-pack200 as the certificate userCertificate user: push ✅ new branch createdboth cell ssh-receive-packrequests 200,post_receive200,meta.remote_ip= client IPSame user, plain SSH key ✅ denied (exit 128)in-cluster /allowed401 before any routing, the same as the direct pathRead-only deploy key: clone ✅ exit 0deploy keys are exempt from certificate enforcement, as on the direct path - With enforcement off (baseline), all four cases succeed. 0
Rejected Gitlab-Shell-Api-Request claimslines. - Group SSH certificates on gitlab.com only work for Enterprise Users of the group (
Not an Enterprise User of the groupotherwise). The staging fixture user was linked with agitlab-rails runner; the fixtures are deleted.
- With enforcement off (baseline), all four cases succeed. 0
- Manual SSH tests must disable connection sharing (
-o ControlMaster=no -o ControlPath=none), otherwise the first key gets reused. The first staging run was invalid because of this. - Original wording:, with the results posted here before the next attempt: (a) clone with a deploy key that has no owner; (b) clone with a deploy key whose owner has lost access; (c) push with a deploy key allowed by a protected-branch rule; (d) clone into a Premium group with an IP allowlist, from an allowed IP; (e) an SSH-certificate user. Each must behave the same with routing on as with routing off.
- P13 — Routed SSH-certificate access in groups that enforce SSH certificates. Done 2026-11-06: gitlab-shell
0a180921"Carry certificate namespaces in Cells Shell JWTs" in v14.61.0 (gprd-cnyv14.61.0-20261106.061748), and gitlab-org/gitlab!267101 (merged) merged 2026-11-05 (6705929e4c2e), in gprd package19-6-202611060606-d59525708f2. It reusescells_shell_jwt_key_claims, so no new flag. P12 (e) passes. Original text: Found in Phase 1 attempt 3. gitlab-shell must add the certificate's namespace (thenamespace_pathit already sends to/allowed) to the Shell JWT, and Rails must pass it asallowed_namespace_pathin the SSH-over-HTTP access check inRepositories::GitHttpController, behind a feature flag like P11. Both must be released and deployed to gprd, then P12 (e) passes on staging. - P14 — Routed SSH pack requests survive idle clients (request-body framing). Found in attempt 4 (failure mode A); the cause is confirmed in the "Case A analysis" comment by @vyaklushin (2026-11-09).
- Fix: gitlab-org/gitlab-shell!1663 (merged) (gitlab-shell) and gitlab-org/gitlab!269512 (merged) (Rails and Workhorse), both Draft. gitlab-shell wraps the pack request body in frames (5-byte header, up to 64 KiB of data) and sends a keepalive frame after 5 idle seconds. Workhorse strips the frames and forwards the original git bytes to Gitaly. Neither the git client nor Gitaly changes. Framing is negotiated through
/allowed(ssh_over_http_body_framing: 'v1', headerGitlab-Shell-Body-Framing: v1), behind the feature flagcells_ssh_body_framing. Without the flag, requests use raw bodies as today. - It covers: pull idle after
ls-refs, pull idle after the server sendsready, and push. Protocol v0, v1 and v2. - Changing Cloudflare isn't an option (details in the thread under the attempt 4 summary comment): no documented or configurable Cloudflare limit is ~15 s; the reset comes from the Workers runtime (a probe Worker was also reset at ~15.1 s, while the same zone without a Worker route held 60 s); and Worker route patterns can't exclude
*/ssh-*-pack. A separate hostname with no Worker route is a possible long-term option, but not a fix for this rollout. - Rollout order (strict):
- gitlab-org/gitlab!269512 (merged) deployed to every cell. Old Workhorse can't decode framed bodies.
- gitlab-shell release with gitlab-org/gitlab-shell!1663 (merged), and
GITLAB_SHELL_VERSIONbumped. - Enable
cells_ssh_body_framingon staging. - Staging:
idle_fetch_test.sh gitlab-org/gitlab git@staging.gitlab.com "5 16 18 30 60"passes for every pause, plus an LFS push with a pre-push hook over 60 s. Use-o ControlMaster=no -o ControlPath=none. - Performance check under concurrency (per-request memory is about 64 KiB, so watch gitlab-shell and Workhorse RSS).
- Enable on gprd, then the next Phase 1 attempt.
- Canary gate:
PROTOCOL_ERRORon routed sessions below 0.005% over a 60-minute soak (attempt 4: 0.044%). gitlab-shell logs show framing mode on routed requests. - Rollback: disabling
cells_ssh_body_framingturns framing off without a deploy.
- Fix: gitlab-org/gitlab-shell!1663 (merged) (gitlab-shell) and gitlab-org/gitlab!269512 (merged) (Rails and Workhorse), both Draft. gitlab-shell wraps the pack request body in frames (5-byte header, up to 64 KiB of data) and sends a keepalive frame after 5 idle seconds. Workhorse strips the frames and forwards the original git bytes to Gitaly. Neither the git client nor Gitaly changes. Framing is negotiated through
- P15 — SSH authentication is not exposed to the edge latency tail. Found in attempt 4 (failure mode B). Not addressed by P14:
authorized_keysand/allowedare short requests with no idle body, and they fail on gitlab-shell's own 10 s timeout while waiting for a stalled edge response. No fix is in progress yet. Either a hedged or retriedauthorized_keysrequest, or the internal API kept in-cluster with only pack traffic routed. Canary gate: edge auth timeouts within 2× the canary's own baseline over a 60-minute soak. - P16 — SLI accounting and rollback triggers fit the routed path.
- gitlab-shell stops counting a client EOF after a complete response as a session error.
- Triggers #1 (closed) and #2 (closed) become rates, for example:
PROTOCOL_ERRORabove 0.01% of routed sessions over 10 min;Internal API unreachableon edge URLs above 3× the canary baseline over 10 min. To be agreed with the change reviewer.
Preparation
Note
The following checklists must be done in advance, before setting the label changescheduled
Change Reviewer checklist
- Check if the following applies:
- The scheduled day and time of execution of the change is appropriate.
- The change plan is technically accurate.
- The change plan includes estimated timing values based on previous testing.
- The change plan includes a viable rollback plan.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
- Check if the following applies:
- The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
- The change plan includes success measures for all steps/milestones during the execution.
- The change adequately minimizes risk within the environment/service.
- The performance implications of executing the change are well-understood and documented.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
- If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
- The change has a primary and secondary SRE with knowledge of the details available during the change window.
- The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
- The labels blocks deployments and/or blocks feature-flags are applied as necessary.
Change Technician checklist
- The Change Criticality has been set appropriately and requirements have been reviewed.
- The change plan is technically accurate.
- The rollback plan is technically accurate and detailed enough to be executed by anyone with access.
- This Change Issue is linked to the appropriate Issue and/or Epic
- Change has been tested in staging and results noted in a comment on this issue.
- A dry-run has been conducted and results noted in a comment on this issue.
- The change execution window respects the Production Change Lock periods.
- Once all boxes above are checked, mark the change request as scheduled:
/label ~"change::scheduled" - For C1 and C2 change issues, the change event is added to the GitLab Production calendar by the change-scheduler bot.
- For C1 and C2 change issues, Platform Leadership provides approval with the platform_leadership_approved label on the issue. Mention
@gitlab-org/saas-platforms/change-review-leadershipin this issue with a reference to review guidelines to get approval and provide visibility to all infrastructure managers. - For C1, C2, or blocks deployments change issues, confirm with Release managers that the change does not overlap or hinder any release process (In
#productionchannel, mention@release-managersand this issue and await their acknowledgment.) - For C1 change issues or C2 change issues happening during weekend, SREs on-call must be informed at least 2 weeks in advance. Check the incident.io GitLab.com Production EOC schedule to find who will be on-call at the scheduled day and time.
- All Prerequisites P1–P16 are complete.
Detailed steps for the change
Rollout shape
releases/gitlab/values/gprd.yaml.gotmpl is shared by every gprd-prefixed environment, so putting the config there would enable all 308 pods at once. To get a staged rollout, the block goes into the per-environment files instead:
| Phase | File | Environment | Pods | SSH traffic |
|---|---|---|---|---|
| 1 | releases/gitlab/values/gprd-cny.yaml.gotmpl |
gprd-cny |
8 | low but real — 6 authenticated sessions observed in 30 min |
| 2 | releases/gitlab/values/gprd-us-east1-b.yaml.gotmpl |
gprd-us-east1-b |
100 | ~1/3 of production |
| 3 | releases/gitlab/values/gprd-us-east1-c.yaml.gotmpl |
gprd-us-east1-c |
100 | ~1/3 of production |
| 4 | releases/gitlab/values/gprd-us-east1-d.yaml.gotmpl |
gprd-us-east1-d |
100 | remaining ~1/3 |
Phase 3 was split into c and d following review: no single step affects more than a third of traffic, and us-east1-d serves as a control group during Phase 3.
Each per-environment file is layered after gprd.yaml.gotmpl and deep-merged, so the new gitlab-shell.config.topologyService block keeps the existing config keys (loginGraceTime, proxyProtocol). The Topology Service address and mTLS secret come from the existing global.appConfig.cell.topologyServiceClient in gprd.yaml.gotmpl.
Regional gprd and gprd-gitlab-gke use regional_cluster_defaults with gitlab-shell: enabled: false, so they are unaffected.
gprd-cny is behind an internal load balancer (10.216.8.61, cloud.google.com/load-balancer-type: Internal) but unlike gstg-cny it does receive real authenticated SSH traffic, so Phase 1 is a genuine if low-volume canary.
Rollout timing per cluster: maxSurge 25%, maxUnavailable 0, minReadySeconds 30, terminationGracePeriodSeconds 30, PDB minAvailable 5%. Expect 10–15 minutes for a 100-pod cluster to fully cycle.
Pre-execution steps
Note
The following steps should be done right at the scheduled time of the change request. The preparation steps are listed below.
- Make sure all tasks in Change Technician checklist are done
- For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out and has provided approval with the eoc_approved label on the issue.
- Check for active ~severity::1 / ~severity::2 incidents before requesting approval. If one overlaps with this change's blast radius, reschedule. If it does not overlap, include a brief risk assessment in the
@sre-oncallapproval request; if the EOC is unavailable or does not approve, reschedule.
- Check for active ~severity::1 / ~severity::2 incidents before requesting approval. If one overlaps with this change's blast radius, reschedule. If it does not overlap, include a brief risk assessment in the
- For C1, C2, or blocks deployments change issues, Release managers have been informed prior to change being rolled out. (In
#productionchannel, mention@release-managersand this issue and await their acknowledgment.) - Confirm P1 is still active: verify the Cloudflare Configuration Rule is present and enabled on the production zone, and has not been reverted by a Terraform apply since it was created. This is the single most important pre-flight check. Since 2026-10-30 the rule is managed in config-mgmt
environments/cloudflare-cfgrules-gprd. Check that the live rulesetad55efc9…contains Disable Gitlab-shell SSH buffering. - Confirm P9 is still active:
GET /zones/5b1bc06af128a829167e3a1212d86c28/settings/proxy_read_timeoutreturns1800. - Confirm P6 on a pod in every target cluster:
gitlab-sshd -versionreports a release containing gitlab-shell!1570. - Confirm no Atlantis plan or apply is pending for
environments/gprdin config-mgmt that touchescloudflare_ruleset.gitlab_com_config_rule(P1 drift risk). - Confirm
/chatops run feature get cells_shell_jwt_key_claimsshows the flag enabled on gprd (P11), plus the P13 flag once that exists. - Confirm no gprd auto-deploy is running or queued. The enable MR's pipeline waits behind it, and the deploy's Topology Service warm-up surge (
ListLeases/GetCell) can be mistaken for routing load. - Record the Topology Service baseline: Classify latency
histogram_quantile(0.99, sum by (le) (rate(gitlab_shell_topology_request_duration_seconds_bucket{env="gstg"}[5m])))from gstg, the only environment with Classify traffic before Phase 1, plus Topology Service (Runway) request rate, p99 latency, error rate, CPU and instance count. - Record a baseline from the git service dashboard:
gitlab_sshdapdex and error ratio,workhorse_auth_apiapdex,haproxy_backend_sessions_total{backend=~"ssh|altssh"}rate, andgitlab-shellcontainer RSS. Paste into a comment. - Open a thread in
#productionand post the baseline plus the rollback trigger list.
Change steps - steps to take to execute the change
Estimated Time to Complete (mins) - 330
- Set label changein-progress
/label ~change::in-progress
Phase 1 — gprd-cny (8 pods). Estimated 45 min including soak.
- Mark !5966 (attempt 4) ready and merge it. Attempt 3 was !5950. Earlier: mark !5909 ready and merge it (re-adds the
topologyServiceblock toreleases/gitlab/values/gprd-cny.yaml.gotmpl; same change as !5877, which was merged and reverted in9e8ea73f5). - Wait for the deploy pipeline and confirm the rollout completed:
kubectl --context=gke_gitlab-production_us-east1_gprd-gitlab-gke -n gitlab-cny rollout status deploy/gitlab-cny-gitlab-shell. - Confirm the config landed in a pod:
kubectl ... exec <pod> -c gitlab-shell -- sed -n '/topology_service/,/cell_endpoint/p' /srv/gitlab-shell/config.ymlshowsenabled: true,address: "topology-grpc.runway.gitlab.net:443",cell_endpoint: {scheme: https, port: 443}. - Success criteria, 30 min soak: at least one
ssh-upload-packand onessh-receive-packtohttps://gitlab.com:443withstatus: 200followed bysession: handleShell: command executed successfully; zeroPROTOCOL_ERROR; zeroInternal API unreachable; zerocells routing: receive-pack response: Not found;gitlab_sshdapdex within baseline. - Perform an explicit manual validation from outside the fleet, against the canary. Canary gets only a small share of SSH traffic, so pin the session to a canary pod:
kubectl --context=gke_gitlab-production_us-east1_gprd-gitlab-gke -n gitlab-cny port-forward pod/<gitlab-cny-gitlab-shell pod> 12722:2222, thenexport GIT_SSH_COMMAND='ssh -o HostName=127.0.0.1 -p 12722 -o HostKeyAlias=gitlab.com -o ControlMaster=no -o ControlPath=none'. WithoutControlMaster=no, a shared connection reuses the first key and can bypass the pod.GIT_SSH_COMMANDis also honoured bygit-lfs-authenticate. Use a dedicated test project on gitlab.com, not a real one, and remove the LFS objects and branches afterwards. Confirm in the pod's logs that each test hit it (Cells: using SSH-over-HTTP …).- Full clone of a large repository over SSH. It must exceed 60s of pack streaming, to prove the buffering rule works at production scale.
- Small
git push. -
git lfs pullof at least one object of 100 MB or more, followed bygit lfs fsck --objects. - Single-command
git pushthat uploads a new LFS object of 1 GiB or more, so the pre-push hook takes longer than 60s. Expectok refs/heads/.... Workhorse should log twossh-receive-packrequests for the session, both200, minutes apart. A404on the second one means gitlab-shell!1570 isn't running.
- Delete the test branch afterwards.
- Check routed sessions by authentication type in gitlab-shell logs: SSH keys, deploy keys, and SSH certificates (
certificate_scope). A403/404onssh-upload-pack/ssh-receive-packafter/allowedreturned200is a rollback trigger (#13 (closed)). - P5 measurements (needed before Phase 2): from the cny pods, record HTTPS requests per SSH operation (
gitlab_shell_http_requests_totalrate ÷ session rate), new outbound connections per second, Cloud NATgitlab-gkeport usage and dropped packets, and any429or403returned to gitlab-shell. Post them in a comment and extrapolate to 100 pods. - If any rollback trigger fires, go to Rollback.
Phase 2 — gprd-us-east1-b (100 pods). Estimated 90 min including soak.
- Confirm P5 is signed off using the Phase 1 measurements. Do not start Phase 2 without it.
- Mark !5878 ready and merge it (
releases/gitlab/values/gprd-us-east1-b.yaml.gotmpl). -
kubectl --context=gke_gitlab-production_us-east1-b_gprd-us-east1-b -n gitlab rollout status deploy/gitlab-gitlab-shell— expect 10–15 min. - Confirm the traffic shift:
gitlab_workhorse_http_requests_total{type="internal-api"}drops for theus-east1-bregion while Cloudflare and HAProxyhttpsfrontend request rate rises correspondingly. - Success criteria, 60 min soak:
gitservice error ratio within monitoring threshold 0.9999;gitlab_sshdapdex >= 0.998; zeroPROTOCOL_ERRORand zeroreceive-pack response: Not foundacross the cluster;ssh-receive-packnon-200 rate in Workhorse at ~0; Topology Service Classify p99 within baseline and Topology Service (Runway) error rate and saturation unchanged;gitlab-shellcontainer RSS stable versus the other two zonal clusters, which act as a live control group. - Compare
us-east1-bagainstus-east1-cand-dside by side. Any divergence in SSH error rate or latency that is attributable to this change is a rollback trigger.
Phase 3 — gprd-us-east1-c (100 pods). Estimated 90 min including soak.
- Mark !5879 ready and merge it (
releases/gitlab/values/gprd-us-east1-c.yaml.gotmpl). -
kubectl --context=gke_gitlab-production_us-east1-c_gprd-us-east1-c -n gitlab rollout status deploy/gitlab-gitlab-shell— expect 10–15 min. - Success criteria, 60 min soak: same thresholds as Phase 2. Topology Service load is now about 2/3 of production SSH, so watch Classify p99 and Runway saturation closely.
us-east1-dis the remaining control group.
Phase 4 — gprd-us-east1-d (100 pods). Estimated 90 min including soak.
- Mark !5880 ready and merge it (
releases/gitlab/values/gprd-us-east1-d.yaml.gotmpl). -
kubectl --context=gke_gitlab-production_us-east1-d_gprd-us-east1-d -n gitlab rollout status deploy/gitlab-gitlab-shell— expect 10–15 min. - Success criteria, 60 min soak: same thresholds as Phase 2, now with no control group. Watch
haproxy_backend_sessions_total{backend=~"ssh|altssh"}for any drop indicating clients failing before reaching authentication. - Consolidate the per-cluster blocks into
releases/gitlab/values/gprd.yaml.gotmplin a follow-up MR after the change window, as a no-op cleanup. - Post final metrics to this issue and to
#production. - Set label changecomplete
/label ~change::complete
Rollback triggers
Roll back immediately, without further discussion, on any of:
- Any
PROTOCOL_ERRORon assh-upload-packorssh-receive-packrequest. - Any
Internal API unreachablelog line. gitlab_sshderror ratio breaching the monitoring threshold (0.998 for this SLI) for more than 5 minutes.gitlab_sshdapdex below 0.998, orworkhorse_auth_apiapdex degrading below 0.998, sustained 5 minutes.- A drop in
haproxy_backend_sessions_total{backend=~"ssh|altssh"}completion rate versus baseline. gitlab-shellcontainer RSS trending toward the 3Gi limit, or any OOMKill.- Any user report of clone, fetch, push or LFS failure over SSH that correlates with the window.
- The Cloudflare Configuration Rule being found absent or reverted at any point.
- Any
cells routing: receive-pack response: Not found(or404on*/ssh-receive-packor*/ssh-upload-packwith user agentGitLab-Shell). This is the signature of an expired or rejected Shell JWT, meaning gitlab-shell!1570 isn't effective. - A sustained rise in
524responses on*/gitlab-lfs/objects/*PUTs, indicatingproxy_read_timeouthas been reverted. - Topology Service Classify p99 (
gitlab_shell_topology_request_duration_seconds) above 500 ms for 5 minutes, or a risingstatus="fail"rate ongitlab_shell_topology_requests_total. (status="not_found"is the normal answer for projects not on a cell, from gitlab-shell!1605 onward; before that releaseNotFoundwas counted asfail, so usestatus="fail"only once !1605 is deployed.) - Topology Service (Runway) error rate rising or saturation approaching its limits (CPU, instance count at max) during a phase.
- Any repeatable class of routed sessions where
/allowedreturns200and the cell then rejects the pack request with403or404(for example deploy keys, IP-restricted groups, SSH certificates). Attempts 2 and 3 were both rolled back on this.
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 30 per phase, 45 if all four phases are live (revert MRs can be merged in one go)
Important
There is no feature flag for this. Rollback is a Helm deploy and takes as long as a rolling restart of the affected clusters. The pre-created revert MRs from P7 exist specifically so that this is a single merge and not an authoring exercise under pressure.
- Merge the pre-created revert MR for the most recently enabled phase: !5967 (cny, attempt 4; !5951 was attempt 3; !5910 was attempt 2), !5882 (b), !5883 (c), !5884 (d). Check its target branch is
masterfirst. If the failure is not clearly scoped to one cluster, merge every revert MR whose phase is live. - Confirm the deploy pipeline runs. If CI is blocked by the
gitlab-shellchange lock, setCHANGE_LOCK_OVERRIDE— a rollback must never be gated on the lock. - Watch
kubectl rollout status deploy/gitlab-gitlab-shellfor each affected cluster (10–15 min per 100-pod cluster). - Verify the config is gone from a pod:
topology_serviceabsent, and/api/v4/internal/allowedcalls back onhttp://gitlab-webservice-internal-api.gitlab.svc.cluster.local:8181. - Confirm
gitlab_sshdapdex and error ratio return to baseline. - If rollback does not restore service, escalate to the EOC and treat as an incident; the pre-change state is a plain Helm revert with no data or schema component, so there is no forward-only path to worry about.
- Set label changeaborted
/label ~change::aborted
Monitoring
Key metrics to observe
- Metric:
gitlab_sshdapdex —gitlab_shell_sshd_session_established_duration_seconds_bucket{type="git"}- Location: git service overview
- What changes to this metric should prompt a rollback: apdex below 0.998 sustained 5 minutes. Satisfied threshold is 1s and tolerated is 5s; the added edge latency of 100–250ms per operation consumes a meaningful share of that budget.
- Metric:
gitlab_sshderror ratio —gitlab_sli:shell_sshd_connections:errors_total{type="git"}overgitlab_sli:shell_sshd_connections:total{type="git"}- Location: git service overview
- What changes to this metric should prompt a rollback: breach of the 0.998 monitoring threshold for this SLI for more than 5 minutes.
- Metric:
workhorse_auth_apiapdex and request rate —gitlab_workhorse_http_request_duration_seconds_bucket{type="internal-api", route="^/api/"}- Location: git service overview
- What changes to this metric should prompt a rollback: apdex degradation below 0.998. A drop in request rate is expected and is the positive confirmation that traffic has shifted to the edge — do not treat it as an error, but do confirm the corresponding rise appears at the edge.
- Metric:
haproxy_backend_sessions_total{backend=~"ssh|altssh"}- Location: git service overview, HAProxy SSH load balancer SLI
- What changes to this metric should prompt a rollback: any sustained drop versus baseline, which indicates clients failing before or during authentication.
- Metric:
PROTOCOL_ERRORandInternal API unreachablelog lines- Location: Kibana
shellindex,json.type: git, filtered tojson.url: *ssh-upload-pack* OR *ssh-receive-pack* - What changes to this metric should prompt a rollback: any occurrence. Zero is the only acceptable value; this is the exact signature of the three staging reverts.
- Location: Kibana
- Metric: Cells pack-request status in Workhorse:
json.route_id: (ssh_upload_pack OR ssh_receive_pack) AND json.user_agent: "GitLab-Shell", grouped byjson.status- Location: Kibana
workhorseindex. Correlate with theshellindex forcells routing: receive-pack response: Not found. - What changes to this metric should prompt a rollback: any
404, which is the expired or rejected Shell JWT signature fixed by gitlab-shell!1570.ssh-receive-packnormally shows two requests per push: a short advertisement, then the real push.
- Location: Kibana
- Metric: Topology Service Classify latency —
gitlab_shell_topology_request_duration_seconds(histogram)- Location: Prometheus / git service overview (ad-hoc panel:
histogram_quantile(0.99, sum by (le, cluster) (rate(gitlab_shell_topology_request_duration_seconds_bucket{env="gprd"}[5m])))) - What changes to this metric should prompt a rollback: p99 above 500 ms for 5 minutes. Classify runs on every SSH Git operation before authentication, so its latency is added directly to
gitlab_sshdapdex.
- Location: Prometheus / git service overview (ad-hoc panel:
- Metric: Topology Service (Runway) saturation and errors
- Location: Runway service dashboard for the Topology Service (
topology-grpc.runway.gitlab.net): request rate, error rate, p99 latency, CPU, instance count - What changes to this metric should prompt a rollback: error rate rising, or CPU or instance count approaching their limits. Each phase adds about a third of gitlab.com SSH operations as Classify RPCs.
- Location: Runway service dashboard for the Topology Service (
- Metric:
gitlab_shell_topology_requests_total- Location: Kibana
shellindex / Prometheus - What changes to this metric should prompt a rollback: a rising
status="fail"rate indicates Topology Service Classify RPC failures (not_foundis expected and excluded). Note this counts only the Classify call and is not sufficient on its own as a success signal — a completedssh-upload-packwithstatus: 200followed bycommand executed successfullyis the real proof.
- Location: Kibana
- Metric:
gitlab-shellcontainer RSS —kube_container_rssfor thegitlab-shelldeployment- Location: git service overview, Kubernetes resources
- What changes to this metric should prompt a rollback: trend toward the 3Gi limit or any OOMKill. Holding pack streams open across the edge for minutes at a time is a new memory profile for this fleet and has never been exercised at production concurrency.
- Metric: Cloudflare and HAProxy
httpsfrontend request rate and error rate- Location: Cloudflare service dashboard
- What changes to this metric should prompt a rollback: 5xx rate increase, or edge saturation attributable to the added internal-API and pack traffic.
Related
- Production rollout plan: gitlab-org/gitlab-shell#874
- Staging validation: gitlab-org/gitlab-shell#864 (closed)
- Parent epic: gitlab-org&13532
- Staging enablement MR: k8s-workloads/gitlab-com!5836
- Prior reverted attempts: !5810, !5811, !5823, !5824, !5808
- Chart support: gitlab-org/charts/gitlab!5167 (merged)
- Pull-body fix (necessary but not sufficient): gitlab-org/gitlab-shell!1546 (merged)
- Receive-pack server-first handshake: gitlab-org/gitlab-shell!1561 (merged, v14.57.4)
- Per-request Shell JWT: gitlab-org/gitlab-shell!1570 (merged, v14.57.6)
- Rollout MRs: enable !5909 (cny, attempt 2), !5878, !5879, !5880; revert !5910, !5882, !5883, !5884. First cny attempt: !5877 (merged, reverted in
9e8ea73f5), !5881 (closed). - Cloudflare
proxy_read_timeout1800s: gstg config-mgmt!15473, gprd config-mgmt!15474 (both applied). The original 400s came from infrastructure#8475. - Staging LFS and push evidence, 2026-09-23: gitlab-org/gitlab-shell#874 (comment)
- Security risk: StORM #61, prod-eng#13744
- Attempt 4: enable !5966, revert !5967 (deploy held for production#23436).
#productionthread: https://gitlab.slack.com/archives/C101F3796/p1793962830494849 - Attempt 3: enable !5950, revert !5951. Authorization fix: gitlab-org/gitlab-shell!1619 (merged), gitlab-org/gitlab!265396 (merged), flag rollout gitlab-org/gitlab#636655 (closed).
- Cloudflare Configuration Rules on provider v5: config-mgmt!15834 (bootstrap), config-mgmt!15835 (gstg move), config-mgmt!15837 (gprd move), config-mgmt!15838 (restore the buffering rules). Deleted by: config-mgmt!15703.
- Feature change lock: feature-change-locks#106 (INC-11437)