Enable Topology Service client for gitlab-shell on gprd (Cells SSH routing)

Production Change

Change Summary

Enable the Topology Service gRPC client (Cells SSH routing) for gitlab-shell on gprd, so that GitLab Shell resolves the owning cell for each SSH Git operation through the Topology Service and reaches that cell's internal API over https://gitlab.com:443 instead of the in-cluster webservice-internal-api service.

This is the production counterpart of the staging enablement in k8s-workloads/gitlab-com!5836 and completes the rollout planned in gitlab-org/gitlab-shell#874, validated on staging in gitlab-org/gitlab-shell#864 (closed).

The configuration change itself is four lines per cluster:

gitlab:
  gitlab-shell:
    config:
      topologyService:
        enabled: true
        cellEndpoint:
          scheme: https
          port: 443

Warning

This change moves the SSH authentication critical path off in-cluster networking and onto the public Cloudflare/HAProxy edge for 100% of gitlab.com SSH Git traffic. It was reverted three times on staging before the root cause was found. Read Risk assessment before reviewing.

Status (updated 2026-11-06 12:30 UTC)

Important

Phase 1 attempt 4 (gprd-cny, !5966) went live at 11:13 UTC and was soaked for ~70 min. Revert !5967 merged at 12:20 UTC, but its deploy is on hold at the EOC's request (a gprd-cny Helm upgrade starts the canary migrations pod, see INC-16026 and production#23436). Routing stays on in canary until the next k8s-workloads pipeline from master at or after 667f73546.

  • Fixed and confirmed in production: SSH certificates (790 routed certificate sessions, 0 cell 403s), deploy keys and IP restrictions, transport (every pack POST 200, 0 cannot rewind body), Topology Service (0 Classify failures).
  • Two edge-related failure modes block Phase 2 (details in Phase 1 attempt 4 outcome):
    1. Idle protocol-v2 streams reset at ~15 s (PROTOCOL_ERROR): 0.044% of routed sessions. The cause is clients pausing after ls-refs while gitlab-shell holds the HTTP request open. New prerequisite P14.
    2. SSH logins failing on a ~10 s edge latency tail for authorized_keys: 0.053% of connections, ~14× the canary baseline. New prerequisite P15.
  • Rollback triggers #1 (closed) and #2 (closed) ("any occurrence") can't be met as written, because the direct path logs Internal API unreachable too. Rate-based replacements are proposed in P16.

Note

Before attempt 4 (2026-11-06, morning). Phase 1 attempt 4 was ready to schedule: enable !5966 (draft), revert !5967.

  • P13 done. gitlab-shell v14.61.0 carries the SSH certificate namespace in the Shell JWT, and gitlab-org/gitlab!267101 (merged) (6705929e4c2e) honours it under the existing cells_shell_jwt_key_claims flag (on in gstg and gprd). Both are deployed to every gstg and gprd cluster.
  • P12 (e) passes on staging (2026-11-06 09:59 UTC, routing on, group with Enforce SSH certificates): certificate clone and push succeed through the cell, a plain key is denied by /allowed exactly as on the direct path, and a deploy key still clones. See P12.
  • P1 re-verified 2026-11-06: ruleset ad55efc9… at version 16, Disable Gitlab-shell SSH buffering enabled.
  • Remaining before change::scheduled: EOC and release-manager acknowledgement, and a change window outside a gprd auto-deploy.

Note

Attempt 3 (2026-10-30). Phase 1 attempt 3 (gprd-cny, !5950, live from 17:57 UTC on 2026-10-30) is being rolled back via !5951.

  • Transport and authorization for SSH keys are fixed. All 3,393 routed pack POSTs returned 200 (p50 0.43 s, p99 0.58 s), with no PROTOCOL_ERROR or cannot rewind body. Key-authenticated routed sessions: 3,352 OK and 2 failed (0.06%). The deploy-key and client-IP fix (P11) works. Attempt 2 failed ~8.8% here.
  • SSH-certificate users fail. In groups that enforce SSH certificates, the cell answers 403 on ssh-upload-pack after /allowed returned 200: 3 of 7 certificate sessions, across 2 groups. The cell's access check gets no certificate namespace, because the Shell JWT doesn't carry it. That's P12 case (e), which wasn't tested on staging. A new prerequisite, P13, covers the fix. Details: Phase 1 attempt 3 outcome.
  • P1 had silently regressed. The Cloudflare buffering rule on gitlab.com was deleted on 2026-10-27 by a provider v4 Terraform apply (config-mgmt!15703). It's now restored, and managed in Terraform on provider v5 (config-mgmt!15834, config-mgmt!15835, config-mgmt!15837, config-mgmt!15838). See P1.

Note

Earlier attempts. Phase 1 (gprd-cny) was executed on 2026-10-12 14:32–15:20 UTC and rolled back. Under production load (~50 SSH ops/s) about 2.3% of routed pulls failed with Internal API unreachable: net/http: cannot rewind body after connection loss (rollback trigger #2 (closed)). Root cause and fix: gitlab-shell!1605, validated on gprd-cny with a custom image at 0.00% over ~4,450 routed operations. Details in Phase 1 outcome.

Blocked again. Phase 1 attempt 2 (!5909, 2026-10-14 11:15 UTC) fixed the transport problem (0 cannot rewind across 5,481 routed operations), but ~8.8% of routed pulls failed with 404/403 from the cell: command success was 89.7% vs 97.3% without routing. Rollback !5910 is with the EOC. Root cause: the cell re-authorizes the routed request with only the user from the Shell JWT and the request's own IP, so deploy keys act as their owner (or the Ghost user) and group IP restrictions are checked against the Cloud NAT IP. Needs a Rails fix plus a gitlab-shell change (P11) and new staging tests (P12). Details: Phase 1 attempt 2 outcome.

Full staging evidence: gitlab-org/gitlab-shell#874 (comment).

# Failure mode Fix Staging gprd
1 Idle pack-body streams reset by the Cloudflare Workers runtime at ~15s Cloudflare Configuration Rule disabling body buffering on */ssh-upload-pack and */ssh-receive-pack (P1) ✅ applied, now in Terraform (cloudflare-cfgrules-gstg) ✅ restored 2026-10-30 17:25 UTC in Terraform (cloudflare-cfgrules-gprd). It had been deleted 2026-10-27 by config-mgmt!15703
2 git push stream left idle while Git's LFS pre-push hook runs, then reset gitlab-org/gitlab-shell!1561 (merged): advertise refs with one request, open the push request only once the client starts sending. Shipped in v14.57.4 ✅ ✅ deployed
3 Git LFS PUT gets HTTP 524 after exactly 400s, and every retry restarts from byte 0 Cloudflare proxy_read_timeout 400s → 1800s: gstg config-mgmt!15473, gprd config-mgmt!15474 (P9) ✅ ✅ applied 2026-09-23 21:58 UTC
4 remote: ERROR: cells routing: receive-pack response: Not found when Git's pre-push hook runs longer than ~60s gitlab-org/gitlab-shell!1570 (merged): sign the 1-minute Shell JWT for each request instead of once per session (P6) ✅ released in v14.57.6; re-validated on the standard image 2026-09-24 ✅ deployed (gprd runs v14.58.0-20261005.083629, which includes gitlab-shell!1570)
5 Cell authorizes routed deploy keys as their owner or the Ghost user, and checks group IP restrictions against the Cloud NAT IP (404/403) gitlab-org/gitlab-shell!1619 (merged) (JWT claims gl_key_id, gl_key_type, check_ip; v14.60.0) plus gitlab-org/gitlab!265396 (merged) behind cells_shell_jwt_key_claims (P11) ✅ flag on; P12 a–d pass ✅ flag on 2026-10-30; attempt 3: key-authenticated routed failures 0.06%
6 Cell returns 403 for routed SSH-certificate users in groups that enforce SSH certificates (no certificate namespace in the Shell JWT) gitlab-shell 0a180921 (v14.61.0) plus gitlab-org/gitlab!267101 (merged), same flag (P13) ✅ P12 (e) passes 2026-11-06 ✅ confirmed in attempt 4: 790 routed certificate sessions, 0 cell 403s
7 Routed protocol-v2 upload-pack stream reset after ~15 s of idleness (stream error: … PROTOCOL_ERROR; received from peer) when the client pauses after ls-refs Request-body framing with 5 s keepalives: gitlab-org/gitlab-shell!1663 (merged) and gitlab-org/gitlab!269512 (merged), behind cells_ssh_body_framing (P14); a Cloudflare-side change isn't possible not reproduced (staging traffic too low) ❌ found in attempt 4: 0.044% of routed sessions
8 SSH login fails when /api/v4/internal/authorized_keys over the edge hits gitlab-shell's 10 s timeout (Cloudflare ATL latency tail, no origin response) Hedged retry for authorized_keys, or keep the internal API in-cluster (P15) not reproduced ❌ found in attempt 4: 0.053% of connections

Rollout MRs (P7): all eight are pre-created as drafts, one enable and one revert MR per phase.

Phase Environment Enable Revert
1 gprd-cny !5950 (attempt 3, merged 2026-10-30 17:32 UTC); attempt 2 !5909 and attempt 1 !5877 were merged and reverted. Attempt 4: !5966 (merged 2026-11-06 11:08 UTC, live from 11:13). Attempt 4: !5967 (merged 12:20 UTC; deploy held, see Status); !5951 merged for attempt 3; !5910 merged for attempt 2; !5881 closed as stale
2 gprd-us-east1-b !5878 !5882
3 gprd-us-east1-c !5879 !5883
4 gprd-us-east1-d !5880 !5884

Change Details

  1. Services Impacted - ~"Service::GitLab Shell" ServiceGit ServiceCloudflare ServiceHAProxy ServiceTopology Service
  2. Change Technician - @vglafirov
  3. Change Reviewer - {+TBD: spin the Reviewer Roulette scoped to production; must not be the current EOC+}
  4. Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM) - {+TBD: to be scheduled once all Prerequisites are complete. The original 2026-09-17 09:00 slot was dropped.+}
  5. Time tracking - 6.5h (5.5h execution across four phases with soak windows, 1h reserved for rollback)
  6. Downtime Component - None expected. SSH Git availability is at risk throughout, but no planned downtime and no maintenance window required.

Risk assessment

What actually changes at runtime

Today on gprd, every SSH Git operation authenticates in-cluster:

msg: "Finished HTTP request"
url: http://gitlab-webservice-internal-api.gitlab.svc.cluster.local:8181/api/v4/internal/allowed
duration_s: 0.084   status: 200

With the Topology Service client enabled, the same call leaves the cluster. Measured on gstg after !5836:

msg: "Finished HTTP request"
url: https://staging.gitlab.com:443/api/v4/internal/allowed
duration_s: 0.17 - 0.34   status: 200

So this change does two things, not one:

  1. The Git pack transfer (ssh-upload-pack / ssh-receive-pack) is routed to the cell endpoint returned by the Topology Service.
  2. /api/v4/internal/allowed and /api/v4/internal/authorized_keys — the authentication critical path for every SSH connection — also traverse the public edge.

Blast radius

Value
gitlab-shell pods, gprd-us-east1-{b,c,d} 100 each (KEDA HPA min 100 / max 150)
gitlab-shell pods, gprd-cny 8
Git SSH ops observed, 3 pods over 66s 829 git-upload-pack, 25 git-receive-pack
Fleet-wide SSH rate (measured 2026-10-12, haproxy_backend_sessions_total{env="gprd",backend="ssh"}) ~1,600 sessions/s now, ~1,990/s 7-day peak
gitlab-shell → internal API today (gitlab_shell_http_requests_total, in-cluster) ~4,400 req/s (~2.75 per SSH session)
Added edge load when fully rolled out ~6k req/s, ~7.5k at SSH peak (+13% on the HAProxy https frontend). See P5.
Staging validation volume for comparison 134 ops over 33 minutes (~0.07/s)

Production carries roughly four orders of magnitude more SSH traffic than the window in which this was validated on staging. That gap is the single largest source of residual risk and it cannot be closed by more staging testing.

SLIs directly threatened

From metrics-catalog/services/git.jsonnet:

  • gitlab_sshd — apdex on gitlab_shell_sshd_session_established_duration_seconds_bucket, satisfied 1s / tolerated 5s. The added edge latency lands squarely inside this budget.
  • workhorse_auth_api — apdex satisfied threshold 1s, with the in-repo comment "1s is too slow for an auth request". Note this SLI is emitted by internal-api and will see traffic shift away from it, which is itself a signal: a sharp drop in gitlab_workhorse_http_requests_total{type="internal-api"} is the confirmation the change took effect.
  • gitlab_shell — haproxy_backend_sessions_total{backend=~"ssh|altssh"}.
  • git service error ratio, monitoring threshold 0.9999.

Why this failed three times on staging

Attempts !5810, !5823 and the gstg-cny variant were all reverted. Every Git pack transfer failed, clustered at 15.04–15.07s:

http2: Transport: cannot retry err [stream error: stream ID 1;
PROTOCOL_ERROR; received from peer] after Request.Body was written

Three hypotheses were raised and are now retracted: Cloudflare terminating long-lived streams, a ~60s intermediary idle client-body timeout, and the pull-body fix in gitlab-shell!1546 (shipped in 14.57.3). Pods were verified running v14.57.3-20260907.103559 and still failed at 15s.

Isolation testing identified the real cause:

Target Result
internal-gateway.gstg:11443 held idle request body 60s
kas.staging.gitlab.com — same CF zone, no Worker route survived 60s
Any Worker on the production runtime, including a 25-line probe that never reads the body reset at ~15.1s

It is a Cloudflare Workers runtime reset of the idle inbound HTTP/2 request body at ~15s. Not http-router code, not HAProxy, not Workhorse. The mitigation is to disable request and response body buffering on the pack paths, which is Prerequisite P1 below and is the reason this CR cannot proceed without a Cloudflare change on the production zone.

git push with Git LFS: failure modes found on 2026-09-23

Pulls and small pushes were fine, but a normal single-command git push that uploads LFS objects still failed. Git opens git-receive-pack and receives the ref advertisement before running the LFS pre-push hook. The hook then uploads the objects over HTTPS, which can take many minutes, and only after that does Git send its commands and pack.

  1. Idle push stream. Before gitlab-shell v14.57.4, the Cells push path held one streaming POST /ssh-receive-pack open throughout the hook. Cloudflare cancelled it after ~15s (Worker outcome canceled at 14,995 ms, Ray a3c6b5edfbc3b574, HAProxy term_state=CD--). The pack then went into a dead stream: PROTOCOL_ERROR, unexpected disconnect. Fix: gitlab-org/gitlab-shell!1561 (merged) (v14.57.4). It fetches the advertisement with a short 0000 request, then opens the real push request only once the client starts sending.
  2. Expired Shell JWT. With gitlab-shell!1561, the second request can start minutes after the first, but it reused the Shell JWT (1-minute TTL) signed at session start. Rails rejects the expired token and answers 404 by design: remote: ERROR: cells routing: receive-pack response: Not found. On staging, the second ssh-receive-pack returned 404 in 174 ms, 10m44s after the advertisement. Fix: gitlab-org/gitlab-shell!1570 (merged). It signs the JWT for each request, for push and pull. Pull wasn't affected in practice, because its single request starts immediately and the JWT is only checked when a request starts.
  3. Cloudflare 400s read timeout on LFS uploads (not Cells-specific). A LFS object upload is a single PUT that gets no response until the whole body has arrived, so proxy_read_timeout (400s on both zones since 2020) covered the whole upload. The result was 524 at 400,220 ms (Ray a3fbfa2bc8280cb6), with only 268,953,929 bytes received. The basic adapter can't resume uploads, so every retry restarted from byte 0. Fix: raise proxy_read_timeout to 1800s on both zones (config-mgmt!15473 / config-mgmt!15474, now applied).

The object transfer itself never touches gitlab-shell. Only git-lfs-authenticate is routed through the Topology Service; objects go directly over HTTPS to Workhorse.

Staging evidence

Observed on gstg on 2026-09-14 after !5836, window 09:46:40–10:19:25 UTC:

Metric gstg zonal b/c/d gstg-cny
Commands 108 pull / 26 push 5 pull / 2 push
command executed successfully 133 / 134 7 / 7
Pack requests non-200 0 / 134 0 / 7
PROTOCOL_ERROR 0 0
Internal API unreachable 0 0
Session p50 / p95 / max 1.68s / 9.85s / 431.23s 1.77s / 2.98s / 2.98s

The decisive data point is a complete gitlab-org/gitlab.git clone:

10:09:49.942  .../gitlab-org/gitlab.git/ssh-upload-pack   200   0.38s   <- headers
10:17:00.418  session: handleShell: command executed successfully
10:17:00.418  connection: handleRequests: done   duration_s: 431.23

430.5 seconds of continuous pack streaming, completed, zero PROTOCOL_ERROR — 28x past the wall that killed the three prior attempts. One earlier 210s session ended in a clean client-side EOF with no upstream error; the same clone from the same client succeeded nine minutes later, so it is treated as an operator-aborted attempt rather than an infrastructure limit.

Git LFS evidence (2026-09-22 → 2026-09-23, gitlab-com/gl-infra/cmbr on gstg)

Test Configuration Result
Standalone git lfs push, 1 GiB before the timeout raise ✅ Workhorse PUT 200, Rails upload_finalize 200
git lfs fetch of 8 objects, including 524 MB and 105 MB — ✅ all 200, git lfs fsck --objects OK
Single-command push, 1 GiB 400s timeout, v14.57.4 ❌ LFS PUT 524 at 400s, retry succeeded in 269s. Then the push 404 (expired JWT)
Single-command push, 2 GiB 1800s timeout, v14.57.4 ✅ LFS PUT 200 in 637s. ❌ push 404 at +10m44s (expired JWT)
Single-command push, 2 GiB 1800s timeout, custom image with gitlab-shell!1570 on all gstg clusters (b, c, d, cny), originals scaled to 0 ✅ LFS PUT 200 in 539s. Second ssh-receive-pack 200 at +9m07s. 1a063cc..f3d401d HEAD -> vg/test-lfs-rule

The custom image was gitlab-shell:dev-vg-cells-jwt-3d03b493 (sha256:2b794e3c…): the current staging image with only the binaries rebuilt from gitlab-shell!1570. Staging has since been restored to its normal deployments.

Known gaps carried into production

  • Git LFS: validated on staging with the gitlab-shell!1570 fix, pending re-validation on the standard image. See Git LFS evidence and P4/P10. The earlier F4 … LFS content differs result from the 2026-09-13 matrix wasn't reproduced: every downloaded object passed git lfs fsck --objects.
  • Pure-SSH LFS (git-lfs-transfer) is not enabled (lfs.pure_ssh_protocol: false). Git LFS 3.x tries it first, gets Unknown command, and falls back to HTTPS, adding ~1.5s per LFS operation. This isn't a regression, but users will see the error text in verbose output.
  • LFS uploads are still limited to 1800s per object. At 1800s, a 5 GiB object (the zone's max_upload) needs more than ~2.9 MB/s. Slower uploads still get a 524 and restart from zero. This applies to gitlab.com as a whole and isn't specific to Cells.
  • Disabling body buffering disables Cloudflare WAF body inspection on */ssh-upload-pack and */ssh-receive-pack. Requires security sign-off for the production zone.
  • Routing the internal API over the public edge is a registered risk: StORM #61 and prod-eng#13744. Long term the Topology Service needs to advertise an internally-routable endpoint; its address doubles as the mTLS certificate CN, so it cannot simply be changed to an internal host today.
  • No feature flag. The only control is Helm values per cluster, so rollback is a deploy, not a toggle. See Rollback.
  • Terraform drift: resolved 2026-10-30. This risk came true: the gprd provider v4 apply of config-mgmt!15703 deleted both hand-added rules from gitlab.com on 2026-10-27 at 03:05 UTC (ruleset ad55efc9… version 14 → 15), and its plan didn't show it. Both zones' Configuration Rules now live in provider v5 environments, cloudflare-cfgrules-gstg and cloudflare-cfgrules-gprd, with the buffering rules codified (config-mgmt!15834, config-mgmt!15835, config-mgmt!15837, config-mgmt!15838). The rest of this bullet is kept for history. On staging, two rules added by hand live outside Terraform: Disable Gitlab-shell SSH buffering (pack paths) and a test rule disabling buffering on /gitlab-lfs/objects/ and /info/lfs/. gstg applies since 2026-09-14 haven't removed them, but every config-mgmt plan touching the zone's config ruleset must be checked. The LFS-path rule is not required for this change, because the 524 was caused by the read timeout, not by buffering. It is a leftover from testing and can be removed from staging.

Phase 1 outcome (2026-10-12)

UTC Event
14:32 !5877 merged; rollout complete ~14:50. All 8 pods routed via https://gitlab.com:443, v14.58.0.
14:49–14:53 11,102 SSH commands, 10,518 routed pulls, 272 routed pushes. /allowed and authorized_keys over the edge: 22k requests, all 200. Classify p50 well under baseline.
14:50–14:53 241 routed pulls failed (2.3%) with cannot rewind body after connection loss, in bursts of up to 7/s, each failing in ~40 ms. 7 PROTOCOL_ERROR, 11 upload-pack: EOF. Zonal clusters without routing: 0 in the same window. Trigger #2 (closed) fired.
15:20 Reverted via 9e8ea73f5 (direct revert; !5881 left open). Pods back on in-cluster routing at 16:04.

Root cause. Each gitlab-shell pod multiplexes its pack requests over a shared HTTP/2 connection to the edge. When Cloudflare closed that connection (idle close or GOAWAY) with requests in flight, Go could not retry them: the pack body streamed SSH stdin through an io.Pipe, which has no Request.GetBody and cannot be replayed. In-memory requests (/allowed, authorized_keys) retry transparently and were unaffected. Staging never showed it because at ~0.07 ops/s every request got a fresh connection.

Fix. gitlab-shell!1605: a replayable pack body (git.ReplayBody, 1 MiB limit, streaming preserved) so the transport resends on a new connection; plus Topology Service NotFound moved from a WARN and status="fail" to debug and status="not_found" (it produced 2,041 warnings in 3.5 min and would have tripped trigger #11 (closed) on healthy traffic).

Validation on gprd-cny (16:02–16:13 UTC). Custom image from 6eb5ea87 as 2 extra pods in the canary Service (~20% of canary SSH), routing on: 4,319 pulls + 129 pushes, 0 cannot rewind failures (baseline would be ~100), 0 NotFound warnings; PROTOCOL_ERROR/EOF at the same low rate as before (unrelated: clients idle >15s mid-negotiation). Test pods and image removed afterwards. Evidence: gitlab-shell!1605 (comment).

Also learned. Classify is issued on every SSH op; Topology Service and edge handled ~50 ops/s without visible latency impact. P5 measurements for canary are in the comment above.

Phase 1 attempt 2 outcome (2026-10-14)

Transport and classification: fixed and healthy.

Result
gitlab-shell v14.59.0 on all 8 canary pods (includes gitlab-shell!1605); rollout complete 11:15 UTC
cannot rewind body failures 0 across 5,481 routed operations (attempt 1: 2.29%)
Topology Service Classify ~107/s ok, ~10/s not_found (matches unknown SSH keys: in-cluster authorized_keys 404 at ~8.9/s), 0 fail, p50 18 ms / p99 34 ms
Routing 100% of routed pack requests to https://gitlab.com:443, which is correct with only cell-1 in production

Authorization on the cell: broken. Routed operations, gitlab-shell logs, 122 s on 8 pods:

Total OK 404 403 Other (EOF, disconnects)
Pulls 5,329 4,780 (89.7%) 383 (7.2%) 84 (1.6%) 82 (1.5%)
Pushes 148 143 (96.6%) 2 (1.4%) 0 3

Without routing (zonal b, same period): 97.3% of SSH commands succeeded, with no pack-level 403/404. In every failing session /api/v4/internal/allowed returned 200. Then the routed POST /<project>.git/ssh-upload-pack was rejected by the cell, and the session exited with status 1.

Root cause. On the direct path, gitlab-shell authorizes once (/allowed with key_id and the client's check_ip) and then talks to Gitaly. On the routed path, the cell's Repositories::GitHttpController re-authorizes using only the Shell JWT gl_id and the request's own IP:

  1. Deploy keys (403). /allowed returns gl_id: Gitlab::GlId.gl_id(actor.user). For a deploy key that's the key's owner, or the Ghost user when there is none (DeployKey#user). The cell authenticates as that user, so the deploy key's own permissions are never checked. Every 403 in a 47-request Rails sample was meta.user: ghost1, across 9 root namespaces.
  2. Client IP (404). The cell sees remote_ip = the Cloud NAT IP (34.74.226.x), not the client's IP (check_ip in /allowed). Group IP restrictions therefore deny the read. 82% of the 404s were in Premium/Ultimate groups (against 58% of successful routed pulls), and 8% were free (against 34%). The 3 free-plan 404s fit deploy keys whose owner has lost access.

Also affected, even when requests succeed: routed requests are attributed to the deploy key's owner; protected-branch deploy-key push rules don't apply; and audit IPs and IP restrictions use GitLab's NAT range. IP restrictions fail closed today, but fail open for any group allowlisting 34.74.226.0/24.

Why staging missed it: gstg Rails logged only 3 routed SSH requests in 24 h, all gitlab-qa with a personal key. The whole validation matrix (P4, P10) used personal keys, with no deploy keys and no IP-restricted groups.

Projected at full rollout: ~140 failed pulls/s (~8k/min), mostly paying customers' CI/CD and GitOps automation. The failure is deterministic, so retries don't help.

Phase 1 attempt 4 outcome (2026-11-06)

Summary. Attempt 4 confirms every earlier fix in production, including the P13 SSH-certificate fix that attempt 3 was rolled back for. Two failure modes are left. Both come from the Cloudflare edge path, both run at about 0.05% of routed sessions, and neither is visible in apdex. Neither breaks the SLO at canary scale, so canary was kept on to soak, but at full rollout they would add up to about 120 failed SSH operations per minute. They block Phase 2.

Timeline (UTC)

Time Event
09:59 P12 (e) passes on staging (routed SSH certificates in a group that enforces them).
11:00–11:08 #production thread with @sre-oncall. EOC (Nidhey Indurkar) approves !5966 and adds eoc_approved.
11:08 !5966 merged. Pipeline 6525192; gprd-cny Helm revision 13853 at 11:13:24.
11:13:53–11:15:32 All 8 canary pods replaced: gitlab-sshd v14.61.0-20261106.083153, topology_service in config.yml, TLS files present.
11:15 Classify from canary steady at ~145 req/s ok, ~5 req/s not_found.
11:18–11:28 Canary gitlab_sshd error ratio 0.0011 → 0.0019 (baseline 0.0001–0.0003, trigger #3 (closed) at 0.002). Apdex unchanged.
11:20–12:15 Investigation: shell, Workhorse, Gitaly, Rails and Sidekiq logs; Grafana; Cloudflare analytics. Results below.
12:16 Decision to revert, to stop the exposure while the fixes are built.
12:20 !5967 merged.
12:21 Revert pipeline 6525575 cancelled at the EOC's request (Shreya Shah): a gprd-cny Helm upgrade starts the canary migrations job, which set off INC-16026 earlier today. It waits for production#23436 (delete ci_partitions row 118).
12:22 Release-manager auto-deploy 6525472 upgrades gprd-cny (Helm revision 13854). It was cut at 8dd5204bc (attempt 4 enable), so routing stays on. The revert ships with the next pipeline cut from master at or after 667f73546.

Results, soak window 11:15–12:25 UTC (70 min)

Compared with the same canary before routing (09:15–11:13) and with gprd-us-east1-b and -c in the same window. Sources: esquery on pubsub-shell-inf-gprd*, Grafana mimir-gitlab-gprd and mimir-runway.

Metric Canary, routing on Canary baseline Zonal b + c
SSH commands 292,855 502,756 3,938,867
Routed pack sessions 276,237 upload-pack + 7,331 receive-pack n/a n/a
Non-zero exit status 2.54% 2.40% 2.36%
PROTOCOL_ERROR (failure mode A) 125 (0.044% of routed) 0 0
cells routing: … EOF (client hung up, not a failure) 230 (0.081% of routed) n/a n/a
authorized_keys 10 s timeouts (failure mode B) 224 (0.053% of 425,656 connections) 28 (0.004% of 738,442) 0
/allowed cancelled by the client 68 1 2
SSH-certificate routed sessions 790, 3 non-zero, 0 pack 403s n/a n/a
Pack POSTs all 200; p50 0.42 s, p99 0.56 s n/a n/a
Topology Service Classify ~145/s ok, ~5/s not_found, 0 fail, 1 retried timeout; p99 44–46 ms; server apdex 1.000, error ratio 0 none none
gitlab_sshd apdex 0.9996–0.9997 0.9994–0.9998 0.9996–0.9998
gitlab_sshd error ratio 0.0009–0.0019 0.0001–0.0003 0.0001–0.0003
workhorse_auth_api apdex (cny) 0.9947–0.9978 0.9979–0.9987 n/a

Overall, routing costs about 0.14 percentage points of SSH sessions, against the canary's own baseline.

Confirmed working

  • SSH certificates (P13): 790 routed certificate sessions with 0 cell 403s. Attempt 3 had 3 of 7 fail.
  • Deploy keys and client-IP restrictions (P11): no repeatable 403/404 class on routed pack requests.
  • Transport (P6, !1605): 0 cannot rewind body. Every pack POST returned 200, and routed fetches that keep moving data ran for 16.5 s to over 120 s.
  • Cloudflare buffering rule (P1): active; streams that carry data are not cut.
  • Topology Service: no client or server errors, and no measurable latency impact.

Failure mode A: idle protocol-v2 streams reset after ~15 s (PROTOCOL_ERROR)

68 sessions traced end to end, from gitlab-shell (canary) through Workhorse (gitlab.com, all via Cloudflare ATL) to Gitaly:

  1. The client authenticates and sends its first request. The routed POST …/ssh-upload-pack gets 200 in ~0.4 s.
  2. gitlab.com answers in full. In 39 of 67 sessions the request was only ls-refs (Gitaly logs packfile negotiation complete, command: ls-refs, wants: 0, 173–418 response bytes). Some had already received the whole pack (up to 9.4 MB).
  3. The client then sends nothing: no next command, flush or EOF. Protocol v2 lets the client wait before its next command, so both directions of the HTTP/2 stream are idle.
  4. After 14.6–16.0 s the stream is reset. gitlab-shell gets stream error: stream ID 1; PROTOCOL_ERROR; received from peer and exits 1. At the same instant Workhorse ends the request (duration_ms ≈ 15,000, status 200) and Gitaly cancels upload-pack (Canceled … signal: terminated: context canceled, time_ms ≈ 15,000).
  • The clients are ordinary git CLI users: protocol v2 in 57 of 68; macOS, Windows and Linux; git 2.30 to 2.56; 62 different repositories; spread evenly over all pods.
  • gitlab-shell keeps the stream open by design: readUploadPackRequest (internal/command/githttp/util.go) closes the HTTP request body only when it sees done or SSH stdin reaches EOF. ls-refs never sends done, and neither does a fetch the server ends early with ready.

Proof that the failure is the idle limit on the routed path. Gitaly SSHUploadPackWithSidechannel, 11:20–11:35 UTC, all gprd:

Session length Direct (gitlab-shell client) Routed (gitlab-workhorse via edge)
≥ 14.5 s, all kinds 1,012 of 1.19M (0.085%) 45 of 60k (0.075%)
ls-refs only, 14.5–16.5 s 27 18
ls-refs only, 16.5–30 s 102 3
ls-refs only, 30–120 s 164 0
ls-refs only, > 120 s 13 0

The same share of clients pause on both paths. Directly, they continue after 30–120 s and succeed. Routed, every idle stream ends at ~15 s, while routed streams that carry data live well past 15 s.

Cause: an idle-stream limit of ~15 s (no bytes in either direction) on the Cloudflare edge path through the production-gitlab-com-cells-http-router Worker. It's the same ~15 s limit found during the staging attempts; the P1 rule removes buffering but not idleness. It isn't configurable on our side; to be confirmed with Cloudflare.

Fix (P14, updated 2026-11-11): request-body framing with keepalives, gitlab-org/gitlab-shell!1663 (merged) and gitlab-org/gitlab!269512 (merged). gitlab-shell sends a keepalive frame after 5 idle seconds and Workhorse strips the frames, so a request is never silent for long enough to be reset. That's better than the per-command requests first proposed here: it also covers the idle-after-ready case and pushes, and works with every protocol version. Changing Cloudflare isn't an option; see P14.

Failure mode B: SSH logins failing on a ~10 s edge latency tail

  • 224 GET /api/v4/internal/authorized_keys requests over the edge hit gitlab-shell's 10 s timeout (context deadline exceeded). 73 of the 74 sampled sessions never authenticated, so the client sees a failed login. Another 68 /allowed calls ended when the client gave up (~66 s).
  • The timeouts are isolated single requests (79 of 82 alone, evenly over the 8 pods), so it's not a stalled shared connection.
  • Ruled out:
    • Cloud NAT gitlab-gke: 0 dropped packets, 0 allocation failures.
    • HAProxy api: queue 0, 0 connection errors, 5xx flat at 0.17%.
    • Workhorse: 4 of 68,586 canary edge requests took > 5 s, none ended 499.
    • Rails: the matched requests answered in ~5 ms.
  • Cloudflare: every stall is at ATL, with no origin response (499 when gitlab-shell gives up). The cells-http-router Worker has 0 script errors. Other clients' GET /api/v4/* through ATL show the same tail: 0.04–0.12% take > 9.5 s or are abandoned, against 0.07% for gitlab-shell. So the tail already existed; routing puts SSH authentication behind it.

Fix (P15): do one of these:

  • send a second authorized_keys request after 2–3 s and take the first answer (it's an idempotent GET), or use a shorter timeout with a retry;
  • or keep authorized_keys and /allowed in-cluster and route only pack traffic. That also addresses StORM #61.

Not a failure: cells routing: upload-pack: EOF

There were 230. The error comes 0–0.3 s after the POST, at the moment the client disconnects, and 35 of 40 sampled are protocol v0. These are clients that close once they have the advertisement (ls-remote style). The direct path doesn't log this as an error, but on the routed path it makes up more than half of the gitlab_sshd session errors. gitlab-shell should treat a client EOF after a complete response as success (P16).

Projected impact at full rollout

Canary carried ~4,050 routed sessions/min. The fleet runs ~1,600 SSH sessions/s, about 24× that. Without P14 and P15, that means about 45 failed fetches/min (mode A) and about 75 failed SSH logins/min (mode B), for clients that did nothing wrong.

Rollback trigger review

  • #1 (closed) "any PROTOCOL_ERROR" and #2 (closed) "any Internal API unreachable" can't be met as written. The direct path logs Internal API unreachable all the time: canary baseline 0.013% of commands, and 1.27% in gprd-us-east1-d today from an unrelated fault (below). Proposed rate-based triggers are in P16.
  • #3 (closed) (gitlab_sshd error ratio > 0.002 for 5 min): came within 0.0001. More than half of that ratio is the EOF accounting above.
  • #13 (closed) (repeatable /allowed 200 then cell 403/404): did not fire. This is the trigger that attempts 2 and 3 were rolled back on.

Unrelated finding (direct path)

Two pods in gprd-us-east1-d (gitlab-gitlab-shell-…58zct, …ds68r) fail 1.27% of that zone's SSH commands with dial tcp 10.69.134.87:8181: connect: connection timed out and kube-dns lookup timeouts for gitlab-webservice-internal-api, since at least 09:13 UTC (about 36k log lines in 2 h). Routing is off in that cluster. To be raised with the EOC separately.

Phase 1 attempt 3 outcome (2026-10-30)

UTC Event
14:28 cells_shell_jwt_key_claims enabled on gprd (P11). No impact, since routing was off.
14:57 Pre-flight: an isolated gprd-cny test pod (not in the Service) with routing on fails its routed ssh-upload-pack, reset after 15 s with PROTOCOL_ERROR.
15:30–17:25 Cause found: the P1 rule was deleted on 2026-10-27 by config-mgmt!15703. Restored through Terraform on provider v5 (config-mgmt!15834, config-mgmt!15835, config-mgmt!15837, config-mgmt!15838). Ruleset ad55efc9… is now at version 16.
17:28 The same isolated pod passes: routed ls-remote, clone and fetch, with ssh-upload-pack returning 200 in 0.43–0.48 s.
17:32 !5950 merged. Its deploy pipeline queued behind an auto-deploy and started the gprd-cny upgrade at 17:54. New pods from 17:57.
17:58–17:59 Routed 403s for SSH-certificate users. Rollback MR !5951 created.

Routed sessions on the new pods, 17:57–17:59:33 UTC (gitlab-shell logs, completed sessions):

Auth OK Failed Notes
SSH keys (incl. deploy keys) 3,352 2 (0.06%) Both pack POSTs returned 200. One push ended with cells routing: receive-pack response: EOF.
Group-scoped SSH certificates 4 3 (403) Phaidra/… once, mxtechnologies/… twice. /allowed returned 200, then the cell returned 403 on ssh-upload-pack.
  • Transport: 3,238 ssh-upload-pack and 155 ssh-receive-pack POSTs, all 200. p50 0.43 s, p99 0.58 s, max 1.62 s. 0 PROTOCOL_ERROR, 0 cannot rewind body, 0 Internal API unreachable.
  • Topology Service Classify (canary): ~120–145/s ok, ~5/s not_found, 0 fail. p50 18 ms, p99 50 ms. Server-side Classify apdex 1.000 at ~140 req/s.
  • gitlab_sshd session error ratio (canary): rose from ~0.0001 to 0.0015 at 18:05 UTC, against the 0.002 trigger.
  • Sessions that /allowed rejected before routing (no access, not found): 98 of 201 non-routed sessions. That's normal user error, unrelated to routing.
  • Topology Service warm-up surge: each gprd deploy wave makes new Rails processes send ListLeases (ClaimService#warmup!) and GetCell, up to ~55 req/s. It's unrelated to routing, but it overlapped this rollout.

Root cause of the certificate 403s. ee/lib/ee/gitlab/git_access_project.rb, allowed_access_namespace?: if a request has no allowed_namespace_path, access is allowed only when the root group does not enforce SSH certificates (return !enforced_ssh_certificates? if allowed_namespace_path.blank?). On the direct path, gitlab-shell passes the certificate's namespace to /allowed. On the routed path, the cell's Repositories::GitHttpController re-runs the access check for ssh-upload-pack, and the Shell JWT has no namespace claim. Every certificate user in a group with Enforce SSH certificates is therefore denied with namespace_forbidden. The failure is deterministic. Certificate users in groups that don't enforce certificates are unaffected (4 OK).

Prerequisites

These are blocking and must all be complete before the ~"change::scheduled" label is applied.

  • P1 — Cloudflare Configuration Rule on the production gitlab.com zone (zone ID 5b1bc06af128a829167e3a1212d86c28) setting request_body_buffering: none and response_body_buffering: none, scoped to */ssh-upload-pack and */ssh-receive-pack. Use the staging rule Disable Gitlab-shell SSH buffering on zone 7e9c74d7c62f2f734c1d1b048b0844be as the template. Present (created 2026-10-12 14:13 UTC, re-verified 2026-10-14): the rule Disable Gitlab-shell SSH buffering is enabled with the expression and parameters below. A second rule, Disable buffering for LFS objects (14:15 UTC), is also present; it is not required by this change (see Known gaps). Without the pack-path rule, every clone of any non-trivial repository fails at ~15s.
    • ⚠️ 2026-10-30: found deleted, now restored in Terraform. The provider v4 gprd apply of config-mgmt!15703 (2026-10-27 03:05 UTC) replaced the whole ruleset and dropped both hand-added rules without showing them in the plan. The ruleset is now owned by environments/cloudflare-cfgrules-gprd on provider v5. The rules Disable Gitlab-shell SSH buffering (ref = disable_gitlab_shell_ssh_buffering) and Disable buffering for LFS objects were restored as in version 14 by config-mgmt!15838 (applied 17:25 UTC, ruleset version 16). Staging was moved the same way (cloudflare-cfgrules-gstg, ruleset version 29 unchanged). The drift risk below no longer applies.
    • Expression: (http.host eq "gitlab.com" and (ends_with(http.request.uri.path, "/ssh-upload-pack") or ends_with(http.request.uri.path, "/ssh-receive-pack")))
    • Action parameters: {"request_body_buffering": "none", "response_body_buffering": "none"}
    • ⚠️ Terraform drift risk is higher than on staging. cloudflare_ruleset.gitlab_com_config_rule in environments/gprd/cloudflare-configrules.tf manages the whole entrypoint ruleset, so any gprd Atlantis apply that touches it would delete a rule added by hand. Before adding the rule, agree on one of: codifying it (needs Cloudflare provider v5), adding lifecycle { ignore_changes = [rules] } to that resource, or a documented manual rule with a pre-flight check (below).
  • P2 — Security sign-off on disabling WAF body inspection for those two paths on the production zone, referencing StORM #61 (closed) and prod-eng#13744.
  • P3 — cell-1-production-mtls-cert is valid (checked 2026-10-05). It is present in gitlab on gprd-us-east1-{b,c,d} and in gitlab-cny on gprd-gitlab-gke, CN=gitlab.com, OU=cells. Current certificates expire 2026-11-04 (zonal) and 2026-11-06 (gprd-gitlab-gke). They are issued and renewed automatically by cert-manager (Certificate/cell-1-production-mtls-cert, Ready=True, ClusterIssuer/vault-cells-legacy-production). Re-check notAfter in pre-flight if the window falls after 2026-11-04.
  • P4 — Git LFS validated on gstg. Standalone LFS push and pull, fsck OK, and single-command push with 1–2 GiB objects. Three failure modes were found and fixed (Status). The LFS content differs result wasn't reproduced. Evidence: gitlab-org/gitlab-shell#874 (comment).
  • P5 — Edge capacity for the added request volume. Gates Phase 2 onward; Phase 1 (cny, 8 pods, ~+150 req/s) may proceed and provides the real per-operation numbers.
    • Load estimate (measured on gprd 2026-10-12):
      • SSH sessions: ~1,600/s now, ~1,990/s 7-day peak.
      • gitlab-shell → internal API: ~4,400 req/s today (2.7k POST, 1.4k GET, 0.3k 404), all in-cluster. With the Topology Service client on, all of it moves to the public edge, plus 1 HTTPS pack request per pull and 2 per push (~1.5k req/s).
      • Total added: ~6k req/s, ~7.5k at SSH peak. That is +13% on the HAProxy https frontend (46.9k now, 62k 7-day peak), +16% on api (28k), +15% on https_git (9.6k).
      • Per phase: Phase 1 ~+150 req/s, Phases 2–4 ~+2k req/s each (~+2.5k at peak).
    • HAProxy (haproxy-main, 60 nodes): confirm headroom for +13%. Today: fleet CPU saturation 44%, busiest single node 67% (a single-node 7-day peak touched 97%), https frontend sessions at 25% of the limit.
    • HAProxy per-IP API rate limit: the chef role gprd-base-haproxy-main-config sets api.rate_limit_http_rate_per_minute: 2000. Spread over the 130 Cloud NAT egress IPs, 4.4k req/s is about 2,000 req/min per IP, right at that limit. The attribute is not referenced in the current gitlab-haproxy cookbook templates, so it looks stale, but HAProxy owners must confirm no per-IP limit applies to the NAT ranges.
    • Cloudflare WAF and rate limiting: gitlab-shell egresses through Cloud NAT gitlab-gke (34.74.90.64/28, 34.74.226.0/24), which matches module.common_ips.gitlab_ips, part of rate_limit_bypass_ips in environments/gprd/cloudflare-general.tf. Confirm this bypass covers every rate-limit and WAF phase these requests hit.
    • Cloudflare Worker (production-gitlab-com-cells-http-router on gitlab.com/*): about 13% more invocations at peak. Confirm Worker limits, cost and zone request entitlement are fine. The Worker also adds latency to every internal-API call.
    • API fleet: the ~4.5k req/s moves from webservice-internal-api (476 pods) to the public webservice-api fleet (1,245 pods, busiest pod at ~73% of its CPU request). Confirm headroom, including NGINX ingress.
    • Connections and Cloud NAT: gitlab-shell uses Go's default HTTP transport (2 idle connections per host), so expect more new TLS connections and NAT port use. In Phase 1, measure HTTPS requests per SSH operation, new connections per second, and Cloud NAT port usage and drops for the gitlab-gke NAT (min_ports_per_vm 4096, max 32768), and extrapolate to 100 pods before Phase 2.
    • Sign-off from the ServiceHAProxy and ServiceCloudflare owners, as a comment on this issue, using the Phase 1 measurements.
  • P6 — gitlab-shell on gprd includes !1561, !1570 and !1605.
    • !1605 (replayable pack bodies; found in Phase 1) shipped in v14.59.0 (gitlab-shell!1610, gitlab-org/gitlab!261825 (merged), merged 2026-10-13 15:50 UTC). ✅
    • Verified on 2026-10-14 with gitlab-sshd -version on every cluster: gprd-cny v14.59.0-20261014.061858 (19-5-202610140606-1dfbab7e88e, since ~2026-10-13 21:25 UTC); gprd-us-east1-b/c/d v14.59.0-20261014.041722 (19-5-202610140407-44b913bf4df, 100/100 each, since ~08:40 UTC). Topology Service routing is off on all of them, as expected after the Phase 1 rollback.
    • gitlab-shell!1561 shipped in v14.57.4, and gprd runs v14.57.4-20260923.153251 (gitlab-shell:19-5-202609231516-123ea848f7c) ✅.
    • gitlab-shell!1570 shipped in v14.57.6 (gitlab-shell!1572, gitlab-org/gitlab!257493 (merged)). As of 2026-10-05, every gprd cluster (us-east1-b/c/d, cny) runs v14.58.0-20261005.083629, and git merge-base --is-ancestor 572faf78 v14.58.0 is true ✅.
    • Note the exact build in a comment. Without gitlab-shell!1570, any push whose LFS pre-push hook takes longer than ~60s fails with Not found.
  • P7 — Revert MRs pre-created, one per phase, so rollback is a single merge: !5910 (cny, attempt 2), !5882 (b), !5883 (c), !5884 (d). Each targets its phase's branch until that phase merges, then GitLab retargets it to master. The enable MRs are !5909 (cny), !5878, !5879, !5880. !5881 (revert for the first cny attempt) was closed as stale.
  • P8 — Confirm the change window does not fall in a Production Change Lock period.
  • P9 — Cloudflare proxy_read_timeout = 1800s on the production zone. config-mgmt!15474 was applied by Atlantis on 2026-09-23 21:58 UTC (commit 18af620d). Re-verified on 2026-10-05: the API returns 1800. The staging zone was set by config-mgmt!15473 and validated with LFS uploads of 539s and 637s.
  • P10 — Staging re-validated on the standard image (2026-09-24). With all gstg clusters on v14.57.6 and no custom pods, a single-command git push uploaded a new 2 GiB LFS object (PUT → 200 after 8m09s), then completed ok refs/heads/vg/test-lfs-rule (f3d401d..a4bbac6). Details are in a comment on this issue.
  • P11 — The cell authorizes routed requests as the original actor, from the original IP. Done 2026-10-30. gitlab-org/gitlab-shell!1619 (merged) sends gl_key_id, gl_key_type and check_ip in the Shell JWT (released in v14.60.0, on every gstg and gprd cluster). gitlab-org/gitlab!265396 (merged) (b840c238b57d) honours them behind the feature flag cells_shell_jwt_key_claims (rollout gitlab-org/gitlab#636655 (closed)), which is enabled on gstg and gprd. Attempt 3 confirmed it in production: 0.06% key-authenticated routed failures, down from ~8.8%. The cell logs the client IP in meta.remote_ip; the top-level remote_ip still shows the NAT address by design. Found in Phase 1 attempt 2. Needs a gitlab-shell change (put the key identity and client IP in the Shell JWT) and a Rails change (Repositories::GitHttpController SSH-over-HTTP actions authorize the DeployKey actor, and use the signed client IP for Gitlab::IpAddressState). Both must be released and deployed to gprd. Tracking issue: TBD.
  • P12 — Staging regression tests for routed SSH (a–d pass, 2026-10-30 14:21 UTC, flag on; (e) failed in production, see attempt 3 and P13):
    • (a) deploy key with no owner (Ghost): clones.
    • (b) deploy key whose owner lost access: clones.
    • (c) push with a write deploy key to a branch where only that key may push: accepted. A read-only deploy key push is rejected with "This deploy key does not have write access".
    • (d) IP-allowlisted Premium group, non-admin user, from the allowed IP: clones. From an IP outside the allowlist: "project not found", the same as direct SSH.
    • (e) SSH-certificate user, passes 2026-11-06 09:59 UTC (routing on, gitlab-shell v14.61.0, flag on; Premium group with a group SSH CA and Enforce SSH certificates on). Each case on its own SSH connection:
      Case Result Evidence
      Certificate user: clone ✅ exit 0 /allowed 200 with namespace_path; cell ssh-upload-pack 200 as the certificate user
      Certificate user: push ✅ new branch created both cell ssh-receive-pack requests 200, post_receive 200, meta.remote_ip = client IP
      Same user, plain SSH key ✅ denied (exit 128) in-cluster /allowed 401 before any routing, the same as the direct path
      Read-only deploy key: clone ✅ exit 0 deploy keys are exempt from certificate enforcement, as on the direct path
      • With enforcement off (baseline), all four cases succeed. 0 Rejected Gitlab-Shell-Api-Request claims lines.
      • Group SSH certificates on gitlab.com only work for Enterprise Users of the group (Not an Enterprise User of the group otherwise). The staging fixture user was linked with a gitlab-rails runner; the fixtures are deleted.
    • Manual SSH tests must disable connection sharing (-o ControlMaster=no -o ControlPath=none), otherwise the first key gets reused. The first staging run was invalid because of this.
    • Original wording:, with the results posted here before the next attempt: (a) clone with a deploy key that has no owner; (b) clone with a deploy key whose owner has lost access; (c) push with a deploy key allowed by a protected-branch rule; (d) clone into a Premium group with an IP allowlist, from an allowed IP; (e) an SSH-certificate user. Each must behave the same with routing on as with routing off.
  • P13 — Routed SSH-certificate access in groups that enforce SSH certificates. Done 2026-11-06: gitlab-shell 0a180921 "Carry certificate namespaces in Cells Shell JWTs" in v14.61.0 (gprd-cny v14.61.0-20261106.061748), and gitlab-org/gitlab!267101 (merged) merged 2026-11-05 (6705929e4c2e), in gprd package 19-6-202611060606-d59525708f2. It reuses cells_shell_jwt_key_claims, so no new flag. P12 (e) passes. Original text: Found in Phase 1 attempt 3. gitlab-shell must add the certificate's namespace (the namespace_path it already sends to /allowed) to the Shell JWT, and Rails must pass it as allowed_namespace_path in the SSH-over-HTTP access check in Repositories::GitHttpController, behind a feature flag like P11. Both must be released and deployed to gprd, then P12 (e) passes on staging.
  • P14 — Routed SSH pack requests survive idle clients (request-body framing). Found in attempt 4 (failure mode A); the cause is confirmed in the "Case A analysis" comment by @vyaklushin (2026-11-09).
    • Fix: gitlab-org/gitlab-shell!1663 (merged) (gitlab-shell) and gitlab-org/gitlab!269512 (merged) (Rails and Workhorse), both Draft. gitlab-shell wraps the pack request body in frames (5-byte header, up to 64 KiB of data) and sends a keepalive frame after 5 idle seconds. Workhorse strips the frames and forwards the original git bytes to Gitaly. Neither the git client nor Gitaly changes. Framing is negotiated through /allowed (ssh_over_http_body_framing: 'v1', header Gitlab-Shell-Body-Framing: v1), behind the feature flag cells_ssh_body_framing. Without the flag, requests use raw bodies as today.
    • It covers: pull idle after ls-refs, pull idle after the server sends ready, and push. Protocol v0, v1 and v2.
    • Changing Cloudflare isn't an option (details in the thread under the attempt 4 summary comment): no documented or configurable Cloudflare limit is ~15 s; the reset comes from the Workers runtime (a probe Worker was also reset at ~15.1 s, while the same zone without a Worker route held 60 s); and Worker route patterns can't exclude */ssh-*-pack. A separate hostname with no Worker route is a possible long-term option, but not a fix for this rollout.
    • Rollout order (strict):
      1. gitlab-org/gitlab!269512 (merged) deployed to every cell. Old Workhorse can't decode framed bodies.
      2. gitlab-shell release with gitlab-org/gitlab-shell!1663 (merged), and GITLAB_SHELL_VERSION bumped.
      3. Enable cells_ssh_body_framing on staging.
      4. Staging: idle_fetch_test.sh gitlab-org/gitlab git@staging.gitlab.com "5 16 18 30 60" passes for every pause, plus an LFS push with a pre-push hook over 60 s. Use -o ControlMaster=no -o ControlPath=none.
      5. Performance check under concurrency (per-request memory is about 64 KiB, so watch gitlab-shell and Workhorse RSS).
      6. Enable on gprd, then the next Phase 1 attempt.
    • Canary gate: PROTOCOL_ERROR on routed sessions below 0.005% over a 60-minute soak (attempt 4: 0.044%). gitlab-shell logs show framing mode on routed requests.
    • Rollback: disabling cells_ssh_body_framing turns framing off without a deploy.
  • P15 — SSH authentication is not exposed to the edge latency tail. Found in attempt 4 (failure mode B). Not addressed by P14: authorized_keys and /allowed are short requests with no idle body, and they fail on gitlab-shell's own 10 s timeout while waiting for a stalled edge response. No fix is in progress yet. Either a hedged or retried authorized_keys request, or the internal API kept in-cluster with only pack traffic routed. Canary gate: edge auth timeouts within 2× the canary's own baseline over a 60-minute soak.
  • P16 — SLI accounting and rollback triggers fit the routed path.
    • gitlab-shell stops counting a client EOF after a complete response as a session error.
    • Triggers #1 (closed) and #2 (closed) become rates, for example: PROTOCOL_ERROR above 0.01% of routed sessions over 10 min; Internal API unreachable on edge URLs above 3× the canary baseline over 10 min. To be agreed with the change reviewer.

Preparation

Note

The following checklists must be done in advance, before setting the label changescheduled

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary.

Change Technician checklist

  • The Change Criticality has been set appropriately and requirements have been reviewed.
  • The change plan is technically accurate.
  • The rollback plan is technically accurate and detailed enough to be executed by anyone with access.
  • This Change Issue is linked to the appropriate Issue and/or Epic
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • The change execution window respects the Production Change Lock periods.
  • Once all boxes above are checked, mark the change request as scheduled: /label ~"change::scheduled"
  • For C1 and C2 change issues, the change event is added to the GitLab Production calendar by the change-scheduler bot.
  • For C1 and C2 change issues, Platform Leadership provides approval with the platform_leadership_approved label on the issue. Mention @gitlab-org/saas-platforms/change-review-leadership in this issue with a reference to review guidelines to get approval and provide visibility to all infrastructure managers.
  • For C1, C2, or blocks deployments change issues, confirm with Release managers that the change does not overlap or hinder any release process (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • For C1 change issues or C2 change issues happening during weekend, SREs on-call must be informed at least 2 weeks in advance. Check the incident.io GitLab.com Production EOC schedule to find who will be on-call at the scheduled day and time.
  • All Prerequisites P1–P16 are complete.

Detailed steps for the change

Rollout shape

releases/gitlab/values/gprd.yaml.gotmpl is shared by every gprd-prefixed environment, so putting the config there would enable all 308 pods at once. To get a staged rollout, the block goes into the per-environment files instead:

Phase File Environment Pods SSH traffic
1 releases/gitlab/values/gprd-cny.yaml.gotmpl gprd-cny 8 low but real — 6 authenticated sessions observed in 30 min
2 releases/gitlab/values/gprd-us-east1-b.yaml.gotmpl gprd-us-east1-b 100 ~1/3 of production
3 releases/gitlab/values/gprd-us-east1-c.yaml.gotmpl gprd-us-east1-c 100 ~1/3 of production
4 releases/gitlab/values/gprd-us-east1-d.yaml.gotmpl gprd-us-east1-d 100 remaining ~1/3

Phase 3 was split into c and d following review: no single step affects more than a third of traffic, and us-east1-d serves as a control group during Phase 3.

Each per-environment file is layered after gprd.yaml.gotmpl and deep-merged, so the new gitlab-shell.config.topologyService block keeps the existing config keys (loginGraceTime, proxyProtocol). The Topology Service address and mTLS secret come from the existing global.appConfig.cell.topologyServiceClient in gprd.yaml.gotmpl.

Regional gprd and gprd-gitlab-gke use regional_cluster_defaults with gitlab-shell: enabled: false, so they are unaffected.

gprd-cny is behind an internal load balancer (10.216.8.61, cloud.google.com/load-balancer-type: Internal) but unlike gstg-cny it does receive real authenticated SSH traffic, so Phase 1 is a genuine if low-volume canary.

Rollout timing per cluster: maxSurge 25%, maxUnavailable 0, minReadySeconds 30, terminationGracePeriodSeconds 30, PDB minAvailable 5%. Expect 10–15 minutes for a 100-pod cluster to fully cycle.

Pre-execution steps

Note

The following steps should be done right at the scheduled time of the change request. The preparation steps are listed below.

  • Make sure all tasks in Change Technician checklist are done
  • For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out and has provided approval with the eoc_approved label on the issue.
    • Check for active ~severity::1 / ~severity::2 incidents before requesting approval. If one overlaps with this change's blast radius, reschedule. If it does not overlap, include a brief risk assessment in the @sre-oncall approval request; if the EOC is unavailable or does not approve, reschedule.
  • For C1, C2, or blocks deployments change issues, Release managers have been informed prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • Confirm P1 is still active: verify the Cloudflare Configuration Rule is present and enabled on the production zone, and has not been reverted by a Terraform apply since it was created. This is the single most important pre-flight check. Since 2026-10-30 the rule is managed in config-mgmt environments/cloudflare-cfgrules-gprd. Check that the live ruleset ad55efc9… contains Disable Gitlab-shell SSH buffering.
  • Confirm P9 is still active: GET /zones/5b1bc06af128a829167e3a1212d86c28/settings/proxy_read_timeout returns 1800.
  • Confirm P6 on a pod in every target cluster: gitlab-sshd -version reports a release containing gitlab-shell!1570.
  • Confirm no Atlantis plan or apply is pending for environments/gprd in config-mgmt that touches cloudflare_ruleset.gitlab_com_config_rule (P1 drift risk).
  • Confirm /chatops run feature get cells_shell_jwt_key_claims shows the flag enabled on gprd (P11), plus the P13 flag once that exists.
  • Confirm no gprd auto-deploy is running or queued. The enable MR's pipeline waits behind it, and the deploy's Topology Service warm-up surge (ListLeases/GetCell) can be mistaken for routing load.
  • Record the Topology Service baseline: Classify latency histogram_quantile(0.99, sum by (le) (rate(gitlab_shell_topology_request_duration_seconds_bucket{env="gstg"}[5m]))) from gstg, the only environment with Classify traffic before Phase 1, plus Topology Service (Runway) request rate, p99 latency, error rate, CPU and instance count.
  • Record a baseline from the git service dashboard: gitlab_sshd apdex and error ratio, workhorse_auth_api apdex, haproxy_backend_sessions_total{backend=~"ssh|altssh"} rate, and gitlab-shell container RSS. Paste into a comment.
  • Open a thread in #production and post the baseline plus the rollback trigger list.

Change steps - steps to take to execute the change

Estimated Time to Complete (mins) - 330

Phase 1 — gprd-cny (8 pods). Estimated 45 min including soak.

  • Mark !5966 (attempt 4) ready and merge it. Attempt 3 was !5950. Earlier: mark !5909 ready and merge it (re-adds the topologyService block to releases/gitlab/values/gprd-cny.yaml.gotmpl; same change as !5877, which was merged and reverted in 9e8ea73f5).
  • Wait for the deploy pipeline and confirm the rollout completed: kubectl --context=gke_gitlab-production_us-east1_gprd-gitlab-gke -n gitlab-cny rollout status deploy/gitlab-cny-gitlab-shell.
  • Confirm the config landed in a pod: kubectl ... exec <pod> -c gitlab-shell -- sed -n '/topology_service/,/cell_endpoint/p' /srv/gitlab-shell/config.yml shows enabled: true, address: "topology-grpc.runway.gitlab.net:443", cell_endpoint: {scheme: https, port: 443}.
  • Success criteria, 30 min soak: at least one ssh-upload-pack and one ssh-receive-pack to https://gitlab.com:443 with status: 200 followed by session: handleShell: command executed successfully; zero PROTOCOL_ERROR; zero Internal API unreachable; zero cells routing: receive-pack response: Not found; gitlab_sshd apdex within baseline.
  • Perform an explicit manual validation from outside the fleet, against the canary. Canary gets only a small share of SSH traffic, so pin the session to a canary pod: kubectl --context=gke_gitlab-production_us-east1_gprd-gitlab-gke -n gitlab-cny port-forward pod/<gitlab-cny-gitlab-shell pod> 12722:2222, then export GIT_SSH_COMMAND='ssh -o HostName=127.0.0.1 -p 12722 -o HostKeyAlias=gitlab.com -o ControlMaster=no -o ControlPath=none'. Without ControlMaster=no, a shared connection reuses the first key and can bypass the pod. GIT_SSH_COMMAND is also honoured by git-lfs-authenticate. Use a dedicated test project on gitlab.com, not a real one, and remove the LFS objects and branches afterwards. Confirm in the pod's logs that each test hit it (Cells: using SSH-over-HTTP …).
    • Full clone of a large repository over SSH. It must exceed 60s of pack streaming, to prove the buffering rule works at production scale.
    • Small git push.
    • git lfs pull of at least one object of 100 MB or more, followed by git lfs fsck --objects.
    • Single-command git push that uploads a new LFS object of 1 GiB or more, so the pre-push hook takes longer than 60s. Expect ok refs/heads/.... Workhorse should log two ssh-receive-pack requests for the session, both 200, minutes apart. A 404 on the second one means gitlab-shell!1570 isn't running.
    • Delete the test branch afterwards.
  • Check routed sessions by authentication type in gitlab-shell logs: SSH keys, deploy keys, and SSH certificates (certificate_scope). A 403/404 on ssh-upload-pack/ssh-receive-pack after /allowed returned 200 is a rollback trigger (#13 (closed)).
  • P5 measurements (needed before Phase 2): from the cny pods, record HTTPS requests per SSH operation (gitlab_shell_http_requests_total rate ÷ session rate), new outbound connections per second, Cloud NAT gitlab-gke port usage and dropped packets, and any 429 or 403 returned to gitlab-shell. Post them in a comment and extrapolate to 100 pods.
  • If any rollback trigger fires, go to Rollback.

Phase 2 — gprd-us-east1-b (100 pods). Estimated 90 min including soak.

  • Confirm P5 is signed off using the Phase 1 measurements. Do not start Phase 2 without it.
  • Mark !5878 ready and merge it (releases/gitlab/values/gprd-us-east1-b.yaml.gotmpl).
  • kubectl --context=gke_gitlab-production_us-east1-b_gprd-us-east1-b -n gitlab rollout status deploy/gitlab-gitlab-shell — expect 10–15 min.
  • Confirm the traffic shift: gitlab_workhorse_http_requests_total{type="internal-api"} drops for the us-east1-b region while Cloudflare and HAProxy https frontend request rate rises correspondingly.
  • Success criteria, 60 min soak: git service error ratio within monitoring threshold 0.9999; gitlab_sshd apdex >= 0.998; zero PROTOCOL_ERROR and zero receive-pack response: Not found across the cluster; ssh-receive-pack non-200 rate in Workhorse at ~0; Topology Service Classify p99 within baseline and Topology Service (Runway) error rate and saturation unchanged; gitlab-shell container RSS stable versus the other two zonal clusters, which act as a live control group.
  • Compare us-east1-b against us-east1-c and -d side by side. Any divergence in SSH error rate or latency that is attributable to this change is a rollback trigger.

Phase 3 — gprd-us-east1-c (100 pods). Estimated 90 min including soak.

  • Mark !5879 ready and merge it (releases/gitlab/values/gprd-us-east1-c.yaml.gotmpl).
  • kubectl --context=gke_gitlab-production_us-east1-c_gprd-us-east1-c -n gitlab rollout status deploy/gitlab-gitlab-shell — expect 10–15 min.
  • Success criteria, 60 min soak: same thresholds as Phase 2. Topology Service load is now about 2/3 of production SSH, so watch Classify p99 and Runway saturation closely. us-east1-d is the remaining control group.

Phase 4 — gprd-us-east1-d (100 pods). Estimated 90 min including soak.

  • Mark !5880 ready and merge it (releases/gitlab/values/gprd-us-east1-d.yaml.gotmpl).
  • kubectl --context=gke_gitlab-production_us-east1-d_gprd-us-east1-d -n gitlab rollout status deploy/gitlab-gitlab-shell — expect 10–15 min.
  • Success criteria, 60 min soak: same thresholds as Phase 2, now with no control group. Watch haproxy_backend_sessions_total{backend=~"ssh|altssh"} for any drop indicating clients failing before reaching authentication.
  • Consolidate the per-cluster blocks into releases/gitlab/values/gprd.yaml.gotmpl in a follow-up MR after the change window, as a no-op cleanup.
  • Post final metrics to this issue and to #production.
  • Set label changecomplete /label ~change::complete

Rollback triggers

Roll back immediately, without further discussion, on any of:

  1. Any PROTOCOL_ERROR on a ssh-upload-pack or ssh-receive-pack request.
  2. Any Internal API unreachable log line.
  3. gitlab_sshd error ratio breaching the monitoring threshold (0.998 for this SLI) for more than 5 minutes.
  4. gitlab_sshd apdex below 0.998, or workhorse_auth_api apdex degrading below 0.998, sustained 5 minutes.
  5. A drop in haproxy_backend_sessions_total{backend=~"ssh|altssh"} completion rate versus baseline.
  6. gitlab-shell container RSS trending toward the 3Gi limit, or any OOMKill.
  7. Any user report of clone, fetch, push or LFS failure over SSH that correlates with the window.
  8. The Cloudflare Configuration Rule being found absent or reverted at any point.
  9. Any cells routing: receive-pack response: Not found (or 404 on */ssh-receive-pack or */ssh-upload-pack with user agent GitLab-Shell). This is the signature of an expired or rejected Shell JWT, meaning gitlab-shell!1570 isn't effective.
  10. A sustained rise in 524 responses on */gitlab-lfs/objects/* PUTs, indicating proxy_read_timeout has been reverted.
  11. Topology Service Classify p99 (gitlab_shell_topology_request_duration_seconds) above 500 ms for 5 minutes, or a rising status="fail" rate on gitlab_shell_topology_requests_total. (status="not_found" is the normal answer for projects not on a cell, from gitlab-shell!1605 onward; before that release NotFound was counted as fail, so use status="fail" only once !1605 is deployed.)
  12. Topology Service (Runway) error rate rising or saturation approaching its limits (CPU, instance count at max) during a phase.
  13. Any repeatable class of routed sessions where /allowed returns 200 and the cell then rejects the pack request with 403 or 404 (for example deploy keys, IP-restricted groups, SSH certificates). Attempts 2 and 3 were both rolled back on this.

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 30 per phase, 45 if all four phases are live (revert MRs can be merged in one go)

Important

There is no feature flag for this. Rollback is a Helm deploy and takes as long as a rolling restart of the affected clusters. The pre-created revert MRs from P7 exist specifically so that this is a single merge and not an authoring exercise under pressure.

  • Merge the pre-created revert MR for the most recently enabled phase: !5967 (cny, attempt 4; !5951 was attempt 3; !5910 was attempt 2), !5882 (b), !5883 (c), !5884 (d). Check its target branch is master first. If the failure is not clearly scoped to one cluster, merge every revert MR whose phase is live.
  • Confirm the deploy pipeline runs. If CI is blocked by the gitlab-shell change lock, set CHANGE_LOCK_OVERRIDE — a rollback must never be gated on the lock.
  • Watch kubectl rollout status deploy/gitlab-gitlab-shell for each affected cluster (10–15 min per 100-pod cluster).
  • Verify the config is gone from a pod: topology_service absent, and /api/v4/internal/allowed calls back on http://gitlab-webservice-internal-api.gitlab.svc.cluster.local:8181.
  • Confirm gitlab_sshd apdex and error ratio return to baseline.
  • If rollback does not restore service, escalate to the EOC and treat as an incident; the pre-change state is a plain Helm revert with no data or schema component, so there is no forward-only path to worry about.
  • Set label changeaborted /label ~change::aborted

Monitoring

Key metrics to observe

  • Metric: gitlab_sshd apdex — gitlab_shell_sshd_session_established_duration_seconds_bucket{type="git"}
    • Location: git service overview
    • What changes to this metric should prompt a rollback: apdex below 0.998 sustained 5 minutes. Satisfied threshold is 1s and tolerated is 5s; the added edge latency of 100–250ms per operation consumes a meaningful share of that budget.
  • Metric: gitlab_sshd error ratio — gitlab_sli:shell_sshd_connections:errors_total{type="git"} over gitlab_sli:shell_sshd_connections:total{type="git"}
    • Location: git service overview
    • What changes to this metric should prompt a rollback: breach of the 0.998 monitoring threshold for this SLI for more than 5 minutes.
  • Metric: workhorse_auth_api apdex and request rate — gitlab_workhorse_http_request_duration_seconds_bucket{type="internal-api", route="^/api/"}
    • Location: git service overview
    • What changes to this metric should prompt a rollback: apdex degradation below 0.998. A drop in request rate is expected and is the positive confirmation that traffic has shifted to the edge — do not treat it as an error, but do confirm the corresponding rise appears at the edge.
  • Metric: haproxy_backend_sessions_total{backend=~"ssh|altssh"}
    • Location: git service overview, HAProxy SSH load balancer SLI
    • What changes to this metric should prompt a rollback: any sustained drop versus baseline, which indicates clients failing before or during authentication.
  • Metric: PROTOCOL_ERROR and Internal API unreachable log lines
    • Location: Kibana shell index, json.type: git, filtered to json.url: *ssh-upload-pack* OR *ssh-receive-pack*
    • What changes to this metric should prompt a rollback: any occurrence. Zero is the only acceptable value; this is the exact signature of the three staging reverts.
  • Metric: Cells pack-request status in Workhorse: json.route_id: (ssh_upload_pack OR ssh_receive_pack) AND json.user_agent: "GitLab-Shell", grouped by json.status
    • Location: Kibana workhorse index. Correlate with the shell index for cells routing: receive-pack response: Not found.
    • What changes to this metric should prompt a rollback: any 404, which is the expired or rejected Shell JWT signature fixed by gitlab-shell!1570. ssh-receive-pack normally shows two requests per push: a short advertisement, then the real push.
  • Metric: Topology Service Classify latency — gitlab_shell_topology_request_duration_seconds (histogram)
    • Location: Prometheus / git service overview (ad-hoc panel: histogram_quantile(0.99, sum by (le, cluster) (rate(gitlab_shell_topology_request_duration_seconds_bucket{env="gprd"}[5m]))))
    • What changes to this metric should prompt a rollback: p99 above 500 ms for 5 minutes. Classify runs on every SSH Git operation before authentication, so its latency is added directly to gitlab_sshd apdex.
  • Metric: Topology Service (Runway) saturation and errors
    • Location: Runway service dashboard for the Topology Service (topology-grpc.runway.gitlab.net): request rate, error rate, p99 latency, CPU, instance count
    • What changes to this metric should prompt a rollback: error rate rising, or CPU or instance count approaching their limits. Each phase adds about a third of gitlab.com SSH operations as Classify RPCs.
  • Metric: gitlab_shell_topology_requests_total
    • Location: Kibana shell index / Prometheus
    • What changes to this metric should prompt a rollback: a rising status="fail" rate indicates Topology Service Classify RPC failures (not_found is expected and excluded). Note this counts only the Classify call and is not sufficient on its own as a success signal — a completed ssh-upload-pack with status: 200 followed by command executed successfully is the real proof.
  • Metric: gitlab-shell container RSS — kube_container_rss for the gitlab-shell deployment
    • Location: git service overview, Kubernetes resources
    • What changes to this metric should prompt a rollback: trend toward the 3Gi limit or any OOMKill. Holding pack streams open across the edge for minutes at a time is a new memory profile for this fleet and has never been exercised at production concurrency.
  • Metric: Cloudflare and HAProxy https frontend request rate and error rate
    • Location: Cloudflare service dashboard
    • What changes to this metric should prompt a rollback: 5xx rate increase, or edge saturation attributable to the added internal-API and pack traffic.

🤖 Created with the help of OpenCode Duo

Edited by Vladimir Glafirov