## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/632171) currently behind the `route_gitaly_through_gitway` feature flag. The flag makes `Gitlab::GitalyClient.address` return a storage's `gitway_address` instead of its `gitaly_address`. That routes the storage's Gitaly traffic through [gitway](https://gitlab.com/gitlab-org/gitway), a transparent gRPC proxy in front of Gitaly. The switch covers Rails, Workhorse and gitlab-shell, which all take the address from Rails. Gitaly-to-Gitaly traffic always stays on `gitaly_address`. The flag has no effect on a storage with no `gitway_address`. - DRI: @jcaigitlab - Team Slack channel: `#g_gitaly` - Introduced by: gitlab-org/gitlab!259697 - Deployment epic: https://gitlab.com/groups/gitlab-org/-/work_items/23627 > [!note] > This is an `ops` flag and is expected to live for the whole gitway pilot. Review it every 12 months and bump `milestone` in the YAML definition, as the ops-flag guidelines require. ## Prerequisites The flag does nothing until storages have a `gitway_address`. Before the first step in any environment: - [ ] Rails change merged and deployed: gitlab-org/gitlab!259697 - [ ] Helm chart renders `gitway_address` for `global.gitaly.external`: gitlab-org/charts/gitlab#6695 - [ ] Omnibus support and docs, for Omnibus-managed nodes: gitlab-org/omnibus-gitlab#10131 - [ ] `gitway_address` configured for each storage in the target environment, pointing at that storage's gitway listener - [ ] gitway deployed and reachable from web, API, Sidekiq and gitlab-shell, and from KAS, Zoekt and the indexers, which also take their address from Rails. Tracked under the deployment epic. - [ ] gitway dashboard and runbook published: gitlab-com/gl-infra/production#22968, gitlab-com/gl-infra/production-engineering#29807 ## What could go wrong? - **Blast radius.** Every Gitaly RPC for a routed storage passes through gitway. If gitway fails, that storage's repository reads and writes fail, including SSH and HTTP clone and push through gitlab-shell and Workhorse. - **Sidechannels.** Clone and fetch use sidechannel RPCs (`SSHUploadPackWithSidechannel`, `PostUploadPackWithSidechannel`). These can fail through gitway even while ordinary RPCs look healthy. Watch fetch error rates specifically. - **Rollback.** Setting the flag to `false` takes effect within the L1 feature-flag cache TTL (about 1 minute). No restart or config change is needed. The cache keys connections by storage and address, so Rails' own connections move as well. - **Connection count.** A process keeps its connection to whichever address it is no longer using until it restarts. During rollout, each Rails process can hold connections to both Gitaly and gitway for a storage. This is bounded at two channels per storage per process. - **Latency.** Each RPC takes one extra hop. gitway's added latency per RPC is the go/no-go signal for each step. Dashboards: - gitway dashboard (gitlab-com/gl-infra/production#22968): added latency per RPC, status-code fidelity, sidechannel upgrade success. - Gitaly client SLIs and Gitaly service dashboards on https://dashboards.gitlab.net: error rate and latency for routed storages. ## Events - [Exceptions with route_gitaly_through_gitway:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22route_gitaly_through_gitway:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with route_gitaly_through_gitway:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22route_gitaly_through_gitway:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) ## Rollout We ramp by **percentage of time**, not per storage, and not by percentage of actors. Each Gitaly call independently goes direct or through gitway, so traffic moves gradually across every configured storage at once. `--actors` is deliberately left off the commands below. Which storages can be routed at all is decided by which ones have a `gitway_address`. Storage targeting through ChatOps isn't supported: `Feature::Target` has no storage actor. Run all production ChatOps in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_gitaly`. **Non-production (staging first)** ``` /chatops gitlab run feature set route_gitaly_through_gitway 1 --staging /chatops gitlab run feature set route_gitaly_through_gitway 10 --staging /chatops gitlab run feature set route_gitaly_through_gitway 50 --staging /chatops gitlab run feature set route_gitaly_through_gitway true --staging ``` **Production.** Production waits until gitway has run in staging for an agreed soak period. Wait at least 15 minutes between steps, and longer at low percentages, watching the dashboards above. ``` /chatops gitlab run feature set route_gitaly_through_gitway 1 /chatops gitlab run feature set route_gitaly_through_gitway 5 /chatops gitlab run feature set route_gitaly_through_gitway 25 /chatops gitlab run feature set route_gitaly_through_gitway 50 /chatops gitlab run feature set route_gitaly_through_gitway true ``` ## Before global rollout - [ ] [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened for each production step. Every Gitaly RPC is in scope, so treat this as high-risk. - [ ] Runbook covers bypassing gitway with this flag, and each gitway alert links to it. - [ ] Docs: `gitway_address` is internal to the pilot and documented only in `gitlab.yml.example`. No user-facing documentation is needed while the flag is off for self-managed. ## Cleanup This flag is temporary. Once gitway is permanent, collapse the configuration instead of keeping two addresses: 1. Point each storage's `gitaly_address` at gitway, and remove `gitway_address`. 2. Remove the flag, `route_through_gitway?`, and the `gitway_address` handling in `Gitlab::GitalyClient` and `stub_storage_settings`. The `[storage, address]` cache key can then go back to being keyed by storage alone. 3. Remove the key from the Helm chart (gitlab-org/charts/gitlab#6695) and the Omnibus docs (gitlab-org/omnibus-gitlab#10131). ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete route_gitaly_through_gitway --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set route_gitaly_through_gitway false # production /chatops gitlab run feature set route_gitaly_through_gitway false --dev --pre --staging --staging-ref # non-production ``` All traffic returns to `gitaly_address` within about a minute, with no restart.