## Problem Canary drains and server drains on the HAProxy fleet work by connecting to every node from outside. HAProxy binds an admin socket on the node's IP (`stats socket ipv4@<node-ip>:23646 level admin`), ChatOps looks up all node IPs through the Chef server API, opens raw TCP to each, and issues [`set server ... state ready|drain|maint`](https://docs.haproxy.org/2.8/management.html#9.3-set%20server). The `set-server-state` scripts in chef-repo drive the same socket over `knife ssh` for zonal backend drains during cluster maintenance. Beyond the interface itself, the state model causes recurring problems: - Server state lives only in each node's memory. It is dumped to a server-state file to survive reloads, but the file also caches backend addresses, which went stale and caused incidents (production-engineering#12152, [haproxy#3296](https://github.com/haproxy/haproxy/issues/3296), gitlab-cookbooks/gitlab-haproxy!439). When persistence was disabled in response, every reload (including unattended upgrades) silently reset canary state. - Undraining canary restores full weight instantly, while the canary fleets need time to scale back up. - New or rebooted nodes come up with default state and know nothing about an in-progress drain (production-engineering#27563). ## Proposal Move the desired state out of the HAProxy processes: - ChatOps writes desired state to Consul KV, for example `frontend/<env>/canary-state`, and `frontend/<env>/server-overrides/<backend>/<server>` to replace `set-server-state`. Precedent exists on both ends: ChatOps already has a Consul KV client, and Consul runs in every environment. - HAProxy's native [`agent-check`](https://docs.haproxy.org/2.8/configuration.html#5.2-agent-check) delivers it: canary servers (and any server we want drainable) carry an agent check against a small responder that reads the KV and answers `ready`, `drain`, `maint`, or a weight percentage. [`agent-send`](https://docs.haproxy.org/2.8/configuration.html#5.2-agent-send) encodes the server identity, so one responder answers for the whole fleet. The [agent checks tutorial](https://www.haproxy.com/documentation/haproxy-configuration-tutorials/reliability/health-checks/#agent-checks) has a worked example. - The responder runs next to each HAProxy process: a systemd unit on the VMs, a sidecar in the pods, answering on localhost and reading desired state through the local consul agent. It is new code to build and package, but small: read the KV, print one line. A responder that is down freezes only its own instance and gets restarted by systemd or the kubelet. - Convergence is pull-based. Every process polls at [`agent-inter`](https://docs.haproxy.org/2.8/configuration.html#5.2-agent-inter) (2s default), so reloads, unattended upgrades, and node replacements converge within seconds. A responder outage changes nothing: failure to connect is documented as not an error, state just freezes. - Gradual undrain lives in the responder, which can raise the served weight percentage stepwise. HAProxy's native [`slowstart`](https://docs.haproxy.org/2.8/configuration.html#5.2-slowstart) ([tutorial](https://www.haproxy.com/documentation/haproxy-configuration-tutorials/reliability/circuit-breakers/)) cannot do this uniformly: it fires when a server leaves MAINT or recovers from a failed health check, but not when leaving DRAIN (the server stays operationally UP throughout, so there is no up-transition). Once agent checks carry the state, the server-state-file machinery and the TCP-bound admin socket can both be removed, and ChatOps drops its Chef API dependency for this path. Manual state changes through the stats UI or the unix socket get overwritten at the next agent check. That is the desired-state model working, but it changes the incident muscle memory: on servers with agent checks, the override goes into KV, not into the socket. ```mermaid graph TB chatops["ChatOps canary"] -->|"list node IPs"| chef1["Chef server API"] chef1 -->|"set server ... state"| sock1["admin socket<br/>TCP :23646"] sock1 --> vms1["haproxy VMs"] chatops -->|"canary state"| kv1["Consul KV"] kv1 --> agent1["agent responder"] agent1 -->|"agent-check poll"| vms1 sss["chef-repo set-server-state"] -->|"node query"| chef2["Chef server API"] chef2 -->|"knife ssh + unix socket"| vms2["haproxy VMs"] sss -->|"server overrides"| kv2["Consul KV"] kv2 --> agent2["agent responder"] agent2 -->|"agent-check poll"| vms2 hactl["deployer ha-ctl precheck"] -->|"show stat"| sock3["admin socket<br/>TCP :23646"] hactl -.->|"read path, tbd"| prom["prometheus or KV"] sock3 --> vms3["haproxy VMs"] prom ~~~ vms3 classDef old fill:#ffcdd2,stroke:#b71c1c,color:#4a0d0d classDef new fill:#c8e6c9,stroke:#2e7d32,color:#1b3d1f class chef1,sock1,chef2,sock3 old class kv1,agent1,kv2,agent2,prom new ``` This is deliberately independent of the Kubernetes migration (production-engineering#29644). It lands on the Chef VMs as a cookbook change and fixes real problems there: durable state across reloads, startup convergence, gradual undrain. If the Kubernetes migration proceeds, the pods run the same responder and the control mechanism does not change again. It also opens the path to retiring ChatOps-driven draining altogether: once the state is a KV write, the deployment pipeline can set canary state itself instead of a human running `/chatops run canary` around every deploy. ## Contracts that must survive - `/chatops run canary --gprd|--gstg --ready|--drain|--maint|--disable` stays identical for release managers. - The SRE zonal-drain workflow (`set-server-state -z`) gets an equivalent through the KV override namespace. - The `canary_active_deployment` safety check in ChatOps keeps working (it queries the Chef API today and needs a new source regardless). - `haproxy_backend_up` and related metrics are unaffected: this changes how state is set, not how HAProxy reports it. - The deployer's `ha-ctl` also drives the TCP socket. The active consumer is the deploy precheck: every relevant server must be UP or MAINT (DRAIN blocks the deploy, canary entries checked during the cny stage). The read-path replacement must preserve this gate. The drain/undrain wrapper in the generic command pipeline is vestigial: it drains server entries named after the target VM, and no VM-named entries remain in the config. The ChatOps `deploy` and `deploycmd` commands only gate that wrapper (`--skip-haproxy`). The full socket consumer inventory: ChatOps `canary`, the chef-repo `set-server-state` scripts, and `ha-ctl`. ## Open questions - KV schema: one key per env with the full desired state as a document, or one key per backend/server. A single document is atomic and easy to reason about, per-server keys allow partial updates. - Consul access from ChatOps CI jobs to the gstg and gprd Consul clusters: network reach exists (the jobs connect to the node IPs today), but the HTTP API endpoint, ACLs, and authentication need to be settled. - Detecting a stuck responder: the prometheus exporter does not expose `agent_status`, so agent-check failures need surfacing some other way. - Whether the weight step rate for gradual undrain is fixed or part of the desired state. - Read path after the socket is gone: prometheus already carries per-server state and weight (release-tools reads canary state from it today), or desired state is read straight from KV. ## Rollout A node attribute (a node list or flag in the role JSON) gates the migration per node. The Chef recipe renders the `agent-check` config on nodes that have the attribute, and ChatOps, which already queries the Chef API for node IPs, fetches the attribute in the same search and skips the direct socket for the same nodes. On those nodes only the agent path applies state, so its failures are visible instead of being masked by the socket path writing the same state. The rest of the fleet stays on the old path, driven by the same ChatOps command. Rollback is removing a node from the list. State reads (`show stat` for the ChatOps status display and the ha-ctl precheck) keep using the socket on all nodes during the transition, and get replaced before the socket is removed (the read-path open question above). 1. gstg: agent checks on a few nodes via the attribute, then the whole environment, exercised through a few deploy cycles including a canary drain and undrain. 2. gprd: same, a few nodes at a time, registry and pages before main. 3. Switch the status display off the socket (prometheus or KV). 4. Remove the TCP-bound admin socket, the server-state-file machinery, the Chef API lookup in ChatOps, and the direct-socket code in `chef-repo/bin`. ## Alternatives considered **Watcher daemon on each node.** A daemon watches the KV and applies state through the local unix socket. Same store, but custom socket-driving code plus the convergence and startup handling that `agent-check` provides natively. Rejected in favor of agent checks. **Draining by failing health checks.** The pattern the GCP LB drain script uses for whole nodes. Marks servers DOWN instead of DRAIN or MAINT, fights the kubernetes readiness machinery on the canary backends, and breaks the deployer precheck, which treats MAINT as deployable and DOWN as a blocker. Rejected. **Canary decision in the Cloudflare cells http-router.** The router already runs on every gitlab.com HTTP request. It would set a header and HAProxy routes on it statelessly, removing the runtime state for HTTP canary entirely. A plausible long-term home for that part, but it covers neither ssh and kas (Spectrum direct apps, no HTTP pipeline) nor zonal drains, the router is young, and drain propagation across POPs is unproven. Not pursued now. The KV schema should not preclude moving the HTTP decision there later. **Canary at a GCP Application Load Balancer.** HTTP only, so no ssh. Inline data processing at gitlab.com volume costs an order of magnitude more per month than the whole HAProxy fleet, and URL map changes propagate too slowly for a release-manager drain. Rejected. **Canary weights in config management only.** An MR merge plus converge takes minutes to hours, release managers need seconds. Rejected as the only mechanism, config still holds the defaults. **Other responder placements.** The agent protocol is raw TCP (one ASCII line per connection), so something has to serve it. A central responder per environment behind an ILB means one deployable and fleet-wide consistency, but needs an HA story. consul-template rendering a file served by a socket-activated unit needs no code at all, but gradual undrain moves into the KV writer. The local sidecar keeps the stepping logic and has nothing to design for HA.