global haproxy state file causes certain HAProxy config changes to break
Overview
On staging we recently changed the websockets backend to connect directly to workhorse:
backend websockets
mode http
balance roundrobin
option splice-auto
option http-server-close
timeout tunnel 8s
cookie _gitlab_session prefix nocache
option tcp-check
server git-gke-us-east1-b 10.224.44.217:8181 weight 100 check port 8181 inter 3s fastinter 1s downinter 5s fall 3 cookie git-gke-us-east1-b
server git-gke-us-east1-c 10.224.45.22:8181 weight 100 check port 8181 inter 3s fastinter 1s downinter 5s fall 3 cookie git-gke-us-east1-c backup
server git-gke-us-east1-d 10.224.46.209:8181 weight 100 check port 8181 inter 3s fastinter 1s downinter 5s fall 3 cookie git-gke-us-east1-d backup
This changed the IPs and port, however the state file /etc/haproxy/state/global
remained pointing to the previous IPs:
...
19 websockets 1 git-gke-us-east1-b 10.224.44.2 2 0 100 100 269207 6 3 4 6 0 0 0 - 443 -
19 websockets 2 git-gke-us-east1-c 10.224.45.2 2 0 100 100 124251 6 3 4 6 0 0 0 - 443 -
19 websockets 3 git-gke-us-east1-d 10.224.46.5 2 0 100 100 195594 6 3 4 6 0 0 0 - 443 -
This is quite bad because all connections to the websockets backend were failing with a 503 after the chef run.
Action Items
-
Delete runbook
Edited by Steve Xuereb