global haproxy state file causes certain HAProxy config changes to break

Overview

On staging we recently changed the websockets backend to connect directly to workhorse:

backend websockets
    mode http
    balance roundrobin
    option splice-auto
    option http-server-close
    timeout tunnel 8s
    cookie _gitlab_session prefix nocache
    option tcp-check
    server git-gke-us-east1-b 10.224.44.217:8181 weight 100 check port 8181 inter 3s fastinter 1s downinter 5s fall 3  cookie git-gke-us-east1-b
    server git-gke-us-east1-c 10.224.45.22:8181 weight 100 check port 8181 inter 3s fastinter 1s downinter 5s fall 3  cookie git-gke-us-east1-c backup
    server git-gke-us-east1-d 10.224.46.209:8181 weight 100 check port 8181 inter 3s fastinter 1s downinter 5s fall 3  cookie git-gke-us-east1-d backup

This changed the IPs and port, however the state file /etc/haproxy/state/global remained pointing to the previous IPs:

...
19 websockets 1 git-gke-us-east1-b 10.224.44.2 2 0 100 100 269207 6 3 4 6 0 0 0 - 443 -
19 websockets 2 git-gke-us-east1-c 10.224.45.2 2 0 100 100 124251 6 3 4 6 0 0 0 - 443 -
19 websockets 3 git-gke-us-east1-d 10.224.46.5 2 0 100 100 195594 6 3 4 6 0 0 0 - 443 -

This is quite bad because all connections to the websockets backend were failing with a 503 after the chef run.

Action Items

• Delete runbook