altssh.gitlab.com unavailable after the front-end change
Following infrastructure#1747 altssh.gitlab.com became unavailable.
I have found the root cause in the old lb10 and lb11 not being able to load the new configuration.
The fix was to simply update haproxy.cfg with the new nodes. The outage ended today at 07:22 UTC.
This is what we need to avoid this from happening again in the future:
- Review our monitoring for altssh.
- Move the service to a new set of load balancers in ARM.
- Delete everything from the old platform to avoid confusion.
- As a more general thing, don't leave services/nodes half migrated.