Investigate Container Registry Pods being ready and passing liveness checks were not taking traffic

During a recent incident: #661 (closed) it was discovered that Pods were Ready but not taking any traffic. Haproxy was still serving a 503 until a configuration change was reverted. Determine the cause of why we were passing the healthchecks and not accepting any traffic during this period of time.

This can be observed via our logs: https://log.gitlab.net/goto/b07a5a30f8e6c28281fdc3506e015bd9

This can be seen via our backend metric serving 5xx errors: https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.end_input=2020-02-04%2021%3A33&g0.max_source_resolution=0s&g0.expr=sum(rate(haproxy_backend_http_responses_total%7Benv%3D%22gprd%22%2C%20backend%3D%22registry%22%7D%5B1m%5D))%20by%20(code)&g0.tab=0