2019-02-12 Registry nodes drain timeout

Summary

A brief summary of what happened. Try to make it as executive-friendly as possible.
Service(s) affected : Registry
Team attribution : ?
Minutes downtime or degradation : 5
Timeline

2019-02-12
14:50 UTC - Registry deployment started https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/250823
15:00 UTC - Alert firing for increased registry errors https://gitlab.slack.com/archives/C101F3796/p1549983648038800 . Triage dashboard https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage?orgId=1&from=1549982797580&to=1549984159538
15:05 UTC - The number of errors dropped as registry fleet finished the deploy
RCA

The registry-04-sv-gprd node did not drain correct from the LB and the deployment continued after a timeout:
ok: [registry-04-sv-gprd.c.gitlab-production.internal] => (item=haproxy_wait_drain.stderr_lines) => 
  haproxy_wait_drain.stderr_lines:
  - '[INFO] Tue Feb 12 14:59:08 UTC 2019: Running action: wait_until_drained for registry-04-sv-gprd'
  - '[INFO] Tue Feb 12 14:59:08 UTC 2019: Waiting for connection on registry-04-sv-gprd to be drained'
  - '[INFO] Tue Feb 12 14:59:08 UTC 2019: registry-04-sv-gprd not drained, checking 0 / 60'
  - '[INFO] Tue Feb 12 14:59:09 UTC 2019: registry-04-sv-gprd not drained, checking 1 / 60'
  - '[INFO] Tue Feb 12 14:59:09 UTC 2019: registry-04-sv-gprd not drained, checking 1 / 60'
  - '[INFO] Tue Feb 12 14:59:10 UTC 2019: registry-04-sv-gprd not drained, checking 2 / 60'
  - '[INFO] Tue Feb 12 14:59:11 UTC 2019: registry-04-sv-gprd not drained, checking 3 / 60'
  - '[INFO] Tue Feb 12 14:59:12 UTC 2019: registry-04-sv-gprd not drained, checking 4 / 60'
  - '[INFO] Tue Feb 12 14:59:13 UTC 2019: registry-04-sv-gprd not drained, checking 5 / 60'
  - '[INFO] Tue Feb 12 14:59:14 UTC 2019: registry-04-sv-gprd not drained, checking 6 / 60'
  - '[INFO] Tue Feb 12 14:59:14 UTC 2019: registry-04-sv-gprd not drained, checking 6 / 60'
  - '[INFO] Tue Feb 12 14:59:15 UTC 2019: registry-04-sv-gprd not drained, checking 7 / 60'
  - '[INFO] Tue Feb 12 14:59:16 UTC 2019: registry-04-sv-gprd not drained, checking 8 / 60'
  - '[INFO] Tue Feb 12 14:59:17 UTC 2019: registry-04-sv-gprd not drained, checking 9 / 60'
  - '[INFO] Tue Feb 12 14:59:17 UTC 2019: registry-04-sv-gprd not drained, checking 9 / 60'
  - '[INFO] Tue Feb 12 14:59:18 UTC 2019: registry-04-sv-gprd not drained, checking 10 / 60'
  - '[INFO] Tue Feb 12 14:59:19 UTC 2019: registry-04-sv-gprd not drained, checking 11 / 60'
  - '[INFO] Tue Feb 12 14:59:20 UTC 2019: registry-04-sv-gprd not drained, checking 12 / 60'
  - '[INFO] Tue Feb 12 14:59:21 UTC 2019: registry-04-sv-gprd not drained, checking 13 / 60'
  - '[INFO] Tue Feb 12 14:59:21 UTC 2019: registry-04-sv-gprd not drained, checking 13 / 60'
  - '[INFO] Tue Feb 12 14:59:22 UTC 2019: registry-04-sv-gprd not drained, checking 14 / 60'
  - '[INFO] Tue Feb 12 14:59:23 UTC 2019: registry-04-sv-gprd not drained, checking 15 / 60'
  - '[INFO] Tue Feb 12 14:59:24 UTC 2019: registry-04-sv-gprd not drained, checking 16 / 60'
  - '[INFO] Tue Feb 12 14:59:25 UTC 2019: registry-04-sv-gprd not drained, checking 17 / 60'
  - '[INFO] Tue Feb 12 14:59:25 UTC 2019: registry-04-sv-gprd not drained, checking 17 / 60'
  - '[INFO] Tue Feb 12 14:59:26 UTC 2019: registry-04-sv-gprd not drained, checking 18 / 60'
  - '[INFO] Tue Feb 12 14:59:27 UTC 2019: registry-04-sv-gprd not drained, checking 19 / 60'
  - '[INFO] Tue Feb 12 14:59:28 UTC 2019: registry-04-sv-gprd not drained, checking 20 / 60'
  - '[INFO] Tue Feb 12 14:59:28 UTC 2019: registry-04-sv-gprd not drained, checking 20 / 60'
  - '[INFO] Tue Feb 12 14:59:29 UTC 2019: registry-04-sv-gprd not drained, checking 21 / 60'
  - '[INFO] Tue Feb 12 14:59:30 UTC 2019: registry-04-sv-gprd not drained, checking 22 / 60'
  - '[INFO] Tue Feb 12 14:59:31 UTC 2019: registry-04-sv-gprd not drained, checking 23 / 60'
  - '[INFO] Tue Feb 12 14:59:32 UTC 2019: registry-04-sv-gprd not drained, checking 24 / 60'
  - '[INFO] Tue Feb 12 14:59:32 UTC 2019: registry-04-sv-gprd not drained, checking 24 / 60'
  - '[INFO] Tue Feb 12 14:59:33 UTC 2019: registry-04-sv-gprd not drained, checking 25 / 60'
  - '[INFO] Tue Feb 12 14:59:34 UTC 2019: registry-04-sv-gprd not drained, checking 26 / 60'
  - '[INFO] Tue Feb 12 14:59:35 UTC 2019: registry-04-sv-gprd not drained, checking 27 / 60'
  - '[INFO] Tue Feb 12 14:59:36 UTC 2019: registry-04-sv-gprd not drained, checking 28 / 60'
  - '[INFO] Tue Feb 12 14:59:37 UTC 2019: registry-04-sv-gprd not drained, checking 29 / 60'
  - '[INFO] Tue Feb 12 14:59:37 UTC 2019: registry-04-sv-gprd not drained, checking 29 / 60'
  - '[INFO] Tue Feb 12 14:59:38 UTC 2019: registry-04-sv-gprd not drained, checking 30 / 60'
  - '[INFO] Tue Feb 12 14:59:39 UTC 2019: registry-04-sv-gprd not drained, checking 31 / 60'
  - '[INFO] Tue Feb 12 14:59:40 UTC 2019: registry-04-sv-gprd not drained, checking 32 / 60'
  - '[INFO] Tue Feb 12 14:59:41 UTC 2019: registry-04-sv-gprd not drained, checking 33 / 60'
  - '[INFO] Tue Feb 12 14:59:42 UTC 2019: registry-04-sv-gprd not drained, checking 34 / 60'
  - '[INFO] Tue Feb 12 14:59:42 UTC 2019: registry-04-sv-gprd not drained, checking 34 / 60'
  - '[INFO] Tue Feb 12 14:59:43 UTC 2019: registry-04-sv-gprd not drained, checking 35 / 60'
  - '[INFO] Tue Feb 12 14:59:44 UTC 2019: registry-04-sv-gprd not drained, checking 36 / 60'
  - '[INFO] Tue Feb 12 14:59:45 UTC 2019: registry-04-sv-gprd not drained, checking 37 / 60'
  - '[INFO] Tue Feb 12 14:59:45 UTC 2019: registry-04-sv-gprd not drained, checking 37 / 60'
  - '[INFO] Tue Feb 12 14:59:46 UTC 2019: registry-04-sv-gprd not drained, checking 38 / 60'
  - '[INFO] Tue Feb 12 14:59:47 UTC 2019: registry-04-sv-gprd not drained, checking 39 / 60'
  - '[INFO] Tue Feb 12 14:59:48 UTC 2019: registry-04-sv-gprd not drained, checking 40 / 60'
  - '[INFO] Tue Feb 12 14:59:49 UTC 2019: registry-04-sv-gprd not drained, checking 41 / 60'
  - '[INFO] Tue Feb 12 14:59:49 UTC 2019: registry-04-sv-gprd not drained, checking 41 / 60'
  - '[INFO] Tue Feb 12 14:59:50 UTC 2019: registry-04-sv-gprd not drained, checking 42 / 60'
  - '[INFO] Tue Feb 12 14:59:51 UTC 2019: registry-04-sv-gprd not drained, checking 43 / 60'
  - '[INFO] Tue Feb 12 14:59:52 UTC 2019: registry-04-sv-gprd not drained, checking 44 / 60'
  - '[INFO] Tue Feb 12 14:59:52 UTC 2019: registry-04-sv-gprd not drained, checking 44 / 60'
  - '[INFO] Tue Feb 12 14:59:53 UTC 2019: registry-04-sv-gprd not drained, checking 45 / 60'
  - '[INFO] Tue Feb 12 14:59:54 UTC 2019: registry-04-sv-gprd not drained, checking 46 / 60'
  - '[INFO] Tue Feb 12 14:59:55 UTC 2019: registry-04-sv-gprd not drained, checking 47 / 60'
  - '[INFO] Tue Feb 12 14:59:56 UTC 2019: registry-04-sv-gprd not drained, checking 48 / 60'
  - '[INFO] Tue Feb 12 14:59:56 UTC 2019: registry-04-sv-gprd not drained, checking 48 / 60'
  - '[INFO] Tue Feb 12 14:59:57 UTC 2019: registry-04-sv-gprd not drained, checking 49 / 60'
  - '[INFO] Tue Feb 12 14:59:58 UTC 2019: registry-04-sv-gprd not drained, checking 50 / 60'
  - '[INFO] Tue Feb 12 14:59:59 UTC 2019: registry-04-sv-gprd not drained, checking 51 / 60'
  - '[INFO] Tue Feb 12 14:59:59 UTC 2019: registry-04-sv-gprd not drained, checking 51 / 60'
  - '[INFO] Tue Feb 12 15:00:00 UTC 2019: registry-04-sv-gprd not drained, checking 52 / 60'
  - '[INFO] Tue Feb 12 15:00:01 UTC 2019: registry-04-sv-gprd not drained, checking 53 / 60'
  - '[INFO] Tue Feb 12 15:00:02 UTC 2019: registry-04-sv-gprd not drained, checking 54 / 60'
  - '[INFO] Tue Feb 12 15:00:03 UTC 2019: registry-04-sv-gprd not drained, checking 55 / 60'
  - '[INFO] Tue Feb 12 15:00:04 UTC 2019: registry-04-sv-gprd not drained, checking 56 / 60'
  - '[INFO] Tue Feb 12 15:00:04 UTC 2019: registry-04-sv-gprd not drained, checking 56 / 60'
  - '[INFO] Tue Feb 12 15:00:05 UTC 2019: registry-04-sv-gprd not drained, checking 57 / 60'
  - '[INFO] Tue Feb 12 15:00:06 UTC 2019: registry-04-sv-gprd not drained, checking 58 / 60'
  - '[INFO] Tue Feb 12 15:00:07 UTC 2019: registry-04-sv-gprd not drained, checking 59 / 60'
  - '[INFO] Tue Feb 12 15:00:07 UTC 2019: registry-04-sv-gprd not drained, checking 59 / 60'
  - '[INFO] Tue Feb 12 15:00:08 UTC 2019: registry-04-sv-gprd not drained, checking 60 / 60'
  - '[INFO] Tue Feb 12 15:00:09 UTC 2019: Timed out waiting for registry-04-sv-gprd to be drained. But we can continue.'
Edited Feb 12, 2019 by Marin Jankovski