Skip to content

Pingdom check check:https://gitlab.com/gitlab-org/gitlab-ce/issues down during deploy

Timeline

  • 11:03 Alert fires
  • 11:08 Alert clears

p50 latency spikes

10:42 and 11:04 we saw latency spikes on the web fleet

The following p50 rails latency spikes were also observed Screen_Shot_2018-10-31_at_12.22.14_PM

Workhorse

10:40 and 11:02 we saw traffic dips

Screen_Shot_2018-10-31_at_12.25.58_PM

It appears that takeoff is initiating a hup to unicorn without any haproxy draining, these are also very close to each other.

web-cny-02-sv-gprd.c.gitlab-production.internal Oct 31 07:24:10 web-cny-02-sv-gprd sudo:  takeoff : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-cny-02-sv-gprd.c.gitlab-production.internal Oct 31 07:25:27 web-cny-02-sv-gprd sudo:  takeoff : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-cny-02-sv-gprd.c.gitlab-production.internal Oct 31 10:39:02 web-cny-02-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-01-sv-gprd.c.gitlab-production.internal     Oct 31 10:39:06 web-01-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-01-sv-gprd.c.gitlab-production.internal     Oct 31 11:00:33 web-01-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-09-sv-gprd.c.gitlab-production.internal     Oct 31 10:39:05 web-09-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-09-sv-gprd.c.gitlab-production.internal     Oct 31 11:00:42 web-09-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-13-sv-gprd.c.gitlab-production.internal     Oct 31 10:38:49 web-13-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-13-sv-gprd.c.gitlab-production.internal     Oct 31 11:00:47 web-13-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-06-sv-gprd.c.gitlab-production.internal     Oct 31 10:38:46 web-06-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-06-sv-gprd.c.gitlab-production.internal     Oct 31 11:01:47 web-06-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-12-sv-gprd.c.gitlab-production.internal     Oct 31 10:38:13 web-12-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-12-sv-gprd.c.gitlab-production.internal     Oct 31 10:59:30 web-12-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-11-sv-gprd.c.gitlab-production.internal     Oct 31 10:38:14 web-11-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-11-sv-gprd.c.gitlab-production.internal     Oct 31 11:02:02 web-11-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-10-sv-gprd.c.gitlab-production.internal     Oct 31 10:38:16 web-10-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-10-sv-gprd.c.gitlab-production.internal     Oct 31 10:59:40 web-10-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-04-sv-gprd.c.gitlab-production.internal     Oct 31 10:38:32 web-04-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-04-sv-gprd.c.gitlab-production.internal     Oct 31 11:01:30 web-04-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-02-sv-gprd.c.gitlab-production.internal     Oct 31 10:39:23 web-02-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-02-sv-gprd.c.gitlab-production.internal     Oct 31 11:01:41 web-02-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-05-sv-gprd.c.gitlab-production.internal     Oct 31 10:38:33 web-05-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-05-sv-gprd.c.gitlab-production.internal     Oct 31 11:01:23 web-05-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-cny-02-sv-gprd.c.gitlab-production.internal Oct 31 10:59:55 web-cny-02-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-cny-01-sv-gprd.c.gitlab-production.internal Oct 31 07:24:10 web-cny-01-sv-gprd sudo:  takeoff : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-cny-01-sv-gprd.c.gitlab-production.internal Oct 31 07:25:27 web-cny-01-sv-gprd sudo:  takeoff : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-cny-01-sv-gprd.c.gitlab-production.internal Oct 31 10:38:47 web-cny-01-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-cny-01-sv-gprd.c.gitlab-production.internal Oct 31 11:01:51 web-cny-01-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-07-sv-gprd.c.gitlab-production.internal     Oct 31 10:39:25 web-07-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-07-sv-gprd.c.gitlab-production.internal     Oct 31 11:00:52 web-07-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-08-sv-gprd.c.gitlab-production.internal     Oct 31 10:39:38 web-08-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-08-sv-gprd.c.gitlab-production.internal     Oct 31 11:01:27 web-08-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-14-sv-gprd.c.gitlab-production.internal     Oct 31 10:39:28 web-14-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-14-sv-gprd.c.gitlab-production.internal     Oct 31 11:01:17 web-14-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-03-sv-gprd.c.gitlab-production.internal     Oct 31 10:38:30 web-03-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn
web-03-sv-gprd.c.gitlab-production.internal     Oct 31 11:00:48 web-03-sv-gprd sudo:     root : TTY=pts/0 ; PWD=/home/takeoff ; USER=root ; COMMAND=/usr/bin/gitlab-ctl hup unicorn