Fix Prometheus Unicorn metrics not coming back after a HUP
In https://gitlab.com/gitlab-com/infrastructure/issues/1962, we saw that HTTP metrics would often not return after a HUP. This is what was happening:
- User runs
gitlab-ctl hup unicorn
-
gitlab-unicorn-wrapper
sendsSIGUSR2
tounicorn
, which tells it to spawn a new master - When the new master is running with the right number of worker processes,
gitlab-unicorn-wrapper
sends aSIGQUIT
to the old master -
gitlab-unicorn-wrapper
script quits, causingsv
to restart it - The sv
run
script deletes the metrics files. - The new
unicorn
processes are accessing deleted files
To avoid this problem, we remove the Prometheus .db
files before executing
the unicorn
process in the before_exec
block.
Edited by GitLab Release Tools Bot