Disable Prometheus metrics if initialization fails
Previously if the underlying filesystem ran of space, reads and writes to mmap()
regions would throw an ugly SIGBUS error and crash.
With prometheus-client-mmap v0.10.0 (https://gitlab.com/gitlab-org/prometheus-client-mmap/merge_requests/43), Prometheus::Client.reinitialize_on_pid_change
will now throw an IOError if initialization fails for some reason. If this happens, we disable internal Prometheus metrics to ensure the system stays up.
Closes #24425 (closed)
How to test
- Create a small RAM disk. On Linux:
sudo mount -t tmpfs -o size=2M tmpfs /mnt/ramdisk
- Create a symlink from
/run/gitlab/unicorn
->/mnt/ramdisk
- Fill up the disk with empty files.
- Restart Unicorn.
You should see this error in Sentry: https://sentry.gitlab.net/gitlab/gitlab-geo-internal-develop-is/issues/1163197/
Edited by Stan Hu