Skip to content

Disable Prometheus metrics if initialization fails

Stan Hu requested to merge sh-disable-prom-metrics-on-failure into master

Previously if the underlying filesystem ran of space, reads and writes to mmap() regions would throw an ugly SIGBUS error and crash.

With prometheus-client-mmap v0.10.0 (https://gitlab.com/gitlab-org/prometheus-client-mmap/merge_requests/43), Prometheus::Client.reinitialize_on_pid_change will now throw an IOError if initialization fails for some reason. If this happens, we disable internal Prometheus metrics to ensure the system stays up.

Closes #24425 (closed)

How to test

  1. Create a small RAM disk. On Linux: sudo mount -t tmpfs -o size=2M tmpfs /mnt/ramdisk
  2. Create a symlink from /run/gitlab/unicorn -> /mnt/ramdisk
  3. Fill up the disk with empty files.
  4. Restart Unicorn.

You should see this error in Sentry: https://sentry.gitlab.net/gitlab/gitlab-geo-internal-develop-is/issues/1163197/

Edited by Stan Hu

Merge request reports