Skip to content

Disable Prometheus metrics if initialization fails

Previously if the underlying filesystem ran of space, reads and writes to mmap() regions would throw an ugly SIGBUS error and crash.

With prometheus-client-mmap v0.10.0 (https://gitlab.com/gitlab-org/prometheus-client-mmap/merge_requests/43), Prometheus::Client.reinitialize_on_pid_change will now throw an IOError if initialization fails for some reason. If this happens, we disable internal Prometheus metrics to ensure the system stays up.

Closes #24425 (closed)

How to test

  1. Create a small RAM disk. On Linux: sudo mount -t tmpfs -o size=2M tmpfs /mnt/ramdisk
  2. Create a symlink from /run/gitlab/unicorn -> /mnt/ramdisk
  3. Fill up the disk with empty files.
  4. Restart Unicorn.

You should see this error in Sentry: https://sentry.gitlab.net/gitlab/gitlab-geo-internal-develop-is/issues/1163197/

Edited by Stan Hu

Merge request reports

Loading