Stale Puma metrics do not get cleaned
Summary
It seems that stale puma metrics are not getting cleaned after process restart.
It seems that is not being executed on new process start:
Gitlab::Cluster::LifecycleEvents.on_master_restart do
# The following is necessary to ensure stale Prometheus metrics don't
# accumulate over time. It needs to be done in this hook as opposed to
# inside an init script to ensure metrics files aren't deleted after new
# unicorn workers start after a SIGUSR2 is received.
prometheus_multiproc_dir = ENV['prometheus_multiproc_dir']
if prometheus_multiproc_dir
old_metrics = Dir[File.join(prometheus_multiproc_dir, '*.db')]
FileUtils.rm_rf(old_metrics)
end
end
Steps to reproduce
- Start the Puma process,
- See metrics in
/-/metrics
endpoint, - Restart puma process, see those old metrics are still there with the new ones,
What is the current bug behavior?
Old metrics are present after restart.
What is the expected correct behavior?
Old metrics are gone, only new are present after restart.
Edited by 🤖 GitLab Bot 🤖