Fix metrics server not shutting down when Puma is stopped
What does this MR do and why?
Previously if a separate metrics server were configured and Puma were stopped, the metrics server would continue to run. When Puma is started again, it attempts to spawn a new metrics server, but this fails repeatedly because the port has already been taken.
To fix this, add an at_exit
handler that will shutdown the process
supervisor if Puma exists. This won't help the case if Puma is
abruptly killed, however.
Relates to omnibus-gitlab#8109 (closed)
How to set up and validate locally
- In an Omnibus GitLab instance, enable the metrics server (https://docs.gitlab.com/ee/administration/monitoring/prometheus/web_exporter.html):
puma['exporter_enabled'] = true
puma['exporter_address'] = "127.0.0.1"
puma['exporter_port'] = 8083
- Run
gitlab-ctl reconfigure
. - Run
gitlab-ctl restart puma
. - After Puma starts,
ps -ef | grep metrics-server
should show that the metrics server is running. - Run
gitlab-ctl stop puma
. -
ps -ef | grep metrics-server
should still be running. - Apply this patch in
/opt/gitlab/embedded/service/gitlab-rails/metrics_server/metrics_server.rb
. - Retry steps 4 to 6.
metrics-server
should be gone. -
/var/log/gitlab/gitlab-rails/application_json.log
should show messages:
{"severity":"INFO","time":"2023-12-12T20:41:56.921Z","message":"Puma process 449340 is exiting, shutting down metrics server..."}
{"severity":"INFO","time":"2023-12-12T20:41:56.925Z","message":"Puma process 449338 is exiting, shutting down metrics server..."}
{"severity":"INFO","time":"2023-12-12T20:41:56.925Z","message":"Puma process 449342 is exiting, shutting down metrics server..."}
{"severity":"INFO","time":"2023-12-12T20:41:56.925Z","message":"Puma process 449336 is exiting, shutting down metrics server..."}
{"severity":"INFO","time":"2023-12-12T20:41:57.321Z","message":"Puma process 449300 is exiting, shutting down metrics server..."}
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Stan Hu