Add GVL wait time metrics to Sidekiq structured logs
What does this MR do and why?
Move GVL instrumentation from ServerMetrics into
InstrumentationLogger so that gvl_thread_wait_s and
gvl_process_wait_s flow into the structured job completion logs via
the standard instrumentation pipeline.
ServerMetrics now reads GVL values from the instrumentation hash for
Prometheus, consistent with how it consumes gitaly/redis/elasticsearch
metrics.
References
See gitlab-com/gl-infra/tenant-scale/tenant-services/team#384 (closed)
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
-
In
gdk.yml, make sure you have the prometheus and the sidekiq exporter enabled (rungdk reconfigure && gdk.restartafter making these changes):# gdk.yml prometheus: enabled: true gitlab: lefthook_enabled: false rails_background_jobs: sidekiq_exporter_enabled: trueIf you are using
gdk.testas your GDK host, editconfig/gitlab.ymlto uselocalhostinstead (andgdk restart rails-background-jobsafterwards):sidekiq_exporter: enabled: true - address: gdk.test + address: localhost port: 3807 -
In rails console enable the feature flag
Feature.enable(:enable_sidekiq_gvl_metrics) -
Tail sidekiq
tail -f log/sidekiq.log | grep gvlSee the results streaming in.
-
Visit http://localhost:3807/metrics and observe the GVL metrics there
-
Next, disable the flag
Feature.disable(:enable_sidekiq_gvl_metrics) -
Observe that when tailing, the results eventually stop showing up in
tail -f log/sidekiq.log | grep gvl
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.