Skip to content

Defer ConnectionPool instrumentation setup

Stan Hu requested to merge sh-fix-ruby-metrics-16-8 into 16-8-stable-ee

What does this MR do and why?

This backports !142042 (merged) to 16-8-stable-ee.

While looking at puma worker metrics, some of them appear to be missing. In particular gauges from RubySampler, e.g. ruby_file_descriptors. We only have values for the pid puma_master, none for puma_0, puma_1, etc.

The corresponding prometheus mmap file prefix is gauge_all. It appears we are not properly resetting MmapedValue after forking the puma worker. Thus, workers continue writing to the mmapped file belonging to the puma_master, and likely competing / continuously overwriting each other.

Starting with Ruby 3.1, connection_pool performs logic directly on fork.

This is triggered by puma here:

If we instrument before the fork, then instrumentation logic will get called on fork, which causes metrics to be re-initialized with the wrong pid.

Metrics continue to be attributed to the puma_master, because we do not yet have the process name set:

This process name is needed for proper attribution by PidProvider.

By deferring the instrumentation, we ensure no prometheus metrics are touched until process name is set, and we can explicitly reinitialize via reinitialize_on_pid_change.

This allows the gauge metrics to be correctly attributed to puma_1, puma_2, etc.

Changelog: fixed

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

  • This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
  • The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
  • This MR has a severity label assigned (if applicable).
  • Set the milestone of the merge request to match the target backport branch version.
  • This MR has been approved by a maintainer (only one approval is required).
  • Ensure the e2e:package-and-test-ee job has either succeeded or been approved by a Software Engineer in Test.

Note to the merge request author and maintainer

If you have questions about the patch release process, please:

Edited by Stan Hu

Merge request reports