Pull Jemalloc stats report from Production for Puma and Sidekiq

Details

SRE Support Needed

Context

  • Recently we introduced the ability to dump Jemalloc Stats from Ruby process: issue, howto
  • We want to dump these reports automatically, based on a timer in production: issue, PoC
  • We need to estimate the size of a single report and the time it takes to pull one in a real production environment.

What we need

  1. Execute bundle exec rbtrace -p <PID> -e 'Benchmark.bmbm { |x| x.report { Gitlab::Memory::Jemalloc.dump_stats(path: "/tmp", format: :json) } }' (examples) for multiple processes (see the list below)
  2. Note the output (the time it took to run the report) and post it here (expectation << 1s each run)
  3. Fetch the dump files and attach them to the issue (expectation < 1MB each file)

We need it for:

  1. 2 arbitrary Puma workers
  2. 2 arbitrary Sidekiq workers

We expect to do it via the rbtrace. I am not sure if ENV['ENABLE_RBTRACE'] is set on our prod envs though.

Edited by Aleksei Lipniagov