Analyze performance of Prometheus metrics unicorn scraping
Activating Prometheus metrics seems to take at most 10 seconds according to https://prometheus.gitlab.com/graph?g0.range_input=1h&g0.expr=avg(up%7Bjob%3D%22gitlab-unicorn%22%2C+environment%3D%22prd%22%7D)+without+(fqdn%2Cinstance)&g0.tab=0&g1.range_input=1h&g1.expr=avg(scrape_duration_seconds%7Bjob%3D%22gitlab-unicorn%22%2Cenvironment%3D%22prd%22%7D)+without+(fqdn%2Cinstance)&g1.tab=0
:
In the Rails console, I just ran:
MetricsService.new.metrics_text
This appears to take several seconds and according to strace, spawns a lot of threads.
https://files.slack.com/files-pri/T02592416-F7T87E6C8/download/strace-prom-perf.zip contains an strace of the activity with verbose timings.
/cc: @pchojnacki, @bjk-gitlab, @pcarranza