Rails metrics cardinality review
We should carry out a review of the metrics GitLab-Rails emits and consider whether there are optimisations we can make.
The gitlab-rails job on the web service emits by far the most application metrics: each instance returns 40k metrics on every scrape.
Of these, the following four metrics dominate, generating about 60% of the metrics:
gitlab_sql_duration_seconds_bucketgitlab_transaction_duration_seconds_bucketgitlab_transaction_allocated_memory_bytes_bucketgitlab_transaction_cputime_seconds_bucket
Reducing the volume of metrics would have many positive impacts on GitLab:
- Less compute time spent in instrumentation
- Complications from GIL lockups etc minimized
- A more stable observability platform

