Add telemetry for Sidekiq resource usage limits and throttling

Possible telemetry required based on #3818 (moved) and https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/3775#note_2089502716:

  1. Concurrency limits per worker (or other limits we are using) - these values can change and ease of tracking is important
  2. Job buffer size: this helps to estimate the backlog of work
  3. Worker concurrency: to understand efficacy of throttling
  4. measurement values (i.e. what we use to dictate throttling) this may not be important since we have logs and metrics for db duration and marginalia active count dashboards
  5. Counts of limit being exceeded
  6. Throttling state (disabled/enabled), reason for throttling (which resource exceeded).

Actionable tasks

Closing summary

We have improved telemetry in the application rate limiter and concurrency limit middleware which forms the foundation for understanding the impacts of worker throttling.

Work for throttling metrics can be tracked alongside throttling work in Implement Sidekiq throttling middleware in gitl... (#3815 - closed).

Edited by Sylvester Chin