Combine enqueued_jobs and sidekiq_queueing SLI in Sidekiq
In gitlab-com/runbooks!7074 (merged), we added a new enqueued_jobs
SLI to replace the gitlab_background_jobs:queue
aggregation sets as part of the Mimir migration.
enqueued_jobs
is recording from sidekiq_enqueued_jobs_total
source metric which is a client-side metrics to get the rate of enqueuing jobs for Sidekiq clients (api, web, git, etc). This SLI only has a request rate component.
We also have sidekiq_queueing
application SLI which measures the apdex for duration of jobs waiting in the queue on the server-side. This SLI only has the apdex component.
Quoting from gitlab-com/runbooks!7074 (comment 1864552567), we could try combining them into 1 SLI, which the components would look something like:
opsRate: sidekiq_enqueued_jobs_total
successRate: gitlab_sli_sidekiq_queueing_apdex_total
apdexSuccessRate: gitlab_sli_sidekiq_queueing_apdex_success_total
apdexWeight: gitlab_sli_sidekiq_queueing_apdex_total
The errorRatio
could then be calculated using (opsRate - successRate) / opsRate
, similar to how we allow calculating the apdex-ratio from an errorCounterApdex
.
We'd be needing a shard
label on the source metric that matches the shard for which we're enqueuing a job. This is currently not present on the source metrics src.