Skip to content

Combine enqueued_jobs and sidekiq_queueing SLI in Sidekiq

In gitlab-com/runbooks!7074 (merged), we added a new enqueued_jobs SLI to replace the gitlab_background_jobs:queue aggregation sets as part of the Mimir migration.

enqueued_jobs is recording from sidekiq_enqueued_jobs_total source metric which is a client-side metrics to get the rate of enqueuing jobs for Sidekiq clients (api, web, git, etc). This SLI only has a request rate component.

We also have sidekiq_queueing application SLI which measures the apdex for duration of jobs waiting in the queue on the server-side. This SLI only has the apdex component.

Quoting from gitlab-com/runbooks!7074 (comment 1864552567), we could try combining them into 1 SLI, which the components would look something like:

opsRate: sidekiq_enqueued_jobs_total
successRate: gitlab_sli_sidekiq_queueing_apdex_total
apdexSuccessRate: gitlab_sli_sidekiq_queueing_apdex_success_total
apdexWeight: gitlab_sli_sidekiq_queueing_apdex_total

The errorRatio could then be calculated using (opsRate - successRate) / opsRate, similar to how we allow calculating the apdex-ratio from an errorCounterApdex.

We'd be needing a shard label on the source metric that matches the shard for which we're enqueuing a job. This is currently not present on the source metrics src.