When a worker runs for the first time, it will have sidekiq_jobs_completion_seconds_count set to 1 (because it was just completed). But this is wrong in Prometheus terms, because there was no previous value of 0, and so the rate of change will also be 0.
To work around this, we initialise this bucket for all workers that can be run by the current Sidekiq process before processing the first job. This way we can calculate rates correctly for infrequent workers.
bin/sidekiq-cluster --queue-selector 'feature_category=source_code_management'
We can then force the TrendingProjectsWorker - which normally runs once per day - to run:
bundle exec rails r 'TrendingProjectsWorker.perform_async'
With the feature flag in this MR disabled (
sidekiq_job_completion_metric_initialize), we'll see this in our Prometheus metrics. Note how it goes straight to 1 without passing through 0:
With the feature flag enabled (I also deleted my Prometheus data volume to be sure this change worked):
Also note that the metrics are initialised for all otherworkers, but not all workers that are possible.