Add counter metric for deferred jobs
What does this MR do and why?
Related to #472976 (closed)
This MR adds a counter metric for every job that is deferred by the concurrency limit middleware
The MR also updates documentation and adds info about the available counters to the concurrency limit middleware documentation to help developers to discover them
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
N/A
How to set up and validate locally
patch for testing
diff --git a/ee/app/workers/zoekt/indexer_worker.rb b/ee/app/workers/zoekt/indexer_worker.rb
index 70bbeacd7d16..8fc3b42713eb 100644
--- a/ee/app/workers/zoekt/indexer_worker.rb
+++ b/ee/app/workers/zoekt/indexer_worker.rb
@@ -20,9 +20,10 @@ class IndexerWorker
sidekiq_options retry: 2
idempotent!
pause_control :zoekt
- concurrency_limit -> { 30 if Feature.enabled?(:zoekt_limit_indexing_concurrency) }
+ concurrency_limit -> { 1 }
def perform(project_id, options = {})
+ sleep(30)
return unless ::Gitlab::CurrentSettings.zoekt_indexing_enabled?
return unless ::License.feature_available?(:zoekt_code_search)
- enable sidekiq metrics in gdk
gdk config set gitlab.rails_background_jobs.sidekiq_exporter_enabled true
gdk reconfigure- apply the patch above to manually set this worker very low, and add a sleep timer to cause the worker queue to back up
- in rails console, queue the worker for every project in your database
Project.all.each { |p| Zoekt::IndexerWorker.perform_async(p.id) } - load the metrics page for sidekiq metrics: http://gdk.test:3807/metrics
- ensure the values are updated for the metric
note: you might need to gdk restart rails-background-jobs to make the metrics show up
# HELP sidekiq_concurrency_limit_deferred_jobs_total Multiprocess metric
# TYPE sidekiq_concurrency_limit_deferred_jobs_total counter
sidekiq_concurrency_limit_deferred_jobs_total{worker="Zoekt::IndexerWorker"} 64
# HELP sidekiq_concurrency_limit_max_concurrent_jobs Multiprocess metric
# TYPE sidekiq_concurrency_limit_max_concurrent_jobs gauge
sidekiq_concurrency_limit_max_concurrent_jobs{worker="ElasticCommitIndexerWorker",pid="sidekiq_0"} 30
sidekiq_concurrency_limit_max_concurrent_jobs{worker="PackageMetadata::GlobalAdvisoryScanWorker",pid="sidekiq_0"} 10
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Search::Zoekt::DeleteProjectWorker",pid="sidekiq_0"} 100
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Security::ScanExecutionPolicies::CreatePipelineWorker",pid="sidekiq_0"} 10000
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Security::ScanResultPolicies::AddApproversToRulesWorker",pid="sidekiq_0"} 200
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Zoekt::IndexerWorker",pid="sidekiq_0"} 1
# HELP sidekiq_concurrency_limit_queue_jobs Multiprocess metric
# TYPE sidekiq_concurrency_limit_queue_jobs gauge
sidekiq_concurrency_limit_queue_jobs{worker="ElasticCommitIndexerWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="PackageMetadata::GlobalAdvisoryScanWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="Search::Zoekt::DeleteProjectWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="Security::ScanExecutionPolicies::CreatePipelineWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="Security::ScanResultPolicies::AddApproversToRulesWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="Zoekt::IndexerWorker"} 24Merge request reports
Activity
assigned to @terrichu
added pipelinetier-1 label
- A deleted user
added backend development guidelines docsimprovement documentation maintenancerefactor labels and removed maintenanceperformance label
1 Warning 
The master pipeline status page reported failures in If these jobs fail in your merge request with the same errors, then they are not caused by your changes.
Please check for any on-going incidents in the incident issue tracker or in the#master-brokenSlack channel.2 Messages 
This MR contains docs in the /doc/development directory, but any Maintainer (other than the author) can merge. You do not need tech writer review. This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
-
doc/administration/monitoring/prometheus/gitlab_metrics.md(Link to current live version)
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.mdfiles that you've changed. The first few lines of each*.mdfile identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
Reviewer roulette
Category Reviewer Maintainer backend @aluthra2(UTC+5.5, 9.5 hours ahead of author)
@j.seto(UTC+0, 4 hours ahead of author)
Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.
If needed, you can retry the
danger-reviewjob that generated this comment.Generated by
Edited by Ghost User-
added 1 commit
- 3bda1c3f - Add counter metric for deferred jobs by concurrency limit
changed milestone to %17.3
Resolved by Rémy Coutable- Last reply by Surabhi Suman
requested review from @ghavenga
added pipeline:mr-approved label
added pipelinetier-2 label and removed pipelinetier-1 label
Before you set this MR to auto-merge
This merge request will progress on pipeline tiers until it reaches the last tier: pipelinetier-3. We will trigger a new pipeline for each transition to a higher tier.
Before you set this MR to auto-merge, please check the following:
- You are the last maintainer of this merge request
- The latest pipeline for this merge request is pipelinetier-3 (You can find which tier it is in the pipeline name)
- This pipeline is recent enough (created in the last 8 hours)
If all the criteria above apply, please set auto-merge for this merge request.
See pipeline tiers and merging a merge request for more details.
requested review from @rymai
- Resolved by Rémy Coutable
@rymai could you please do the maintainer review?
- Last reply by Rémy Coutable
E2E Test Result Summary
allure-report-publishergenerated test report!e2e-test-on-gdk:
expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Plan | 67 | 0 | 2 | 0 | 69 | ✅ | | Data Stores | 30 | 0 | 1 | 0 | 31 | ✅ | | Create | 127 | 0 | 12 | 0 | 139 | ✅ | | Verify | 43 | 0 | 2 | 0 | 45 | ✅ | | Govern | 69 | 0 | 0 | 0 | 69 | ✅ | | Analytics | 2 | 0 | 0 | 0 | 2 | ✅ | | Package | 19 | 0 | 12 | 0 | 31 | ✅ | | Fulfillment | 1 | 0 | 0 | 0 | 1 | ✅ | | Monitor | 8 | 0 | 0 | 0 | 8 | ✅ | | Release | 5 | 0 | 0 | 0 | 5 | ✅ | | Secure | 1 | 0 | 0 | 0 | 1 | ✅ | | Manage | 0 | 0 | 1 | 0 | 1 | ➖ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 372 | 0 | 30 | 0 | 402 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+e2e-package-and-test:
expand test summary
+---------------------------------------------------------------------+ | suites summary | +----------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +----------------+--------+--------+---------+-------+-------+--------+ | Govern | 105 | 0 | 8 | 2 | 113 | ✅ | | Create | 417 | 0 | 51 | 27 | 468 | ✅ | | Verify | 51 | 0 | 15 | 2 | 66 | ✅ | | Manage | 30 | 0 | 15 | 2 | 45 | ✅ | | Data Stores | 46 | 0 | 11 | 0 | 57 | ✅ | | Systems | 7 | 0 | 0 | 0 | 7 | ✅ | | GitLab Metrics | 2 | 0 | 1 | 0 | 3 | ✅ | | Analytics | 3 | 0 | 0 | 0 | 3 | ✅ | | Fulfillment | 4 | 0 | 7 | 0 | 11 | ✅ | | Package | 31 | 0 | 14 | 6 | 45 | ✅ | | Plan | 80 | 0 | 11 | 0 | 91 | ✅ | | Configure | 1 | 0 | 3 | 0 | 4 | ✅ | | Monitor | 12 | 0 | 13 | 0 | 25 | ✅ | | Ai-powered | 1 | 0 | 2 | 0 | 3 | ✅ | | Secure | 4 | 0 | 2 | 0 | 6 | ✅ | | Release | 5 | 0 | 1 | 0 | 6 | ✅ | | ModelOps | 0 | 0 | 1 | 0 | 1 | ➖ | | Growth | 0 | 0 | 2 | 0 | 2 | ➖ | +----------------+--------+--------+---------+-------+-------+--------+ | Total | 799 | 0 | 157 | 39 | 956 | ✅ | +----------------+--------+--------+---------+-------+-------+--------+Edited by Ghost User
