Skip to content

Use Feature Flags to allow disabling deduplication of jobs in certain queues

As we're marking workers as idempotent! and automatically deduplicating them, we currently have no way back if we make a mistake when making a worker idempotent.

We could add feature flags based on the queue name, that are defaulted to "enabled", that way, if we see something going wrong, we can disable that duplication by disabling a feature flag using chatops.

For example:

/chatops run feature set deduplicate_authorized_projects false

I think the implementation could be as simple as:

diff --git a/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb b/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb
index 79bbb99752e..df889a2057a 100644
--- a/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb
+++ b/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb
@@ -67,7 +67,7 @@ module Gitlab
         end
 
         def droppable?
-          idempotent? && duplicate?
+          idempotent? && duplicate? && Feature.enabled?("disable_#{qeueue_name}_deduplication")
         end
 
         private
Edited by Bob Van Landuyt