Use Feature Flags to allow disabling deduplication of jobs in certain queues
As we're marking workers as idempotent!
and automatically deduplicating them, we currently have no way back if we make a mistake when making a worker idempotent.
We could add feature flags based on the queue name, that are defaulted to "enabled", that way, if we see something going wrong, we can disable that duplication by disabling a feature flag using chatops.
For example:
/chatops run feature set deduplicate_authorized_projects false
I think the implementation could be as simple as:
diff --git a/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb b/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb
index 79bbb99752e..df889a2057a 100644
--- a/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb
+++ b/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb
@@ -67,7 +67,7 @@ module Gitlab
end
def droppable?
- idempotent? && duplicate?
+ idempotent? && duplicate? && Feature.enabled?("disable_#{qeueue_name}_deduplication")
end
private
Edited by Bob Van Landuyt