Skip to content

Fix Enqueuer capacity conflict with deduplication

Problem

In !83091 (merged), we updated the Enqueuer from deduplicate :until_executing to deduplicate :until_executed.

The EnqueuerWorker does not take any args, so with this new deduplication setting, we have in essence nullified our ability to have more than one import happening at a time. Our feature flags controlling capacity no longer have an affect, and anytime the Enqueuer re-enqueues to reach capacity, those jobs are instantly deduplicated

We can see this happening in the logs.

Solution

We need a way to have multiple jobs in the queue but ensure that the same container repository does not get queued twice. We don't know which container repository we will be using until the worker starts, so we need an alternative method to make the jobs not look like duplicates.

A few ideas are raised:

  1. Use an exclusive lease on the #perform method
  2. Pass a random unique arg to the worker (perhaps a UUID?) so that they are not deduplicated. This essentially overrides the deduplication, but that is what we want here. We want to control the capacity within the worker itself.
Edited by Steve Abrams