Skip to content

Update the Enqueuer deduplication options

🍌 Context

We're currently implementing a data migration on the Container Registry. This migration is going to be driven by the rails backend. For all the nitty-gritty details, see &7316 (comment 897867569).

At the core of the rails logic lies the Enqueuer worker. Its job is to find the next image repository and start the migration on the Container Registry side.

Because this background job can be enqueued by multiple sources (itself, cron schedule or when a migration hits a final state), we put deduplication in place.

During the executions on gitlab.com, we observed that sometimes the deduplication key is created but the job is never started. The key is left behind and for the next 6 hours, all the Enqueuer executions are deduplicated = the whole migration is paused.

That's not great and 6 hours is way too much for this job. This MR updates the deduplication config.

We still don't know how a deduplication key can be left behind without any job running.

What does this MR do and why?

  • Update the ContainerRegistry::Migration::EnqueuerWorker deduplication settings.
  • Update the relevant spec.

No changelog as all the migration pieces are behind feature flag that are enabled on gitlab.com only for now.

🖥 Screenshots or screen recordings

n / a

How to set up and validate locally

I don't think there is a way to test this locally properly.

🚥 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Fernandez

Merge request reports