Skip to content

Limit the number of project loaded by UpdateAllMirrorsWorker

Summary

We load all projects in UpdateAllMirrorsWorker#schedule_mirrors! and then we select all of them that have Project#mirror? return true.

After enabling the feature availability check, a lot of projects piled up at the beginning of the result needed to be skipped from the Rails side, leading to a huge delay in the processing of licensed projects.

I'm thinking of two solutions for this:

  1. Update the query to include the plan check:
left_outer_joins(namespace: :plan).where("(projects.visibility_level = 20 OR plans.name IN ('early_adopter', 'bronze', 'silver', 'gold'))")

This doesn't work: #196862 (closed)

  1. Instead of lazy loading the projects we could load them all and mark_as_failed the ones without a license. After 15 retries they will be dropped from the query.
Edited by 🤖 GitLab Bot 🤖