Change Enqueuer query to return multiple IDs
Problem
We are reaching a limit in the number of container repositories we are able to import at one time. Looking at a given job with loop mode enabled, we see we are able to start imports during the entire 250s that we allow before timeout. In the linked example, 150 imports started. Notice that they all start in sequence. This means it take about 250/150 = 1.67s
to find the next repository and make the request to start it. This is a limitation of the SQL queries and API request, meaning if we are starting these in sequence, we will never be able to start more than 60/1.67*60 => ~2100 imports per hour.
Solution
To start 150 imports, we need to execute 150 ContainerRepository.ready_for_import.next
queries and then 150 API requests to the container registry.
We could execute less queries by returning more than one container repository, essentially change from LIMIT 1
to LIMIT 10
or even a dynamic limit based on the capacity setting. This means we only have 1 query to start 10 imports rather than 10.