Update Migration::EnqueuerWorker capacity logic
🍃 Context
We are preparing to migrate all container repositories to the new container registry. Rails is responsible for driving this migration by using an EnqueuerWorker
to find the next qualified container repository to import and making an API request to the registry to begin the import.
We have a variety of feature flags and settings to allow us to control the speed of the imports. One thing we control is the "capacity", or the number of imports we allow to be in progress at one time. For example, we can set the capacity to 10
. If an EnqueuerWorker job is executing, and sees there are already 10 imports in progress, it will not schedule any more jobs. If there are less than 10 imports in progress, the EnqueuerWorker will schedule another job along the way.
The EnqueuerWorker checks the capacity twice:
- At the beginning of
#perform
to see if it should even proceed with kicking off an import. - At the end of
#perform
to see if it should re-enqueue another Enqueuer job.
In testing, we realized that in the time between (1.) and (2.), the current_capacity
(imports in progress) can change. This led us to realizing we were incorrectly memoizing the current_capacity
and along the way, we noticed that we had inconsistent capacity checks, using <
in one place, and <=
in another place. This is what #357094 (closed) describes.
💭 What does this MR do and why?
This MR removes the strong_memoize
of current_capacity
and updates so all capacity checks correctly use current <= maximum_capacity
.
The import functionality is all behind a feature flag and we will be specifically testing this behavior on staging for the next few weeks.
🖼 Screenshots or screen recordings
See below.
💻 How to set up and validate locally
Trying to change the current_capacity
while the job is running locally is not easy. The best way I've found to do this is with pry
, inserting binding.pry
in the #below_capacity?
method, and then creating additional importing container repositories in the middle of a test:
FactoryBot.create(:container_repository, :importing)
Using any of the tests that use the 're-enqueuing based on capacity'
shared example will test the capacity:
bin/rspec ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:77
bin/rspec ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:149
bin/rspec ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:205
📏 MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #357094 (closed)