Prevent BulkImportEntityWorker from being incorrecly deduplicated
What does this MR do and why?
To fix Fix Direct Transfer stages execution order (!132320 - merged), I changed how BulkImports::EntityWorker initiates the execution of stages and changed the worker deduplicate strategy from until_executing
to until_executed
.
Recently I noticed that some workers were getting incorrectly deduplicated, causing the worker to stop being re-enqueued and causing the migration to get stuck. To fix the problem, we changed the worker to use the option deduplicate :until_executed, if_deduplicated: :reschedule_once
!135786 (merged). However, even with reschedule_once
, workers occasionally aren't rescheduled causing the migration to get stuck.
For example, on Kibana we can see that not all workers that were deduplicated were rescheduled
To fix the problem, this MR changes the deduplicate
strategy back to until_executing
and uses lease guard to prevent the same code from being executed twice
Related to: #432813 (closed)
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.