Skip to content

Prevent BulkImportEntityWorker from being incorrecly deduplicated

What does this MR do and why?

To fix Fix Direct Transfer stages execution order (!132320 - merged), I changed how BulkImports::EntityWorker initiates the execution of stages and changed the worker deduplicate strategy from until_executing to until_executed.

Recently I noticed that some workers were getting incorrectly deduplicated, causing the worker to stop being re-enqueued and causing the migration to get stuck. To fix the problem, we changed the worker to use the option deduplicate :until_executed, if_deduplicated: :reschedule_once !135786 (merged). However, even with reschedule_once, workers occasionally aren't rescheduled causing the migration to get stuck.

For example, on Kibana we can see that not all workers that were deduplicated were rescheduled

Screenshot_2023-11-23_at_15.36.23

To fix the problem, this MR changes the deduplicate strategy back to until_executing and uses lease guard to prevent the same code from being executed twice

Related to: #432813 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports