Skip to content

Fix Direct Transfer stages execution order

Rodrigo Tomonari requested to merge rodrigo/fix-bulk-import-stage-execution-order into master

What does this MR do and why?

Fix Direct Transfer stages execution order.

Direct Transfer is starting stages before completing the previous ones, causing unexpected errors since the sequence of the stages must be followed.

After investigating, I have concluded that certain aspects of the EntityWorker worker's implementation contribute to the issue.

The first problem is that EntityWorker worker was enqueued multiple times with the same arguments when PipelineWorker and FinishBatchedPipelineWorker were finished, and Redis wasn't managing to deduplicate similar workers causing multiple stages to be started and pipelines to be executed more than one time.

The second problem was that EntityWorker was enqueued with the current_stage argument, which was then used to check if the stage was running. The problem was that if an EntityWorker worker were enqueued with current_stage different from the one running, a follow-up stage would be started even if one stage was running.

For example, if stage 2 was running, executing EntityWorker using current_stage = 1, would start stage 3.

So, this MR fixes the above problems by no longer enqueueing EntityWorker with the current_stage argument and only starts a new stage if no other stage is running. Also, update PipelineWorker and FinishBatchedPipelineWorker not to enqueue EntityWorker and instead make EntityWorker to enqueue itself.

And because in-flight migrations won't have an EntityWorker running, a version attribute was added to PipelineWorker and FinishBatchedPipelineWorker. In case the version isn't present, an EntityWorker will be initiated.

Related to: #425716 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Enable Direct Transfer following these steps
  2. Perform a Direct Transfer migration
  3. verify if the stages are executed in order during the migration by checking the bulk_import_trackers table.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports