Don't retry RelationBatchExportWorker when batch doesn't exist

Sam Wordrequested to merge
583390-fix-excessive-errors-on-simultaneous-export into master

What does this MR do and why?

This MR updates RelationBatchExportWorker to return without raising an error when a relation export batch has been deleted before RelationBatchExportWorker could process it. If a user is rapidly creating direct transfer migrations using the same source entity, the source instance will restart a batched relation export by first deleting all of that entity export's batches, even if the batched relation is still processing from an earlier migration from the same user. If this happens, the source instance receives a ActiveRecord::RecordNotFound error for every deleted export batch times the number of worker retries, even though exports generally process just fine.

This scenario doesn't occur often as most users aren't repeatedly exporting the same source entity, but can happen depending on the users' use case. Because this scenario isn't a typical use case, I was hesitant to skip destroying existing relations or skip restarting the relation export in case some users intentionally restart a long-running migration for any reason.

This MR also logs when a batched export relation is restarted and in-progress batches are deleted for easier debugging in the event a more niche race condition occurs.

How to set up and validate locally

This race condition is difficult to reproduce in the GDK, see https://gitlab.com/gitlab-org/gitlab/-/issues/583390#note_3025055165 to reproduce it more reliably

  1. Begin a direct transfer migration using the same source entity at least twice in quick succession
  2. Verify sidekiq logs aren't flooded with ActiveRecord::RecordNotFound errors
  3. Verify export logs have a warning when a batched relation export has been restarted
  4. Verify all migrations finished successfully and entities were imported correctly

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #583390

Edited by Sam Word

Merge request reports