Use sidekiq_interruptions_exhausted in Direct Transfer jobs

Rodrigo Tomonarirequested to merge
rodrigo/handle-interrupted-jobs-gracefully-direct-transfer into master

What does this MR do and why?

Updates Direct Transfer jobs to mark migrations as failed when jobs are interrupted multiple times and sent to the interrupted queue, preventing the migration progress from getting stuck.

Note: In the context of !169181 (merged), a new callback was introduced to the Sidekiq Reliable Fetcher gem. This callback is executed when jobs are completely marked as interrupted.

Related to: #498318 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

BulkImports::RelationExportWorker

  1. Apply the following patch to make BulkImports::RelationExportWorker to take longer to export labels and reduce max_retries_after_interruption to 1.
Patch BulkImports::RelationExportWorker 
diff --git a/app/services/bulk_imports/relation_export_service.rb b/app/services/bulk_imports/relation_export_service.rb
index 337f245cf27f..93667cf4be62 100644
--- a/app/services/bulk_imports/relation_export_service.rb
+++ b/app/services/bulk_imports/relation_export_service.rb
@@ -45,6 +45,8 @@ def find_or_create_export!
     end
 
     def export_service
+      sleep 300 if relation.to_s == 'labels'
+
       @export_service ||= if config.tree_relation?(relation) || config.self_relation?(relation)
                             TreeExportService.new(portable, export_path, relation, user)
                           elsif config.file_relation?(relation)
diff --git a/app/workers/bulk_imports/relation_export_worker.rb b/app/workers/bulk_imports/relation_export_worker.rb
index 404dfa2ff097..e707273d3117 100644
--- a/app/workers/bulk_imports/relation_export_worker.rb
+++ b/app/workers/bulk_imports/relation_export_worker.rb
@@ -13,6 +13,7 @@ class RelationExportWorker
     feature_category :importers
     sidekiq_options status_expiration: StuckExportJobsWorker::EXPORT_JOBS_EXPIRATION, retry: 6
     worker_resource_boundary :memory
+    sidekiq_options max_retries_after_interruption: 1
 
     sidekiq_retries_exhausted do |job, exception|
       perform_failure(job, exception)
  1. Trigger a group/project export via API
curl --location 'http://gdk.test:3000/api/v4/groups/24/export_relations' \
--header 'Content-Type: application/json' \
--header 'Authorization: <ACCESS_TOKEN>' \
--data '{
    "batched": false
}'

  1. Wait for Sidekiq jobs to start

  2. Access the relation status API: http://gdk.test:3000/api/v4/groups/24/export_relations/status?relation=labels. The relation status should be zero and the error should be empty

  3. Stop Sidekiq using gdk stop rails-background-jobs

  4. Access the relation status API again: http://gdk.test:3000/api/v4/groups/24/export_relations/status?relation=labels. The relation status should be -1, and the error should be populated.

BulkImports::RelationBatchExportWorker

  1. Apply the following patch to make BulkImports::RelationBatchExportWorker to take longer to export labels and reduce max_retries_after_interruption to 1.
Patch BulkImports::RelationBatchExportWorker 
diff --git a/app/services/bulk_imports/relation_batch_export_service.rb b/app/services/bulk_imports/relation_batch_export_service.rb
index 3f98d49aa1b1..daba6e16d0dd 100644
--- a/app/services/bulk_imports/relation_batch_export_service.rb
+++ b/app/services/bulk_imports/relation_batch_export_service.rb
@@ -33,6 +33,8 @@ def execute
     delegate :exported_filename, :exported_objects_count, to: :export_service
 
     def export_service
+      sleep 300 if relation.to_s == 'labels'
+
       @export_service ||= config.export_service_for(relation).new(portable, export_path, relation, user)
     end
 
diff --git a/app/workers/bulk_imports/relation_batch_export_worker.rb b/app/workers/bulk_imports/relation_batch_export_worker.rb
index 7d6f44860b0b..c70d4e4f3677 100644
--- a/app/workers/bulk_imports/relation_batch_export_worker.rb
+++ b/app/workers/bulk_imports/relation_batch_export_worker.rb
@@ -10,6 +10,7 @@ class RelationBatchExportWorker
     feature_category :importers
     sidekiq_options status_expiration: StuckExportJobsWorker::EXPORT_JOBS_EXPIRATION, retry: 6
     worker_resource_boundary :memory
+    sidekiq_options max_retries_after_interruption: 1
 
     sidekiq_retries_exhausted do |job, exception|
       perform_failure(job, exception)
  1. Trigger a group/project export via API using the batched mode
curl --location 'http://gdk.test:3000/api/v4/groups/24/export_relations' \
--header 'Content-Type: application/json' \
--header 'Authorization: <ACCESS_TOKEN>' \
--data '{
    "batched": true
}'

  1. Wait for Sidekiq jobs to start

  2. Access the relation status API: http://gdk.test:3000/api/v4/groups/24/export_relations/status?relation=labels. The relation batch status should be zero, and the error should be empty.

  3. Stop Sidekiq using gdk stop rails-background-jobs

  4. Access the relation status API again: http://gdk.test:3000/api/v4/groups/24/export_relations/status?relation=labels. The relation batch status should be -1, and the error should be populated.

  5. Start Sidekiq using gdk start rails-background-jobs and wait for the BulkImports::FinishBatchedRelationExportWorker to execute

  6. Access the relation status API: http://gdk.test:3000/api/v4/groups/24/export_relations/status?relation=labels. The relation's main status should be 1

BulkImports::PipelineWorker

  1. Apply the following patch to make BulkImports::PipelineWorker to take longer to import labels and reduce max_retries_after_interruption to 1.
Patch BulkImports::PipelineWorker 
diff --git a/app/workers/bulk_imports/pipeline_worker.rb b/app/workers/bulk_imports/pipeline_worker.rb
index c0addd96bda0..a74978e0a862 100644
--- a/app/workers/bulk_imports/pipeline_worker.rb
+++ b/app/workers/bulk_imports/pipeline_worker.rb
@@ -16,7 +16,7 @@ class PipelineWorker
     data_consistency :always
     feature_category :importers
     sidekiq_options dead: false, retry: 6
-    sidekiq_options max_retries_after_interruption: 20
+    sidekiq_options max_retries_after_interruption: 1
     worker_has_external_dependencies!
     deduplicate :until_executing
     worker_resource_boundary :memory
@@ -87,6 +87,8 @@ def run
 
       return re_enqueue if export_empty? || export_started?
 
+      sleep 300 if pipeline_tracker.pipeline_class.to_s == 'BulkImports::Common::Pipelines::LabelsPipeline'
+
       if file_extraction_pipeline? && export_status.batched?
         log_extra_metadata_on_done(:batched, true)
  1. Trigger a migration using Direct Transfer
  2. Wait for BulkImports::PipelineWorker worker that migrates labels to start
  3. Stop Sidekiq using gdk stop rails-background-jobs
  4. Start Sidekiq using gdk start rails-background-jobs and wait for the migration to finish
  5. Access the migration details
  6. Check if the migration partially succeeded and if there is an error regarding labels

BulkImports::PipelineBatchWorker

  1. Apply the following patch to make BulkImports::PipelineBatchWorker to take longer to import labels and reduce max_retries_after_interruption to 1.
Patch BulkImports::PipelineBatchWorker 
diff --git a/app/workers/bulk_imports/pipeline_batch_worker.rb b/app/workers/bulk_imports/pipeline_batch_worker.rb
index 2ce8bf9ac93b..c19f28d888c8 100644
--- a/app/workers/bulk_imports/pipeline_batch_worker.rb
+++ b/app/workers/bulk_imports/pipeline_batch_worker.rb
@@ -11,7 +11,7 @@ class PipelineBatchWorker
     data_consistency :always # rubocop:disable SidekiqLoadBalancing/WorkerDataConsistency
     feature_category :importers
     sidekiq_options dead: false, retry: 6
-    sidekiq_options max_retries_after_interruption: 20
+    sidekiq_options max_retries_after_interruption: 1
     worker_has_external_dependencies!
     worker_resource_boundary :memory
     idempotent!
@@ -81,6 +81,7 @@ def run
 
       logger.info(log_attributes(message: 'Batch tracker started'))
       batch.start!
+      sleep 300 if tracker.pipeline_class.to_s == 'BulkImports::Common::Pipelines::LabelsPipeline'
       tracker.pipeline_class.new(context).run
       batch.finish!
       logger.info(log_attributes(message: 'Batch tracker finished'))
  1. Trigger a migration using Direct Transfer
  2. Wait for BulkImports::PipelineBatchWorker worker that migrates labels to start
  3. Stop Sidekiq using gdk stop rails-background-jobs
  4. Start Sidekiq using gdk start rails-background-jobs and wait for the migration to finish
  5. Access the migration details
  6. Check if the migration partially succeeded and if there is an error regarding labels
Edited by Rodrigo Tomonari

Merge request reports