Skip to content

Failing entity if pipeline aborts on failure

Max Fan requested to merge 425887-fail-entity-on-pipeline-worker into master

What does this MR do and why?

Currently if pipelines fail during BulkImports::PipelineWorker the import will still progress. It does not take into account if the pipeline has abort_on_failure? flag enabled.

This causes subsequent pipelines to run and then fail.

This MR changes so when a pipeline fails that has abort_on_failure? the entity is failed and any further processing is skipped.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After
image.png image.png

How to set up and validate locally

Easier way to test is to modify the code directly to trigger failures

  1. Checkout this branch and make this edit
+++ b/app/workers/bulk_imports/pipeline_worker.rb
@@ -51,6 +51,10 @@ def perform_failure(pipeline_tracker_id, entity_id, exception)
     def run
       return skip_tracker if entity.failed?
 
+      # this pipeline has abort_on_failure!
+      if pipeline_tracker.pipeline_class == BulkImports::Projects::Pipelines::RepositoryBundlePipeline
+        raise StandardError, "expected fail"
+      end
+
       raise(Pipeline::FailedError, "Export from source instance failed: #{export_status.error}") if export_failed?
       raise(Pipeline::ExpiredError, 'Empty export status on source instance') if empty_export_timeout?
  1. Running direct transfer will now fail the import (see screenshot after) and finding the entity in the database will have status: -1 (failed).
  2. Running other pipelines will continue the import if they do not have abort_on_failure!
+++ b/app/workers/bulk_imports/pipeline_worker.rb
@@ -51,6 +51,11 @@ def perform_failure(pipeline_tracker_id, entity_id, exception)
     def run
       return skip_tracker if entity.failed?
 
+      # this pipeline does not have abort_on_failure! tag
+      if pipeline_tracker.pipeline_class == BulkImports::Projects::Pipelines::ProjectAttributesPipeline
+        raise StandardError, "expected fail"
+      end
+
       raise(Pipeline::FailedError, "Export from source instance failed: #{export_status.error}") if export_failed?
       raise(Pipeline::ExpiredError, 'Empty export status on source instance') if empty_export_timeout?
  1. Running direct transfer now will allow the import to complete and finish. It'll be "partially complete"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #425887 (closed)

Edited by Max Fan

Merge request reports