Skip to content

Set ProjectAttributesPipeline to be a file extraction pipeline

What does this MR do and why?

Related to: #358745 (closed)

  • Rename the flag ndjson_pipeline! to file_extraction_pipeline!
  • Update the BulkImport pipelines to use the file_extraction_pipeline! flag
  • Add the flag file_extraction_pipeline! to the pipeline ProjectAttributesPipeline

Context

The ndjson_pipeline! needs to be added to the BulkImport pipelines that import the data from files that are generated in the source instance. Basically, the flag makes the BulkImports::EntityWorker to keep re-enqueuing itself until the exported file is ready to be downloaded from the source instance by the target instance.

Initially, the flag was named ndjson_pipeline! because the re-enqueuing mechanist was only used in pipelines that read the information from .ndjson files. However now we are also using the re-enqueuing mechanist to read from .json files, so we decided to rename the flag to file_extraction_pipeline!.

Besides the flag wasn't present in the ProjectAttributesPipeline pipeline that reads information from an exported file hence the flag needs to be added to the pipeline.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

  1. Feature.enable(:bulk_import).
  2. Feature.enable(:bulk_import_projects).
  3. Create a top level group and project in it.
  4. Go to /groups/new#import-group-pane page and enter instance url and access token (needs to be api & read_repository scope).
  5. Select newly created group and click Import.
  6. Wait for Group import to complete and verify the imported group and project data.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports