ProjectAttributesPipeline should be flagged as NDJSON pipeline
Summary
The pipeline should be flagged as an NDJSON pipeline in order for the BulkImports::PipelineWorker worker to keep re-enqueuing the pipeline until the NDJSON file is ready to be downloaded
Technical details
As the NDJSON files are generated in the source GitLab instance and it can take awhile for the file to be generated, we need to keep re-enqueuing the worker until the NDJSON file is ready to be downloaded.
To enable the re-enqueuing mechanism, the pipelines that extract data from a NDJSON file needs to have the flag ndjson_pipeline
enabled like this file otherwise the pipeline will fail to import the file
Proposed Solution
Since the ProjectAttributesPipeline
doesn't use a NDJSON and instead a JSON file, it was decided to rename the flag ndjson_pipeline
to file_extraction_pipeline
. See the thread below
What needs to be done
-
Update the method method name in the Pipeline concern -
Update other places that is using the flag ndjson_pipeline
to usefile_extraction_pipeline
-
Add the flag file_extraction_pipeline
in theProjectAttributesPipeline
pipeline -
Update the BulkImports::PipelineWorker
worker to use the flagfile_extraction_pipeline
instead of thendjson_pipeline