Skip to content

Limit Bulk Import extractors & loaders to 1 each per pipeline

Problem to solve

Currently Bulk Imports Group Migration BulkImports::Pipeline::Runner has an ability to add multiple extractors & loaders.

e.g.

module BulkImports
  module Groups
    module Pipelines
      class MyPipeline
        include Pipeline

        extractor Extractor
        extractor AnotherExtractor

        transformer Transformer 

        loader Loader
        loader AnotherLoader
      end
    end
  end
end

Because of that, BulkImports::Pipeline::Runner#run has to loop through extractors/loaders, increasing it's complexity. In reality, so far (and it might change in the future), there is no need in having multiple extractors or loaders. Usually it's 1 loader and 1 extractor per pipeline.

Proposal

Restrict defining extractors/loaders to 1 each per pipeline in order to reduce BulkImports::Pipeline::Runner#run complexity. If we are going to need to extract/load from/to multiple sources/destination we can revisit it and consider adding such functionality in the future, but there is no need in having it now.