Import group epics using ndjson
With #328216 (closed) it is now possible to export single top level relations for a group, which allows us to import epics using exported ndjson file which includes all of it's relations.
Proposed solution
- At some point before running import pipeline we need to request export of relations
- Create new ETL pipeline, similar to existing
EpicsPipeline
that is going to be handling incoming ndjson epics data. There are going to be a few key differences:- Check and see if epics relation is available to download. Reenqueue pipeline worker if export is not yet available. Set expiration on import to be failed if export did not become available after X amount of time
- Download and decompress exported relation. Could potentially be a
before_run
action for the pipeline worker? - Extractor can be a simple file reader that yields one line at a time
- Each line needs to be processed by doing something similar to what we do in Import - go through all of the relations within a hash and convert them into objects (see POC here !58404 (diffs))
Edited by Kassio Borges