Combine all exported parts of project in one tarball before serving it to the exporter user
Once all partial exports are complete as part of #360684 (closed) we need to be able to combine all the partially exported types of data into one tarball before serving it back to the user.
Diagram
sequenceDiagram
autonumber
participant export_service as Projects#58;#58;ImportExport#58;#58;ExportService
participant relations_worker as Projects#58;#58;ImportExport#58;#58;RelationExportWorker
participant tracker as Project#58;#58;ImportExport#58;#58;TrackExportRelationsWorker
participant export_worker as ProjectExportWorker
participant export_service as Projects#58;#58;ImportExport#58;#58;ExportService
participant after_export_worker as Projects#58;#58;ImportExport#58;#58;AfterExportWorker
par Generate relations in parallel
export_service->>relations_worker: Enqueue worker to generate relation A
export_service->>relations_worker: Enqueue worker to generate relation B
export_service->>relations_worker: Enqueue worker to generate relation C
export_service->>relations_worker: ...
end
export_service->>tracker: Enqueues TrackerExportRelationsWorker
loop
tracker->>tracker: Monitor if all relations were generated. <br />Keep re-enqueing itself until all relations are generated.<br />Notify user in case of an error occurs
end
tracker->>export_worker: Enqueue worker
Note right of export_worker: Download exported relations<br />and create tarball
export_worker->>after_export_worker: Enqueue worker
Note right of after_export_worker: Send the file to S3, email user, etc
Technical details
In this issue, we will implement the code needed for step 8 of the diagram above: download, decompress, and combine all partial export relations into a single tarball with the same structure as the export file currently generated.
After creating the final tarball, we will upload it to the object store similarly to how it is done in ProjectExportWorker.
Edited by Rodrigo Tomonari