Perform top level relations exports in separate sidekiq jobs
Depends on #360683 (closed)
ProjectExportWorker is now executed as part of 1 Sidekiq job as one export process.
In order to parallelize the export process, we need to export relations in separate Sidekiq jobs.
The current export service has a list of exporters that we need to now execute as part of separate Sidekiq jobs.
Diagram
In this issue, we're going to implement the second column of the diagram that is to export each component of the project in a separated job
sequenceDiagram
autonumber
participant export_service as Projects#58;#58;ImportExport#58;#58;ExportService
participant relations_worker as Projects#58;#58;ImportExport#58;#58;RelationExportWorker
participant tracker as Project#58;#58;ImportExport#58;#58;TrackExportRelationsWorker
participant export_worker as ProjectExportWorker
participant export_service as Projects#58;#58;ImportExport#58;#58;ExportService
participant after_export_worker as Projects#58;#58;ImportExport#58;#58;AfterExportWorker
par Generate relations in parallel
export_service->>relations_worker: Enqueue worker to generate relation A
export_service->>relations_worker: Enqueue worker to generate relation B
export_service->>relations_worker: Enqueue worker to generate relation C
export_service->>relations_worker: ...
end
export_service->>tracker: Enqueues TrackerExportRelationsWorker
loop
tracker->>tracker: Monitor if all relations were generated. <br />Keep reenqueing itself until all relations are generarated.<br />Notify user in case of an error occurs
end
tracker->>export_worker: Enqueue worker
Note right of export_worker: Download exported relations<br />and create tarball
export_worker->>after_export_worker: Enqueue worker
Note right of after_export_worker: Send file to S3, email user, etc
Edited by Rodrigo Tomonari