Skip to content

Perform top level relations exports in separate sidekiq jobs

Depends on #360683 (closed)

ProjectExportWorker is now executed as part of 1 Sidekiq job as one export process.

In order to parallelize the export process, we need to export relations in separate Sidekiq jobs.

The current export service has a list of exporters that we need to now execute as part of separate Sidekiq jobs.

Diagram

In this issue, we're going to implement the second column of the diagram that is to export each component of the project in a separated job

sequenceDiagram
    autonumber

    participant export_service as Projects#58;#58;ImportExport#58;#58;ExportService
    participant relations_worker as Projects#58;#58;ImportExport#58;#58;RelationExportWorker
    participant tracker as Project#58;#58;ImportExport#58;#58;TrackExportRelationsWorker
    participant export_worker as ProjectExportWorker
    participant export_service as Projects#58;#58;ImportExport#58;#58;ExportService
    participant after_export_worker as Projects#58;#58;ImportExport#58;#58;AfterExportWorker

    par Generate relations in parallel
        export_service->>relations_worker: Enqueue worker to generate relation A
        export_service->>relations_worker: Enqueue worker to generate relation B
        export_service->>relations_worker: Enqueue worker to generate relation C
        export_service->>relations_worker: ...
    end

    export_service->>tracker: Enqueues TrackerExportRelationsWorker
    loop
        tracker->>tracker: Monitor if all relations were generated. <br />Keep reenqueing itself until all relations are generarated.<br />Notify user in case of an error occurs
    end
 
    tracker->>export_worker: Enqueue worker 
    Note right of export_worker: Download exported relations<br />and create tarball
    export_worker->>after_export_worker: Enqueue worker 
    Note right of after_export_worker: Send file to S3, email user, etc
Edited by Rodrigo Tomonari