Track when relations are exported
ProjectExportWorker is now executed as part of 1 Sidekiq job as one export process. In order to parallelize the export process, we need to export relations in separate Sidekiq jobs, track when they are completed, and trigger the tarball generation.
#360684 (closed) implements a worker to generate the relation in separate Sidekiq jobs. And in this issue, we need to create a worker to monitor the generation of the relation.
The worker has to track the relation generation. In case an error occurs, the worker has to send an email to the users notifying them that the export failed similar to how it's done in the ProjectExportWorker/Service. And also change the of the ProjectExportJob to failed.
The tracker should also monitor if any relation worker hasn't gotten stuck and eventually marked the relation export as failed.
In case, all relations are exported successfully, we have to enqueue the ProjectExportWorker that will continue the export process.
Diagram
In this issue, we're going to implement the third column of the diagram
sequenceDiagram
autonumber
participant export_service as Projects#58;#58;ImportExport#58;#58;ExportService
participant relations_worker as Projects#58;#58;ImportExport#58;#58;RelationExportWorker
participant tracker as Project#58;#58;ImportExport#58;#58;TrackExportRelationsWorker
participant export_worker as ProjectExportWorker
participant export_service as Projects#58;#58;ImportExport#58;#58;ExportService
participant after_export_worker as Projects#58;#58;ImportExport#58;#58;AfterExportWorker
par Generate relations in parallel
export_service->>relations_worker: Enqueue worker to generate relation A
export_service->>relations_worker: Enqueue worker to generate relation B
export_service->>relations_worker: Enqueue worker to generate relation C
export_service->>relations_worker: ...
end
export_service->>tracker: Enqueues TrackerExportRelationsWorker
loop
tracker->>tracker: Monitor if all relations were generated. <br />Keep reenqueing itself until all relations are generarated.<br />Notify user in case of an error occurs
end
tracker->>export_worker: Enqueue worker
Note right of export_worker: Download exported relations<br />and create tarball
export_worker->>after_export_worker: Enqueue worker
Note right of after_export_worker: Send file to S3, email user, etc