Add worker to export projects in parallel
Compare changes
Files
4- Rodrigo Tomonari authored
@@ -12,10 +12,10 @@ def initialize(export_job, current_user, after_export_strategy)
@@ -77,7 +77,7 @@ def log_info(message)
This change is part of a sequence of changes that will export projects using multiple Sidekiq Jobs rather than running the whole export process in just one single job.
This MR adds the worker and service responsible for generating the final project export archive, uploading the file to object store, and notifying the user that the project was exported.
The worker and service are similar to the ProjectExportWorker and Projects::ImportExport::ExportService, and because of that, they share most of the same components.
For context, Projects::ImportExport::ExportService
uses several "savers" classes to export each relation of the project and put them in a folder that is compressed in a tar.gz file. On the other hand, ParallelExportService
uses ExportedRelationsMerger class (introduced in !102815 (merged)) to merge all relations that are exported in parallel into a folder that is compressed in a tar.gz file.
Related to: #360685 (closed)
Because this is a work in progress, the Rails console needs to be used to test the change.
Open the Rails console and execute the commands below. The commands will enqueue several Sidekiq Jobs, and each will export a relation of the project and upload/copy the file to object store.
In the table project_export_jobs
and project_relation_export_uploads
it's possible to check when the export completes.
project = Project.first # pick a project to export
project_export_job = project.export_jobs.create(status: 0, jid: SecureRandom.hex(10))
Projects::ImportExport::RelationExport.relation_names_list.each do |relation|
relation_export = project_export_job.relation_exports.create(relation: relation)
Projects::ImportExport::RelationExportWorker.perform_async(relation_export.id)
end
project_export_job.start!
When all relations are exported, use the command below to execute the worker that will compress the exported files, generate a tar.gz, upload to object store, and notify the user.
Projects::ImportExport::ParallelProjectExportWorker.perform_async(project_export_job.id, User.first.id)
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.