Skip to content
Snippets Groups Projects

Add worker to export projects in parallel

Merged Rodrigo Tomonari requested to merge rodrigo/parallel-export into master

What does this MR do and why?

This change is part of a sequence of changes that will export projects using multiple Sidekiq Jobs rather than running the whole export process in just one single job.

This MR adds the worker and service responsible for generating the final project export archive, uploading the file to object store, and notifying the user that the project was exported.

The worker and service are similar to the ProjectExportWorker and Projects::ImportExport::ExportService, and because of that, they share most of the same components.

For context, Projects::ImportExport::ExportService uses several "savers" classes to export each relation of the project and put them in a folder that is compressed in a tar.gz file. On the other hand, ParallelExportService uses ExportedRelationsMerger class (introduced in !102815 (merged)) to merge all relations that are exported in parallel into a folder that is compressed in a tar.gz file.

Related to: #360685 (closed)

Screenshots or screen recordings

parallel-export

How to set up and validate locally

Because this is a work in progress, the Rails console needs to be used to test the change.

Open the Rails console and execute the commands below. The commands will enqueue several Sidekiq Jobs, and each will export a relation of the project and upload/copy the file to object store.

In the table project_export_jobs and project_relation_export_uploads it's possible to check when the export completes.

project = Project.first # pick a project to export

project_export_job = project.export_jobs.create(status: 0, jid: SecureRandom.hex(10))

Projects::ImportExport::RelationExport.relation_names_list.each do |relation|
  relation_export = project_export_job.relation_exports.create(relation: relation)
  Projects::ImportExport::RelationExportWorker.perform_async(relation_export.id)
end

project_export_job.start!

When all relations are exported, use the command below to execute the worker that will compress the exported files, generate a tar.gz, upload to object store, and notify the user.

Projects::ImportExport::ParallelProjectExportWorker.perform_async(project_export_job.id, User.first.id)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • George Koltsov
  • LGTM :thumbsup: Just a few small questions

  • George Koltsov approved this merge request

    approved this merge request

  • :wave: @georgekoltsov, thanks for approving this merge request.

    This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.

    For more info, please refer to the following links:

  • 🤖 GitLab Bot 🤖 added 1 deleted label

    added 1 deleted label

  • George Koltsov removed review request for @georgekoltsov

    removed review request for @georgekoltsov

  • Rodrigo Tomonari added 1910 commits

    added 1910 commits

    Compare with previous version

  • Ghost User
  • added 1 commit

    • 11bbe68a - Change log message and add test

    Compare with previous version

  • added 1 commit

    • 127617f1 - Change log message and add test

    Compare with previous version

  • added 1 commit

    • a87f4f3f - Change log message and add test

    Compare with previous version

  • Rodrigo Tomonari resolved all threads

    resolved all threads

  • requested review from @dbalexandre

  • removed review request for @dbalexandre

  • added 1 commit

    • 1e58c604 - Change log message and guard clause

    Compare with previous version

  • requested review from @dbalexandre

  • Douglas Barbosa Alexandre approved this merge request

    approved this merge request

  • resolved all threads

  • Douglas Barbosa Alexandre enabled an automatic merge when the pipeline for a53b8ecf succeeds

    enabled an automatic merge when the pipeline for a53b8ecf succeeds

  • Thanks, @rodrigo.tomonari! I triggered a new pipeline and set MWPS :green_heart: :rocket:

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading