Add worker to export projects in parallel
What does this MR do and why?
This change is part of a sequence of changes that will export projects using multiple Sidekiq Jobs rather than running the whole export process in just one single job.
This MR adds the worker and service responsible for generating the final project export archive, uploading the file to object store, and notifying the user that the project was exported.
The worker and service are similar to the ProjectExportWorker and Projects::ImportExport::ExportService, and because of that, they share most of the same components.
For context, Projects::ImportExport::ExportService
uses several "savers" classes to export each relation of the project and put them in a folder that is compressed in a tar.gz file. On the other hand, ParallelExportService
uses ExportedRelationsMerger class (introduced in !102815 (merged)) to merge all relations that are exported in parallel into a folder that is compressed in a tar.gz file.
Related to: #360685 (closed)
Screenshots or screen recordings
How to set up and validate locally
Because this is a work in progress, the Rails console needs to be used to test the change.
Open the Rails console and execute the commands below. The commands will enqueue several Sidekiq Jobs, and each will export a relation of the project and upload/copy the file to object store.
In the table project_export_jobs
and project_relation_export_uploads
it's possible to check when the export completes.
project = Project.first # pick a project to export
project_export_job = project.export_jobs.create(status: 0, jid: SecureRandom.hex(10))
Projects::ImportExport::RelationExport.relation_names_list.each do |relation|
relation_export = project_export_job.relation_exports.create(relation: relation)
Projects::ImportExport::RelationExportWorker.perform_async(relation_export.id)
end
project_export_job.start!
When all relations are exported, use the command below to execute the worker that will compress the exported files, generate a tar.gz, upload to object store, and notify the user.
Projects::ImportExport::ParallelProjectExportWorker.perform_async(project_export_job.id, User.first.id)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Merge request reports
Activity
assigned to @rodrigo.tomonari
- A deleted user
added backend label
1 Warning a87f4f3f: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend Maxime Orefice (
@morefice
) (UTC+1, 1 hour ahead of@rodrigo.tomonari
)Douglas Barbosa Alexandre (
@dbalexandre
) (UTC-3, 3 hours behind@rodrigo.tomonari
)To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
Sidekiq queue changes
This merge request contains changes to Sidekiq queues. Please follow the documentation on changing a queue's urgency.
These queues were added:
projects_import_export_parallel_project_export
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger- Resolved by 🤖 GitLab Bot 🤖
Proper labels assigned to this merge request. Please ignore me.
@rodrigo.tomonari - please see the following guidance and update this merge request.1 Error Please add typebug typefeature, or typemaintenance label to this merge request. Edited by 🤖 GitLab Bot 🤖
Setting label(s) devopsmanage sectiondev based on ~"group::import".
added devopsmanage sectiondev labels
added 1004 commits
-
f6ae4ed3...01f2dc3a - 1003 commits from branch
master
- 4b08fb57 - Add service and worker to export projects in parallel
-
f6ae4ed3...01f2dc3a - 1003 commits from branch
Allure report
allure-report-publisher
generated test report!e2e-review-qa:
test report for ca993f03expand test summary
+-----------------------------------------------------------------------------------------+ | suites summary | +------------------------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +------------------------------------+--------+--------+---------+-------+-------+--------+ | Manage | 56 | 0 | 9 | 4 | 65 | ❗ | | Verify | 12 | 0 | 1 | 0 | 13 | ✅ | | Govern | 10 | 0 | 5 | 0 | 15 | ✅ | | Create | 28 | 0 | 1 | 0 | 29 | ✅ | | Plan | 49 | 0 | 1 | 0 | 50 | ✅ | | Feature flag handler sanity checks | 9 | 0 | 0 | 0 | 9 | ✅ | | Package | 0 | 0 | 1 | 0 | 1 | ➖ | | Configure | 0 | 0 | 1 | 0 | 1 | ➖ | | Version sanity check | 0 | 0 | 1 | 0 | 1 | ➖ | +------------------------------------+--------+--------+---------+-------+-------+--------+ | Total | 164 | 0 | 20 | 4 | 184 | ❗ | +------------------------------------+--------+--------+---------+-------+-------+--------+
changed milestone to %15.7
added 3 commits
-
4b08fb57...702e1e10 - 2 commits from branch
master
- 199f9554 - Add service and worker to export projects in parallel
-
4b08fb57...702e1e10 - 2 commits from branch
added maintenancerefactor typemaintenance labels
added 1 commit
- bf08f8e7 - Add service and worker to export projects in parallel
@georgekoltsov, may I ask you to review this change, please?
It's the continuation of !102815 (merged)
requested review from @georgekoltsov
added 1 commit
- ca993f03 - Add service and worker to export projects in parallel
- Resolved by Rodrigo Tomonari
- Resolved by Rodrigo Tomonari
- Resolved by Rodrigo Tomonari
@georgekoltsov
, thanks for approving this merge request.This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.
For more info, please refer to the following links:
removed review request for @georgekoltsov
added 1910 commits
-
ca993f03...bcdf37e8 - 1908 commits from branch
master
- c4bd674b - Add service and worker to export projects in parallel
- 1fd7a020 - Change log message and add test
-
ca993f03...bcdf37e8 - 1908 commits from branch
- Resolved by Rodrigo Tomonari
- Resolved by Douglas Barbosa Alexandre
@dbalexandre, may I ask you to do the maintainer review, please?
By the way, @georgekoltsov already did the backend review and approved the MR, but after amending the MR with his small comments, I lost his approval and because he is OOO until next week, I assumed it's fine to ask for maintainer review.
Edited by Rodrigo Tomonari
requested review from @dbalexandre
- Resolved by Douglas Barbosa Alexandre
- Resolved by Douglas Barbosa Alexandre
- Resolved by Douglas Barbosa Alexandre
- Resolved by Douglas Barbosa Alexandre
removed review request for @dbalexandre
requested review from @dbalexandre
enabled an automatic merge when the pipeline for a53b8ecf succeeds
Thanks, @rodrigo.tomonari! I triggered a new pipeline and set MWPS