GitLab Next

What does this MR do?

This MR is a part 2 split from the original MR that implements Group relations export API !59295 (closed) and it adds export API and sidekiq worker to perform the export. Part 1 can be found here !59976 (merged)

Description from !59295 (closed)

Background information on the need for this change: &5769

This MR adds Group relations export API. It is similar to Group Export (https://docs.gitlab.com/ee/user/group/settings/import_export.html) with a few differences:

1. Each top level relation is exported in a separate sidekiq worker, compressed and uploaded to Object Storage separately. This way export is distributed across multiple workers, occupying individual workers for less amount of time. Additionally, this brings down total size of the file significantly.
2. Each top level relation is exported to .ndjson file, compressed and stored with carrierwave
3. Each relation has a status API to be able to view it's progress

This functionality is added in order to enable Bulk Import (https://docs.gitlab.com/ee/user/group/import/) group migration. To import group/subgroup structures with one click, instead of having user to migrate groups one by one by dealing with archive files.

It requires a few new models to support the process:

1.BulkImports::Export to track individual top level relation export status

2. BulkImports::ExportUpload to store exported gzip in ObjectStorage and allow it to be downloaded

The intention is to add the same for projects in the follow up MR.

I created 2 followups for this change that I will work on after this MR:

• Doc update #328372 (closed)
• Rate limits #328373

Sequence diagram:

Screenshots (strongly suggested)

screencast_2021-04-19_15-30-32

Does this MR meet the acceptance criteria?

Conformity

• 📋 Does this MR need a changelog?
  • I have included a changelog entry.
  • I have not included a changelog entry because _____.
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team