Skip to content

Add Group relations export API

What does this MR do?

This MR is a part 2 split from the original MR that implements Group relations export API !59295 (closed) and it adds export API and sidekiq worker to perform the export. Part 1 can be found here !59976 (merged)

Description from !59295 (closed)

Background information on the need for this change: &5769

This MR adds Group relations export API. It is similar to Group Export (https://docs.gitlab.com/ee/user/group/settings/import_export.html) with a few differences:

  1. Each top level relation is exported in a separate sidekiq worker, compressed and uploaded to Object Storage separately. This way export is distributed across multiple workers, occupying individual workers for less amount of time. Additionally, this brings down total size of the file significantly.
  2. Each top level relation is exported to .ndjson file, compressed and stored with carrierwave
  3. Each relation has a status API to be able to view it's progress

This functionality is added in order to enable Bulk Import (https://docs.gitlab.com/ee/user/group/import/) group migration. To import group/subgroup structures with one click, instead of having user to migrate groups one by one by dealing with archive files.

It requires a few new models to support the process:

1.BulkImports::Export to track individual top level relation export status

  1. BulkImports::ExportUpload to store exported gzip in ObjectStorage and allow it to be downloaded

The intention is to add the same for projects in the follow up MR.

I created 2 followups for this change that I will work on after this MR:

Sequence diagram:

image

Screenshots (strongly suggested)

screencast_2021-04-19_15-30-32

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov

Merge request reports