Add Group relations export API (!59295) · Merge requests · GitLab.org / GitLab

George Koltsov requested to merge georgekoltsov/bulk_import_group_exports into master Apr 13, 2021

What does this MR do?

Background information on the need for this change: &5769

This MR adds Group relations export API. It is similar to Group Export (https://docs.gitlab.com/ee/user/group/settings/import_export.html) with a few differences:

Each top level relation is exported in a separate sidekiq worker, compressed and uploaded to Object Storage separately. This way export is distributed across multiple workers, occupying individual workers for less amount of time. Additionally, this brings down total size of the file significantly.
Each top level relation is exported to .ndjson file, compressed and stored with carrierwave
Each relation has a status API to be able to view it's progress

This functionality is added in order to enable Bulk Import (https://docs.gitlab.com/ee/user/group/import/) group migration. To import group/subgroup structures with one click, instead of having user to migrate groups one by one by dealing with archive files.

It requires a few new models to support the process:

1.BulkImports::Export to track individual top level relation export status

BulkImports::ExportUpload to store exported gzip in ObjectStorage and allow it to be downloaded

The intention is to add the same for projects in the follow up MR.

Sequence diagram:

Mentions #328216 (closed)

Screenshots (strongly suggested)

screencast_2021-04-19_15-30-32

Migrations output

db/migrate/20210414100914_add_bulk_import_exports_table.rb

== 20210414100914 AddBulkImportExportsTable: migrating ========================
-- create_table(:bulk_import_exports)
   -> 0.0060s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_exports\nADD CONSTRAINT check_24cb010672\nCHECK ( char_length(relation) <= 255 )\nNOT VALID;\n")
   -> 0.0009s
-- current_schema()
   -> 0.0002s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- execute("ALTER TABLE bulk_import_exports VALIDATE CONSTRAINT check_24cb010672;")
   -> 0.0009s
-- execute("RESET ALL")
   -> 0.0006s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_exports\nADD CONSTRAINT check_9ee6d14d33\nCHECK ( char_length(jid) <= 255 )\nNOT VALID;\n")
   -> 0.0008s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_exports VALIDATE CONSTRAINT check_9ee6d14d33;")
   -> 0.0009s
== 20210414100914 AddBulkImportExportsTable: migrated (0.0269s) ===============

Down

== 20210414100914 AddBulkImportExportsTable: reverting ========================
-- drop_table(:bulk_import_exports)
   -> 0.0044s
== 20210414100914 AddBulkImportExportsTable: reverted (0.0045s) ===============

db/migrate/20210414130017_add_foreign_key_to_bulk_import_exports_on_project.rb

== 20210414130017 AddForeignKeyToBulkImportExportsOnProject: migrating ========
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:bulk_import_exports)
   -> 0.0037s
-- execute("ALTER TABLE bulk_import_exports\nADD CONSTRAINT fk_39c726d3b5\nFOREIGN KEY (project_id)\nREFERENCES projects (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0045s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- execute("ALTER TABLE bulk_import_exports VALIDATE CONSTRAINT fk_39c726d3b5;")
   -> 0.0045s
-- execute("RESET ALL")
   -> 0.0012s
== 20210414130017 AddForeignKeyToBulkImportExportsOnProject: migrated (0.0238s)

Down

== 20210414130017 AddForeignKeyToBulkImportExportsOnProject: reverting ========
-- remove_foreign_key(:bulk_import_exports, {:column=>:project_id})
   -> 0.0043s
== 20210414130017 AddForeignKeyToBulkImportExportsOnProject: reverted (0.0086s)

db/migrate/20210414130526_add_foreign_key_to_bulk_import_exports_on_group.rb

== 20210414130526 AddForeignKeyToBulkImportExportsOnGroup: migrating ==========
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:bulk_import_exports)
   -> 0.0033s
-- execute("ALTER TABLE bulk_import_exports\nADD CONSTRAINT fk_8c6f33cebe\nFOREIGN KEY (group_id)\nREFERENCES namespaces (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0025s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- execute("ALTER TABLE bulk_import_exports VALIDATE CONSTRAINT fk_8c6f33cebe;")
   -> 0.0094s
-- execute("RESET ALL")
   -> 0.0008s
== 20210414130526 AddForeignKeyToBulkImportExportsOnGroup: migrated (0.0231s) =

Down

== 20210414130526 AddForeignKeyToBulkImportExportsOnGroup: reverting ==========
-- remove_foreign_key(:bulk_import_exports, {:column=>:group_id})
   -> 0.0034s
== 20210414130526 AddForeignKeyToBulkImportExportsOnGroup: reverted (0.0076s) =

db/migrate/20210414131807_add_bulk_import_exports_table_indexes.rb

== 20210414131807 AddBulkImportExportsTableIndexes: migrating =================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:bulk_import_exports, [:group_id, :relation], {:unique=>true, :where=>"group_id IS NOT NULL", :name=>"partial_index_bulk_import_exports_on_group_id_and_relation", :algorithm=>:concurrently})
   -> 0.0017s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- add_index(:bulk_import_exports, [:group_id, :relation], {:unique=>true, :where=>"group_id IS NOT NULL", :name=>"partial_index_bulk_import_exports_on_group_id_and_relation", :algorithm=>:concurrently})
   -> 0.0073s
-- execute("RESET ALL")
   -> 0.0006s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:bulk_import_exports, [:project_id, :relation], {:unique=>true, :where=>"project_id IS NOT NULL", :name=>"partial_index_bulk_import_exports_on_project_id_and_relation", :algorithm=>:concurrently})
   -> 0.0013s
-- add_index(:bulk_import_exports, [:project_id, :relation], {:unique=>true, :where=>"project_id IS NOT NULL", :name=>"partial_index_bulk_import_exports_on_project_id_and_relation", :algorithm=>:concurrently})
   -> 0.0031s
== 20210414131807 AddBulkImportExportsTableIndexes: migrated (0.0164s) ========

Down

== 20210414131807 AddBulkImportExportsTableIndexes: reverting =================
-- transaction_open?()
   -> 0.0000s
-- indexes(:bulk_import_exports)
   -> 0.0030s
-- execute("SET statement_timeout TO 0")
   -> 0.0008s
-- remove_index(:bulk_import_exports, {:algorithm=>:concurrently, :name=>"partial_index_bulk_import_exports_on_group_id_and_relation"})
   -> 0.0038s
-- execute("RESET ALL")
   -> 0.0010s
-- transaction_open?()
   -> 0.0000s
-- indexes(:bulk_import_exports)
   -> 0.0013s
-- remove_index(:bulk_import_exports, {:algorithm=>:concurrently, :name=>"partial_index_bulk_import_exports_on_project_id_and_relation"})
   -> 0.0028s
== 20210414131807 AddBulkImportExportsTableIndexes: reverted (0.0153s) ========

db/migrate/20210414133310_add_bulk_import_export_uploads_table.rb

== 20210414133310 AddBulkImportExportUploadsTable: migrating ==================
-- create_table(:bulk_import_export_uploads)
   -> 0.0072s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE bulk_import_export_uploads\nADD CONSTRAINT check_5add76239d\nCHECK ( char_length(export_file) <= 255 )\nNOT VALID;\n")
   -> 0.0009s
-- current_schema()
   -> 0.0002s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- execute("ALTER TABLE bulk_import_export_uploads VALIDATE CONSTRAINT check_5add76239d;")
   -> 0.0008s
-- execute("RESET ALL")
   -> 0.0005s
== 20210414133310 AddBulkImportExportUploadsTable: migrated (0.0187s) =========

Down

== 20210414133310 AddBulkImportExportUploadsTable: reverting ==================
-- drop_table(:bulk_import_export_uploads)
   -> 0.0048s
== 20210414133310 AddBulkImportExportUploadsTable: reverted (0.0048s) =========

db/migrate/20210419085714_add_foreign_key_to_bulk_import_export_uploads_on_export.rb

== 20210419085714 AddForeignKeyToBulkImportExportUploadsOnExport: reverting ===
-- remove_foreign_key(:bulk_import_export_uploads, {:column=>:export_id})
   -> 0.0051s
== 20210419085714 AddForeignKeyToBulkImportExportUploadsOnExport: reverted (0.0109s) 

➜  gitlab git:(georgekoltsov/bulk_import_group_exports) ✗ mig
== 20210419085714 AddForeignKeyToBulkImportExportUploadsOnExport: migrating ===
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:bulk_import_export_uploads)
   -> 0.0030s
-- execute("ALTER TABLE bulk_import_export_uploads\nADD CONSTRAINT fk_dfbfb45eca\nFOREIGN KEY (export_id)\nREFERENCES bulk_import_exports (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0016s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- execute("ALTER TABLE bulk_import_export_uploads VALIDATE CONSTRAINT fk_dfbfb45eca;")
   -> 0.0022s
-- execute("RESET ALL")
   -> 0.0007s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:bulk_import_export_uploads, :export_id, {:name=>"index_bulk_import_export_uploads_on_export_id", :algorithm=>:concurrently})
   -> 0.0014s
-- add_index(:bulk_import_export_uploads, :export_id, {:name=>"index_bulk_import_export_uploads_on_export_id", :algorithm=>:concurrently})
   -> 0.0041s
== 20210419085714 AddForeignKeyToBulkImportExportUploadsOnExport: migrated (0.0216s)

Down

== 20210419085714 AddForeignKeyToBulkImportExportUploadsOnExport: reverting ===
-- remove_foreign_key(:bulk_import_export_uploads, {:column=>:export_id})
   -> 0.0041s
-- transaction_open?()
   -> 0.0000s
-- indexes(:bulk_import_export_uploads)
   -> 0.0019s
-- execute("SET statement_timeout TO 0")
   -> 0.0004s
-- remove_index(:bulk_import_export_uploads, {:algorithm=>:concurrently, :name=>"index_bulk_import_export_uploads_on_export_id"})
   -> 0.0017s
-- execute("RESET ALL")
   -> 0.0005s
== 20210419085714 AddForeignKeyToBulkImportExportUploadsOnExport: reverted (0.0149s)

Does this MR meet the acceptance criteria?

Conformity

📋 Does this MR need a changelog?
- I have included a changelog entry.
- I have not included a changelog entry because _____.
Documentation (if required)
Code review guidelines
Merge request performance guidelines
Style guides
Database guides
Separation of EE specific content

Availability and Testing

Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
Tested in all supported browsers
Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Edited Apr 22, 2021 by George Koltsov

Add Group relations export API

What does this MR do?

Screenshots (strongly suggested)

Migrations output

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Merge request reports