Skip to content

Add Group relations export models

George Koltsov requested to merge georgekoltsov/add-group-relation-export-models into master

What does this MR do?

This MR is a part 1 split from the original MR that implements Group relations export API !59295 (closed) and it adds migrations and models only. Splitting the original up for easier review. The other MR is going to be API + worker changes.

Description from !59295 (closed)

Background information on the need for this change: &5769

This MR adds Group relations export API. It is similar to Group Export (https://docs.gitlab.com/ee/user/group/settings/import_export.html) with a few differences:

  1. Each top level relation is exported in a separate sidekiq worker, compressed and uploaded to Object Storage separately. This way export is distributed across multiple workers, occupying individual workers for less amount of time. Additionally, this brings down total size of the file significantly.
  2. Each top level relation is exported to .ndjson file, compressed and stored with carrierwave
  3. Each relation has a status API to be able to view it's progress

This functionality is added in order to enable Bulk Import (https://docs.gitlab.com/ee/user/group/import/) group migration. To import group/subgroup structures with one click, instead of having user to migrate groups one by one by dealing with archive files.

It requires a few new models to support the process:

1.BulkImports::Export to track individual top level relation export status

  1. BulkImports::ExportUpload to store exported gzip in ObjectStorage and allow it to be downloaded

The intention is to add the same for projects in the follow up MR.

I created 2 followups for this change that I will work on after this MR:

Sequence diagram:

image

Screenshots (strongly suggested)

Migrations output

Up 
== 20210414100914 AddBulkImportExportsTable: migrating ========================
-- create_table(:bulk_import_exports, {})
-- quote_column_name(:relation)
   -> 0.0000s
-- quote_column_name(:jid)
   -> 0.0000s
-- quote_column_name(:error)
   -> 0.0000s
   -> 0.0242s
-- quote_table_name("check_24cb010672")
   -> 0.0000s
-- quote_table_name("check_9ee6d14d33")
   -> 0.0000s
-- quote_table_name("check_8f0f357334")
   -> 0.0000s
-- quote_table_name(:bulk_import_exports)
   -> 0.0000s
-- execute("ALTER TABLE \"bulk_import_exports\"\nADD CONSTRAINT \"check_24cb010672\" CHECK (char_length(\"relation\") <= 255),\nADD CONSTRAINT \"check_9ee6d14d33\" CHECK (char_length(\"jid\") <= 255),\nADD CONSTRAINT \"check_8f0f357334\" CHECK (char_length(\"error\") <= 255)\n")
   -> 0.0028s
== 20210414100914 AddBulkImportExportsTable: migrated (0.0379s) ===============

== 20210414130017 AddForeignKeyToBulkImportExportsOnProject: migrating ========
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:bulk_import_exports)
   -> 0.0030s
-- execute("ALTER TABLE bulk_import_exports\nADD CONSTRAINT fk_39c726d3b5\nFOREIGN KEY (project_id)\nREFERENCES projects (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0052s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- execute("ALTER TABLE bulk_import_exports VALIDATE CONSTRAINT fk_39c726d3b5;")
   -> 0.0243s
-- execute("RESET ALL")
   -> 0.0006s
== 20210414130017 AddForeignKeyToBulkImportExportsOnProject: migrated (0.0380s) 

== 20210414130526 AddForeignKeyToBulkImportExportsOnGroup: migrating ==========
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:bulk_import_exports)
   -> 0.0021s
-- execute("ALTER TABLE bulk_import_exports\nADD CONSTRAINT fk_8c6f33cebe\nFOREIGN KEY (group_id)\nREFERENCES namespaces (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0038s
-- execute("ALTER TABLE bulk_import_exports VALIDATE CONSTRAINT fk_8c6f33cebe;")
   -> 0.0111s
== 20210414130526 AddForeignKeyToBulkImportExportsOnGroup: migrated (0.0210s) =

== 20210414131807 AddBulkImportExportsTableIndexes: migrating =================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:bulk_import_exports, [:group_id, :relation], {:unique=>true, :where=>"group_id IS NOT NULL", :name=>"partial_index_bulk_import_exports_on_group_id_and_relation", :algorithm=>:concurrently})
   -> 0.0013s
-- add_index(:bulk_import_exports, [:group_id, :relation], {:unique=>true, :where=>"group_id IS NOT NULL", :name=>"partial_index_bulk_import_exports_on_group_id_and_relation", :algorithm=>:concurrently})
   -> 0.0035s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:bulk_import_exports, [:project_id, :relation], {:unique=>true, :where=>"project_id IS NOT NULL", :name=>"partial_index_bulk_import_exports_on_project_id_and_relation", :algorithm=>:concurrently})
   -> 0.0012s
-- add_index(:bulk_import_exports, [:project_id, :relation], {:unique=>true, :where=>"project_id IS NOT NULL", :name=>"partial_index_bulk_import_exports_on_project_id_and_relation", :algorithm=>:concurrently})
   -> 0.0024s
== 20210414131807 AddBulkImportExportsTableIndexes: migrated (0.0099s) ========

== 20210414133310 AddBulkImportExportUploadsTable: migrating ==================
-- create_table(:bulk_import_export_uploads, {})
-- quote_column_name(:export_file)
   -> 0.0000s
   -> 0.0064s
-- quote_table_name("check_5add76239d")
   -> 0.0000s
-- quote_table_name(:bulk_import_export_uploads)
   -> 0.0000s
-- execute("ALTER TABLE \"bulk_import_export_uploads\"\nADD CONSTRAINT \"check_5add76239d\" CHECK (char_length(\"export_file\") <= 255)\n")
   -> 0.0008s
== 20210414133310 AddBulkImportExportUploadsTable: migrated (0.0110s) =========
Down ``` == 20210414133310 AddBulkImportExportUploadsTable: reverting ================== -- drop_table(:bulk_import_export_uploads) -> 0.0031s == 20210414133310 AddBulkImportExportUploadsTable: reverted (0.0032s) =========

== 20210414131807 AddBulkImportExportsTableIndexes: reverting ================= -- transaction_open?() -> 0.0000s -- indexes(:bulk_import_exports) -> 0.0023s -- execute("SET statement_timeout TO 0") -> 0.0005s -- remove_index(:bulk_import_exports, {:algorithm=>:concurrently, :name=>"partial_index_bulk_import_exports_on_group_id_and_relation"}) -> 0.0027s -- execute("RESET ALL") -> 0.0006s -- transaction_open?() -> 0.0000s -- indexes(:bulk_import_exports) -> 0.0011s -- remove_index(:bulk_import_exports, {:algorithm=>:concurrently, :name=>"partial_index_bulk_import_exports_on_project_id_and_relation"}) -> 0.0014s == 20210414131807 AddBulkImportExportsTableIndexes: reverted (0.0105s) ========

== 20210414130526 AddForeignKeyToBulkImportExportsOnGroup: reverting ========== -- remove_foreign_key(:bulk_import_exports, {:column=>:group_id}) -> 0.0045s == 20210414130526 AddForeignKeyToBulkImportExportsOnGroup: reverted (0.0092s) =

== 20210414130017 AddForeignKeyToBulkImportExportsOnProject: reverting ======== -- remove_foreign_key(:bulk_import_exports, {:column=>:project_id}) -> 0.0073s == 20210414130017 AddForeignKeyToBulkImportExportsOnProject: reverted (0.0162s)

== 20210414100914 AddBulkImportExportsTable: reverting ======================== -- drop_table(:bulk_import_exports) -> 0.0031s == 20210414100914 AddBulkImportExportsTable: reverted (0.0031s) ===============

</details>


## Does this MR meet the acceptance criteria?

### Conformity

- :clipboard: [Does this MR need a changelog?](https://docs.gitlab.com/ee/development/changelog.html#what-warrants-a-changelog-entry)
  - [x] I have included a changelog entry.
  - [ ] I have not included a changelog entry because _____.
- [ ] [Documentation](https://docs.gitlab.com/ee/development/documentation/workflow.html) ([if required](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#when-documentation-is-required))
- [ ] [Code review guidelines](https://docs.gitlab.com/ee/development/code_review.html)
- [ ] [Merge request performance guidelines](https://docs.gitlab.com/ee/development/merge_request_performance_guidelines.html)
- [ ] [Style guides](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/development/contributing/style_guides.md)
- [ ] [Database guides](https://docs.gitlab.com/ee/development/database_review.html)
- [ ] [Separation of EE specific content](https://docs.gitlab.com/ee/development/ee_features.html#separation-of-ee-code)

### Availability and Testing

<!-- What risks does this change pose? How might it affect the quality/performance of the product?
What additional test coverage or changes to tests will be needed?
Will it require cross-browser testing?
See the test engineering process for further guidelines: https://about.gitlab.com/handbook/engineering/quality/test-engineering/ -->

<!-- If cross-browser testing is not required, please remove the relevant item, or mark it as not needed: [-] -->

- [ ] [Review and add/update tests for this feature/bug](https://docs.gitlab.com/ee/development/testing_guide/index.html). Consider [all test levels](https://docs.gitlab.com/ee/development/testing_guide/testing_levels.html). See the [Test Planning Process](https://about.gitlab.com/handbook/engineering/quality/test-engineering).
- [ ] [Tested in all supported browsers](https://docs.gitlab.com/ee/install/requirements.html#supported-web-browsers)
- [ ] Informed Infrastructure department of a default or new setting change, if applicable per [definition of done](https://docs.gitlab.com/ee/development/contributing/merge_request_workflow.html#definition-of-done)

### Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in [the security review guidelines](https://about.gitlab.com/handbook/engineering/security/#when-to-request-a-security-review):

- [ ] Label as ~security and @ mention `@gitlab-com/gl-security/appsec`
- [ ] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [ ] Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov

Merge request reports