Skip to content

Add project uploads export to Relations Export API

What does this MR do and why?

this feature is behind bulk_import_projects feature flag which is off by default (and on .com).

This MR adds project upload export to the relations export API https://docs.gitlab.com/ee/api/project_relations_export.html

It's needed in order to support project uploads migration when using Group Migration feature that ~"group::import" is working on, which allows users with one click to migrate their groups, subgroups & projects https://docs.gitlab.com/ee/user/group/import/

Since project uploads are a 'file' relation (vs json relation e.g. project issues), which is not just text but binary files, we need to archive them before compressing. Current relations export API does only compression (e.g. issues.ndjson -> issues.ndjson.gz). Since project uploads are a collection of files, we need to tape them together using tar before compression.

Additional information can be found here #339401 (closed) specifically on reasons for this implementation. One other implementation was considered to export uploads.ndjson file with a list of URLs to each upload, so that we can just download them directly from source GitLab instance, instead of packing them to the archive. However, this did not work for several reasons: we do not have Uploads API & non-image uploads require authentication with a session cookie and not available with an access token.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

  1. Create project with uploads & issues
  2. Open rails console & perform project uploads export
project = Project.find(id)
BulkImports::RelationExportService.new(User.first, project, 'uploads', nil).execute
project.bulk_import_exports.find_by_relation('uploads').upload.export_file.path

=> "/.../gitlab/public/uploads/-/system/bulk_imports/export_upload/export_file/275/uploads.tar.gz"
  1. Verify all (except avatar) uploads are exported
  2. Open rails console & perform project issues export
project = Project.find(id)
BulkImports::RelationExportService.new(User.first, project, 'issues', nil).execute
project.bulk_import_exports.find_by_relation('issues').upload.export_file.path

=> "/.../gitlab/public/uploads/-/system/bulk_imports/export_upload/export_file/274/issues.ndjson.gz"
  1. Verify all issues are still exported (to double check there is no regression)

-->

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports