Skip to content

Add relations export API for Projects

What does this MR do and why?

Relates to #340309 (closed)

Adds API endpoints

Everything is behind the bulk_import FF enabled by default.

The API endpoints will be ready with this MR, but the real intent of these endpoints is to allow projects to be migrated during the GitLab Group Migration. So, the API here is just a ground work for this. More work will be done on later MRs to support importing this data that we're making exportable here.

API endpoints being added:

  • POST /projects/:id/export_relations: export Project relations to compacted NDJSON files
  • GET /projects/:id/export_relations/status: check the status of the export
  • GET /projects/:id/export_relations/download: download the NDJSON files

Adds API docs

doc/api/project_relations_export.md

Finally, I'll take care of #340309 (comment 677591237) in a separate MR.

Screenshots or screen recordings

I've tested it with the bellow commands and got the below results

How to set up and validate locally

  1. Make sure FF is enabled on your GDK:

    Feature.enable!(:bulk_import)
  2. Create a project and add issues to it.

  3. Export the project relations and check the "accepted" response. Remember to set your private_token with your GitLab access token with the api scope.

    λ  gitlab git:(jcunha-add-relations-export-api-for-projects) ✗ curl --header "Content-Type: application/json" \
    --request POST \
    "https://gdk.joaocunha.eu:3443/api/v4/projects/24/export_relations?private_token=yrak6Tvi4NfGUyEbp7L3"
    {"message":"202 Accepted"}%
  4. Check the status of the export (it can take a while for it to finish):

    λ  gitlab git:(jcunha-add-relations-export-api-for-projects) curl --header "Content-Type: application/json" \
    --request GET \
    "https://gdk.joaocunha.eu:3443/api/v4/projects/24/export_relations/status?private_token=yrak6Tvi4NfGUyEbp7L3"
    [{"relation":"releases","status":1,"error":null,"updated_at":"2021-09-16T15:10:57.596Z"}, 
    {"relation":"project_feature","status":1,"error":null,"updated_at":"2021-09-16T15:11:01.787Z"}, 
    {"relation":"issues","status":1,"error":null,"updated_at":"2021-09-16T15:10:56.795Z"},
    ...
  5. Download the file

    λ  gitlab git:(jcunha-add-relations-export-api-for-projects) curl --header "Content-Type: application/json" \
    --request GET \
    --output issues.ndjson.gz \
    "https://gdk.joaocunha.eu:3443/api/v4/projects/24/export_relations/download?r 
    relation=issues&private_token=yrak6Tvi4NfGUyEbp7L3"
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
    100   678  100   678    0     0    623      0  0:00:01  0:00:01 --:--:--   623
  6. Decompress the file

    λ  gitlab git:(jcunha-add-relations-export-api-for-projects)gunzip issues.ndjson.gz
  7. Read the files contents

    λ  gitlab git:(jcunha-add-relations-export-api-for-projects)cat issues.ndjson
    {"id":551,"title":"New test issue","author_id":1,"project_id":24,"created_at":"2021-09-16T14:18:25.190Z","updated_at":"2021- 
    09-16T14:18:25.190Z","description":"export issue test description","iid":1,"updated_by_id":null,"weight":null,...}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #340309 (closed)

Edited by João Alexandre Cunha

Merge request reports