API endpoint for exporting project

There are situations where it may be useful to generate an export of a project via API endpoint. Ideally this could also be imported via an API to a different instance.

Proposal

• Add API endpoint to trigger preparing project export
• Add API endpoint to check the status of the export job and retrieve download URL when read

Links / references

added ~480950 label

@balasankarc How would it work? Initiate the export and email the user that started it like in the web UI?

@markglenfletcher Probably, yes. Dev team may have better ideas.

Funnily enough I also have need this feature after a discussion with my colleagues earlier today.

We have two separate Gitlab instances with ~300 and ~120 projects respectively and they need to be merged onto a single server. I'm not really keen on the idea of manually exporting 120 projects and importing them onto the other server by hand.

Having an API for exporting and importing would make this process significantly easier.

I could use this feature, too. We need this to backup single projects (repo and wiki can be mirrored, but the issues and other metadata not).

An existing export can be downloaded using a private token. But it is not possible to trigger the export via a script/API. I tried with curl and a PRIVATE_TOKEN header, but it didn't work. Additionally one would need a possibility to check, if an export is ready for download.

We could also use this feature. We have a need to backup our projects with issues and all on a server we don't control so we can't back up the entire database.

I could really use this feature. I've got a bunch of different GitLab instances spread out everywhere in different environments, and now I have this new GitLab environment with enough resources to host everything, and I want to consolidate. It's a huge pain to do it all by hand right now.

This feature would be really awesome to have. I have the same use case as @Xiol described (also with approx. 300 projects), however as we have to run the migration within the next 2 weeks, I will most likely have to do it manually.

In case anybody found a solution for automating parts of project import / export advice or snippets would be highly appreciated ;)

We also could need that feature. We want to first archive old projects and then after some time backup and delete them to free some space and reduce the total number of projects.

So, I don't know if there are consequences to this or not so do it at your own risk, but I did a little digging around and was able to find a way to create an export for all the projects with the following script. Run it with gitlab-rails console (I'm running this on version 9.4.5):

Create exports for all projects:

group = Group.find_by_full_path("GROUP_FULL_PATH_HERE")
admin_user = User.find_by(username: "ADMINISTRATOR_USER_HERE")
group.projects.each do |project|
  puts "Executing export for #{project.full_path}"
  exporter = Projects::ImportExport::ExportService.new(project, admin_user)
  exporter.execute
end; nil

Of course, you can do find_by_id or something as well if you prefer.

Download exports for all the same projects (pseudocode since I didn't feel like typing out the script as this is probably going to be defined on your own taste and language preference) -- second option would be to add another loop in the ruby code to print the project names, put it in a text file, then read the text file one by one to avoid parsing an API call:

# Pseudocode
# projectslist = hit API for list of projects on group
# for each project in projectslist
# projectname = project.name
# projecturl = project.web_url
curl -o projectname.tar.gz --header "PRIVATE-TOKEN: token" -X GET "${projecturl}/download_export"
# end for loop

@jramsay What do you think about opening this one for a Community Contribution with ~"Accepting Merge Requests"?

mentioned in issue #39140 (moved)

Sounds good to me.

added [deprecated] Accepting merge requests label

In order to get this request some attention, this issue has been marked as a potentially interesting proposal as it meets the following criteria:

• Labelled as a feature proposal
• More than 10 upvotes
• Unscheduled (not associated with a milestone)

Thanks for your proposal!

/cc @markglenfletcher

added auto updated potential proposal labels

added DEPRECATED_project export and removed DEPRECATED_import labels

changed the description

@markglenfletcher I have some initial prototyping put together and would like to tackle this!

Proposed endpoints

Start a new export:

POST /projects/:id/export

Check status of export:

GET /projects/:id/export

Download finished export:

GET /projects/:id/export/download

Remove existing exports and start a new export:

PUT /projects/:id/export

Remove existing exports:

DELETE /projects/:id/export

Notes

• Email notification is baked into the export service so that would also carry over into the API.
• The current export functionality does not appear to limit the queue of exports. (e.g. calling export repeatedly will stack up as many export processes and complete them all sequentially)
• The import process currently uses a state machine and tracks the existing import with project.import_jid. A similar state machine and project.export_jid could be helpful for exports.
• The UI provides two different button states for exporting: [Export project] and [Download export, Generate new export].
  • Export project generates a new export.
  • Generate new export removes any existing exports then generates a new export.
• There appears to be an unused action projects_controller.rb#remove_export
• PUT and DELETE methods seem of little end-user value.
• In progress status can be determined by existence of File.join(project.disk_path, 'work') directory.

Questions

• With or without PUT and DELETE … should POST remove existing exports or leave in place?
• What entity information would be helpful from these endpoints?

Related

• #3050 (closed) Add Project Import/Export Functionality
• !3114 (merged) Export project functionality