Export/import group/subgroup/project hierarchies with a single function
GitLab recently gained a group export feature, but this is documented not to export projects, only nested group hierarchies: https://docs.gitlab.com/ee/user/group/settings/import_export.html#exported-contents
GitLab has long had project import/export.
It would be great to have nested group/project export/import to create single artifacts for large groups with many projects.
There is another wrinkle to this request though. The gitlab.com infrastructure team sometimes use project export from single-node gitlab omnibus instances whose postgres DB is a PITR snapshot of our production database right before the given project was accidentally deleted. Docs: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/deleted-project-restore.md
As a large GitLab instance, production uses many Gitaly shards. Our "restoration instances" naturally don't have access to our production gitaly shards, and the production DB metadata therefore references shards and paths that don't exist according to the gitlab rails app.
So a high-level view of the idealized process for deleting an accidentally-deleted group would be:
- Create a "restoration instance" - a single node gitlab omnibus instance whose DB is a snapshot of prod's at some point in the past.
- The infrastructure team can already do this quickly.
- Run a single rails console command to output an artifact containing metadata for the whole nested group/project hierarchy, even though the project metadata references gitaly shards that can't be accessed, and paths that don't exist.
- Our current process would involve creating a single group artifact, and potentially hundreds of project artifacts, and orchestrating their export/import out-of-band or manually.
- Even project artifacts currently require manual manipulation to work around the DB-gitaly mismatch during project restoration.
- On a production rails console, run a single command to import this artifact.
- production now has the whole metadata hierarchy, but all repos don't exist.
- The infrastructure team will orchestrate restoration of repos to their metadata-expected paths on each gitaly shard via gitaly disk snapshot-restore.
This issue would help a huge amount with https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11148.