Skip to content

Allow -server-side flag to be set on gitaly-backup

James Fargher requested to merge integrate_serverside_repository_backups into master

What does this MR do and why?

Closes gitaly#4941 (closed)

Adds a new env to backup.rake REPOSITORIES_SERVER_SIDE that triggers repository backups to be taken server-side.

gitaly-backup docs

Not much can be done in the way of testing because gitaly makes the actual object storage connection. So cannot be mocked.

Docs will be handled separately. It will actually be easier to write docs once this is merged. #417630 (closed)

How to set up and validate locally

  1. Switch gitlab to this feature branch.

    $ git switch integrate_serverside_repository_backups

  2. Switch GDK to the server-side backups branch:

    $ cd <gdk_root>
$ git switch gitaly_server_side_backups

  3. Turn on object storage. gdk.yml:

    object_store:
  enabled: true

  4. Reconfigure and restart.

    $ gdk reconfigure && gdk restart

  5. Open minio in a browser and create a bucket named gitaly-backups if one doesn't already exist.

  6. Back in gitlab, create a server-side backup (skip db so it's faster and less invasive):

    $ cd <gdk_root>/gitlab
$ bundle exec rake gitlab:backup:create REPOSITORIES_SERVER_SIDE=true SKIP=db
...
2023-08-09 03:02:04 UTC -- Backup 1691550107_2023_08_09_16.3.0-pre is done.
2023-08-09 03:02:04 UTC -- Deleting backup and restore PID file ... done

  7. You can browse minio and verify that a bunch of stuff was added 🎉 🎉

  8. Try restore from said backup:

    $ bundle exec rake gitlab:backup:restore BACKUP=1691550107_2023_08_09_16.3.0-pre
...

  9. The output will have a lot of "skipped" errors. This is just how repositories that don't exist are handled at the moment. Hopefully there will be some successful ones, they'll print completed restore:

    {"command":"restore","gl_project_path":"jashkenas/Underscore","level":"info","msg":"completed restore","relative_path":"@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git","storage_name":"default","time":"2023-08-09T03:06:48.899Z"}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by James Fargher

Merge request reports