backup: Add support for incremental server-side backups
Context
Functionality to perform incremental backups exists, but is not yet invoked when executing server-side backups. This is because server-side backups are initiated via an RPC call which didn't include incremental
as a request option.j
This MR modifies the BackupRepositoryRequest
protobuf message to include the incremental
flag, and passes it through to the backup manager on the other end. Surrounding tests have been modified to exercise this option, but no net-new tests have been added to explicitly check that incremental
is being passed correctly.
Testing
This can easily be tested locally via GDK.
- Enable object storage by editing
gdk.yml
:
object_store:
enabled: true
$ gdk reconfigure && gdk restart
- Check that minio is running correctly. It listens on port 9002, but the hostname will differ depending on how you've configured DNS in GDK. For me, it runs at http://gdk.test:9002. Log in with username
minio
and passwordgdk-minio
. - Create a new bucket named
gitaly-backups
. - Create a full server-side backup using the Rake task and note the computed backup ID:
$ cd gitlab
$ bundle exec rake gitlab:backup:create SKIP=db REPOSITORIES_SERVER_SIDE=true
- Browse to
gitaly-backups/manifests/default/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git
(this SHA corresponds to thegitlab-org/gitlab-test
default project) and note the singletoml
file. It should look something like this:
object_format = 'sha1'
[[steps]]
bundle_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.bundle'
ref_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.refs'
custom_hooks_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.custom_hooks.tar'
- Make a small change to the
gitlab-org/gitlab-test
project, e.g. adding a new file. - Now, create an incremental server-side backup using the Rake task, using the previously computed backup ID as the baseline:
$ bundle exec rake gitlab:backup:create SKIP=db REPOSITORIES_SERVER_SIDE=true INCREMENTAL=yes PREVIOUS_BACKUP=1697498176_2023_10_16_16.5.0-pre
- Again, browse to
gitaly-backups/manifests/default/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git
and note an additionaltoml
file. The file should contain two steps and look something like this:
object_format = 'sha1'
[[steps]]
bundle_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.bundle'
ref_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.refs'
custom_hooks_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.custom_hooks.tar'
[[steps]]
bundle_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/002.bundle'
ref_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/002.refs'
previous_ref_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.refs'
custom_hooks_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/002.custom_hooks.tar'
The additional step confirms that an incremental backup of the gitlab-org/gitlab-test
repository has taken place. Since the backup artifacts were uploaded to object storage, this also confirms server-side backups were used.