We are trying to deploy gitlab with separate gitaly that handle the repositories storage, as a transition from gitaly over NFS. We are looking for the best way to do the daily backup of our repositories. Our planned strategy is: - We use AWS EBS as a storage volume for gitaly. (or any incremental snapshot capable xBS from any cloud provider) - At snapshot time, we do a first snapshot of the volume, without shuting down gitaly. This can take a dew minutes depending on the activity of the day, but the snapshot will be incoherent, as the filesystem might still be writting stuff in the block system. - We freeze gitaly activity, and flush the filesystem to disk - We make a second snapshot, which this time is garanteed to be coherent. The snapshot only copies the blocks that are changed since first snapshot, so this *should* be pretty quick (first tests in AWS console is not measurable < time to refresh page) In order to implement that, we need a way to tell gitaly to hold any write operation for a little while. This would allow the backup to happen while introducing very little extra latencies to a few operations. In this something you have in place already? Another option is to kill or offline the gitaly server for a few seconds, and count on GRPC backoff retries to absorb the small offline period, so that it is not visible to frontend users. e.g: ```bash # first snapshot which might take a while as it backups the load of the day fsync snapshotId=$(aws ec2 create-snapshot --volume-id vol-xxxx --description "gitaly-warmup-${date '+%Y%m%d%H%M%S'}" --output json | jq -r ".SnapshotId") aws ec2 wait snapshot-completed --snapshot-ids $snapshotId iptables -I INPUT -p tcp --dport 8075 --syn -j DROP # with hope that tcp is better at exp backoff retry than grpc gitlab-ctl gitaly stop fsync # second snapshot should be much faster as it only backups the writes done while the first backup was ongoing snapshotId=$(aws ec2 create-snapshot --volume-id vol-xxxx --description "gitaly-${date '+%Y%m%d%H%M%S'}" --output json | jq -r ".SnapshotId") aws ec2 wait snapshot-completed --snapshot-ids $snapshotId gitlab-ctl gitaly start iptables -D INPUT -p tcp --dport 8075 --syn -j DROP ```