Add concurrency support for Git repository backups
Problem to solve
sudo gitlab-backup create, backing up Git repositories is very slow. One reason for this is that repositories are backed up one at a time.
Presuming sufficient CPU and memory, this could reduce backup time significantly.
Add a new option to
GIT_BUNDLE_CONCURRENCY to allow concurrent Git bundle commands to run.
- The default value
0would mean be the current serialized behavior
- A value of
1...nwould enable concurrency
When concurrency is enabled with a value of
4 this would mean:
- each Gitaly storage would allow a maximum of 4 repos to be bundled at the same time
- in a typical single storage/shard configuration this would mean 4 bundles being generated in parallel
- in a 3 shard configuration this would mean 12 (= 4 concurrency x 3 shards) bundles being generated in parallel