Enhance gitlab:backup:create by using pigz or other multi-core compressor
My organization performs daily backups of our gitlab instance by running gitlab-rake gitlab:backup:create CRON=1
and then copying the result to backup storage. I have noticed that for much of the time, the backup script is bottlenecking on gzip
, which in top
can be seen burning a single CPU core.
There are compression tools that can utilize multiple cores while still getting similar compression ratios for large data sets. One commonly used one that is a drop-in replacement for gzip
is pigz
.
It would be cool if the rake script could check to see if pigz
is installed on the target system, and if so, use it instead of gzip
. The command line API is the same.
git 32727 1.1 1.4 1095796 458348 ? Sl 00:00 9:48 | \_ /opt/gitlab/embedded/bin/rake gitlab:backup:create CRON=1
git 6036 5.5 0.0 51904 4976 ? S 00:22 45:00 | \_ tar --exclude=lost+found --exclude=./tmp -C /var/opt/gitlab/gitlab-rails/shared/artifacts -cf - .
git 6037 95.7 0.0 31644 3644 ? R 00:22 780:32 | \_ gzip -c -1