Skip to content

Enhance gitlab:backup:create by using pigz or other multi-core compressor

My organization performs daily backups of our gitlab instance by running gitlab-rake gitlab:backup:create CRON=1 and then copying the result to backup storage. I have noticed that for much of the time, the backup script is bottlenecking on gzip, which in top can be seen burning a single CPU core.

There are compression tools that can utilize multiple cores while still getting similar compression ratios for large data sets. One commonly used one that is a drop-in replacement for gzip is pigz.

It would be cool if the rake script could check to see if pigz is installed on the target system, and if so, use it instead of gzip. The command line API is the same.

git      32727  1.1  1.4 1095796 458348 ?      Sl   00:00   9:48          |   \_ /opt/gitlab/embedded/bin/rake gitlab:backup:create CRON=1
git       6036  5.5  0.0  51904  4976 ?        S    00:22  45:00          |       \_ tar --exclude=lost+found --exclude=./tmp -C /var/opt/gitlab/gitlab-rails/shared/artifacts -cf - .
git       6037 95.7  0.0  31644  3644 ?        R    00:22 780:32          |       \_ gzip -c -1