replace gzip with pigz to make backups a lot faster
A user on IRC complained yesterday that we use gzip in our backup task.
He complained because gzip only uses 1 core.
To speed up the backup process on multicore machines I propose to use pigz instead of gzip.
Info about pigz: http://www.zlib.net/pigz/
It is a drop in replacement for gzip and it decreases the time needed for the backup creation a lot!
I benchmarked it a bit with my eclipse installation because i didn't have a gitlab backup lying around and got the following results:
(same commands that we use in gitlabs backup script)
time tar -C ./eclipse/ -cf - . | gzip -c -1 >backup_gzip.tar
real 0m17.750s user 0m17.255s sys 0m1.145s
time tar -C ./eclipse/ -cf - . | pigz -c -1 >backup_pigz.tar
real 0m5.635s user 0m16.333s sys 0m0.801s
As you can see this speeds up the time considerably.
The downside is, that we either have to bundle pigz in the omnibus install or require it as a dependency.
It is in the repos of most distributions, but not installed by default.