Skip to content

Add support for setting the artifact/cache compression level

Arran Walker requested to merge ajwalker/compression-level into master

What does this MR do?

Allows ARTIFACT_COMPRESSION_LEVEL and CACHE_COMPRESSION_LEVEL to be set to

  • fastest
  • fast
  • default
  • slow
  • slowest

to control the compression ratio/performance of the active archiver.

It's intended that not every archiver has to support these different levels, but they made a "best effort" to apply them. Therefore, an unsupported level has no warning.

Only fastzip currently supports these levels, so they will only be useful in combination with FF_USE_FASTZIP for now.

Why was this MR needed?

Adjusting compression levels can allow users to cater for their environment and data: Disk, network, memory and CPU performance all play a part in what is overall faster: better compression or faster/no compression.

What's the best way to test this MR?

Compiling the binary and running cache-archiver at different levels should produce archives at different sizes and speeds.

time FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=fastest ./gitlab-runner cache-archiver --path ".git/**" --file fastest.zip 2> /dev/null && du -h fastest.zip

time FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=fast ./gitlab-runner cache-archiver --path ".git/**" --file fast.zip 2> /dev/null && du -h fast.zip

time FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=default ./gitlab-runner cache-archiver --path ".git/**" --file default.zip 2> /dev/null && du -h default.zip

time FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=slow ./gitlab-runner cache-archiver --path ".git/**" --file slow.zip 2> /dev/null && du -h slow.zip

time FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=slowest ./gitlab-runner cache-archiver --path ".git/**" --file slowest.zip 2> /dev/null && du -h slowest.zip

Example output:

FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=fastest ./gitlab-runner 2  0.16s user 0.23s system 113% cpu 0.342 total
189M	fastest.zip

FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=fast ./gitlab-runner cache-archiver   0.53s user 0.86s system 179% cpu 0.775 total
188M	fast.zip

FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=default ./gitlab-runner 2  0.62s user 0.91s system 186% cpu 0.821 total
188M	default.zip

FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=slow ./gitlab-runner cache-archiver   2.23s user 1.35s system 179% cpu 1.996 total
193M	slow.zip

FF_USE_FASTZIP=1 CACHE_COMPRESSION_LEVEL=slowest ./gitlab-runner 2  2.36s user 1.37s system 181% cpu 2.059 total
193M	slowest.zip

What are the relevant issue numbers?

closes #4907 (closed)

Edited by Arran Walker

Merge request reports