CI: Optimize cache manipulation using rsync
Description
Fetching changes...
Removing .jekyll-metadata
Removing .sass-cache/
Removing _site/
...
Checking cache for default...
Successfully extracted cache
This basically means that Gitlab CI deletes some 90 MB from _site/
directory and then reconstructs almost the same 90 MB from the cache. My build takes about 26 seconds, of which jekyll runs for about 0.5 second = 1.9 % efficiency. That is huge waste of resources.
I use rsync to deploy compiled _site
to my website and it takes about 0.01 seconds to check for changes. Since there are mostly only small changes, it takes about same time to deploy them. Therefore, use of rsync for cache manipulation should speed up things about 1200× \bigl(={26-0.5 \over 2\times 0.01}\bigr)
in my case.
Proposal
Instead of deleting files and copying them back from cache, use rsync -ar --delete $cache_dir $build_dir
to sync files with cache. It should save a lot of time and reduce load on build servers.
rsync
is widely available, so such dependency should be easy to satisfy.