Skip to content

CI: fast initializing `fdroid build` job

yzqzss requested to merge yzqzss/f-droid-data:fast-build into master

Reduced the fdroid build by two minutes!

I discovered the Getting source from Git repository step to be too time-consuming (took ~2m20s) I did some experiments and realized that the reason is mainly:

  • There are too many dirs&files under metadata/, and the IO limit of the GitLab runner is about 1000 IOPS (?). which caused the git checkout to be very slow.

    $find metadata/ | wc -l # it's 120692 !

  • The gitlab CI uses fetch --depth=5000 to pull the last 5000 commits by default, which also slows things down.

I've optimized it as follows:

  • Disable CI/CD built-in git clone/fetch (Getting source from Git repository)
  • Put the .git/ and metadata/ directories on ramdisk.
  • Cache the .git/ directory
  • Use git fetch --filter=tree:0 to fetch the entire commit history while only actually downloading the individual commit we want.
  • Enable fastzip to speed up cache de/compression

It's worth noting that since GitLab's CI/CD cache block runs before the script block. In order to unpack the cache after .git/ ramdisk is loaded, I had to tarball it as gitdir.tar.

Performance:

Edited by yzqzss

Merge request reports