CI performance
This makes the CI a three-stage pipeline, as discussed in https://gitlab.com/top-software/viia/-/issues/547:
-
build-image
builds a docker image and pushes it to our project-specific registry (takes ~4 minutes) -
build
builds the dependencies and the test programs (takes a little under 4 minutes) -
test
contains two jobs:-
test
, which is parallelised over two machines, runscleantest
tests (each takes 10-13 minutes) -
test-concurrency
for the concurrency test (takes a little over 5 minutes)
-
I have configured the Tilaa server such that at most 2 jobs run in parallel. This means that in the test
stage not all three jobs can run simultaneously on this server. As long as we have GitLab CI minutes left, a shared runner can take up one of the jobs. But without shared runners, this configuration takes around 25 minutes to complete, saving around 5 minutes (this is less than the 10 minutes you would expect from parallelisation of test
, presumably due to startup and caching overhead).
Because install_clean_nightly.sh bundle-complete
is done in build-image
, I rebuild the image on every build. And I use docker build --no-cache
because otherwise the apt-get
commands will be ignored and we will not have an up to date image. If we store a particular Clean date (like in VIIA's config/clean_date.sh) in this repository, we could rebuild only when that date is updated, reuse images across pipelines, and save most of the build-image
step.