Fix caching in CI and some other optimizations

mentioned in merge request !62 (closed)

approved this merge request

Nice! I find YAML anchors a bit eye-bleeding but it's probably worth it for the speedups. And we get to remove it all as soon as we can deprecate YBD :)

@samthursfield I don't disagree :)

Out of interest, does buildstream have a mechanism to remove potentially stale artefacts to save disk space? I'm hitting errors from full disk, but wonder if this is because the runner is full of ybd cache.

It might be worth me setting up runners that are specifically for buildstream pipelines.

There's not a convenient cache expiry method in BuildStream at the moment. I tend to just delete the whole cache when my disk gets full.

@samthursfield mmm, that can be a problem: the runners are persisting between builds so if a cleanup is not done automatically maybe we have to add a manual step for the bst pipelines

@jjardon Hence my thinking that we perhaps separate the instances for bst and ybd.

It's probably an hour's hack to get together a cache expiry script, it basically just needs to:

rm -Rf cache/build
rm -R cache/logs
rm -R artifacts/extract

and then the trickier part, enter the artifacts/ostree directly and use ostree prune --keep-younger-than=DATE to remove commits until there's enough disk space free again.

file a buildstream bug and i can look at it

Wait, I'm being dumb. We are only persisting the BuildStream source cache between builds. That should be of relatively fixed size -- it'll only grow when we add new sources to the definitions.