Artifact as a Proto: rework

Description

This MR, changes how artifacts are used internally, using the new artifact proto and services. As this is a refactor no new tests are added, and it's mainly internal changes (predominantly to _artifact.py, _artifactcache.py and cascache.py).

The new artifact protos are stored in an $XDG_CACHE_HOME/artifacts/refs directory, with the protos pointing to various files in the CAS (resulting files, buildtree, log files and public data file).

Main changes proposed in this merge request:

• All code dealing with previous artifact directory structure has been changed to use the artifact proto. This includes remote methods in artifact cache, that use the artifact service along with CAS service to pull and push artifacts.

• As buildtrees can be handled more easily, code dealing with subdirectories (and excluding them) has been removed, now just explicitly downloading (or not) the buildtree directory.

• Code for cache cleanup has been changed as it's not as simple as following refs, so involves callbacks in CASCache that mark related directories and single files in the CAS that are part of an artifact. This allows pruning of blobs that aren't in present artifact or sources, allows clean to work removing older artifacts or sources not used by the current build plan. Callbacks registered with CASQuota used in clean() are now:

  • remove callbacks: a tuple which iterates over unrequired refs (artifact or sources) with both their time and name, and the method to remove them.
  • list refs: list refs, and just use to present to the user the total amount of refs in user messages.

  Callbacks with CASCache used for prune() are:

  • add reachable directory: add iterator over directory digests reachable from artifacts.
  • add reachable digests: add iterator over file digests reachable from artifacts.

This merge request, when approved, will close: #974 (closed)

---