AaaP: Rework artifact class to use artifact proto

Background

As part of #909 (closed), we want to change the internals of buildstream to use the artifact proto instead of dealing with looking in different directories for files, buildtrees, logs etc. This is dependent on both !1252 (merged) and !1259 (merged).

Task description

A few different places will need to be updated for this to work. Predominantly the Artifact class as we have moved most of the logic in there. This is probably an incomplete list that will be updated as more parts are found.

• Change Artifact class methods to use the artifact proto instead of directories. Pretty much every method bar _get_cache_key will need to change, as this predominantly deals with current directory structure of artifacts.

• Change appropriate ArtifactCache methods.

  • The pull and push methods will need to change as it has to upload the different components, using CASCache.push_directory and then push the artifact proto. The subdir and exclude_subdir options aren't necessary now.
  • contains will need to deal with checking that files are actually present
  • list_artifacts
  • remove will need to follow files/logs/buildtree etc. and remove them
  • get_artifact_directory/get_artifact_logs
  • commit will need to deal with creating the artifact proto - This isn't actually used anyway

• Element

  • __pull_directories method can be removed
  • __pull_strong/__pull_weak don't need to specify subdir and exclude_subdirs`
  • _pull_pending needs to be updated to more sensibly check for buildtree presence.

• CASCache

  • _reachable_refs_dir will need to change as it can't follow refs that point to artifact protos. This is artifact specific, so this might need more thought as we don't want to put artifact specific logic here.
  • CASQuotas clean will also need to change as it has to be able to remove unrequired source refs and artifact protos. This will probably require a change to how callbacks are registered with CASQuota

• TestArtifact methods that deal with the artifact directory structure will need to change

  • remove_artifact_from_cache
  • is_cache
  • get_digest
  • extract_buildtree

• Any tests that are affected by these changes

Acceptance Criteria

Buildstream should behave identically, so switch over to the artifact proto with a passing test suite.

---