Source cache

Description

As described in #440 (closed) and ML discussion, a source cache will be used, that keeps sources staged in the local CAS. The architecture will mirror the artifact cache, with sources querying the source cache, which will check local and then configured remote CAS's for presence of source.

Changes proposed in this merge request:

• - Introduced a BaseCache class that both ArtifactCache and SourceCache derive from, mainly for allowing sharing of related CASCache/CASRemote methods.

• Source cache

  • - Introduce new STAGED and BOTH consistency for when sources are in local CAS, and both local CAS and source directory. Better suggestions for names for this are welcome.
  • - get_source_fullname: mirrors get_artifact_fullname and used for creating refs that point to the sources in the CAS.
  • - commit - Add a source to the cache.
  • - export - Get a source from the cache, used when building elements.
  • - Want to avoid regular file system where possible and favor virtual CasBasedDirectorys.
  • - Move clean methods into the CASQuota class.
  • - Config for remote source caches

• Add tests

  • - Test sources are staged into the local CAS.
  • - Check CAS based sources are used where available.
  • - Check sources aren't fetched when only present in the local CAS.
  • - Check sourcer are fetched when needed for workspaces, even when in local CAS.

The push and pull methods have now been put in a separate MR !1214 (merged)

optimizations / tech debt

There's some other bits that may want to be left to technical debt issues later:

• Split ref directories: At the moment I've put source refs just in the refs/heads directory, but we probably want to separate this so that we have a refs/heads directory for artifacts, and refs/sources for sources. This will need some changes at the CASCache level which might end up touching a fair amount of the code base. NB. this may not be necessary with upcoming artifact as a proto work.
• CAS filesystem: As buildstream moves to a more CAS based work flow, the source cache can avoid using regular filesystem more. For example at the moment, sources must stage to a real directory but hopefully that can be a virtual directory later.
• Source caching: At the moment sources are all staged and then cached together, due to some source kinds like patch, requiring previous sources for them to make sense. An additional flag could be added to source (similar to BST_REQUIRES_PREVIOUS_SOURCES_TRACK/FETCH but for staging), which allows sources to be cached separately where possible.

This merge request, when approved, will close:

#440 (closed)