Skip to content

Introduce caching of individual sources, using a Remote Asset API based Asset Cache

We can reduce the load and reliance on additional services by leveraging the Remote Asset API. CAS and Remote Asset API combined can serve as a cache of the content, with the additional benefit that having the content in CAS means it can be referred to in Remote Execution without additional uploads.

To support this we need to extend the Source Plugin API to return the list of URIs and qualifiers as needed by the FetchService. Specifically:

  • FetchDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
  • FetchDirectoryRequest.qualifiers is the minimal set of qualifiers that uniquely identifies the source. It's expected to be closely tied to the value of ref. For example:
    • vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
    • resource_type = application/x-git

We will need to expand the Source Plugin API similarly to return the list of URIs and qualifiers as needed by the PushService. Specifically:

  • PushDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
  • PushDirectoryRequest.qualifiers is the complete set of qualifiers associated with the source. For example:
    • vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
    • resource_type = application/x-git
    • vcs.branch = master

Behavior should be configurable to support the following use cases:

  • client does not use the Remote Asset API to fetch sources, and only uses the source plugin native fetch
  • client uses the Remote Asset API to fetch sources, and falls back to using the source plugin native fetch
  • client uses the Remote Asset API to fetch sources, and does not fall back to using the source plugin native fetch
  • client uses the Remote Asset API to push sources, after using the source plugin native fetch
  • client does not use the Remote Asset API to push sources

See also: https://mail.gnome.org/archives/buildstream-list/2020-February/msg00000.html

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information