Introduce tracking of individual sources, using the Remote Asset API

We can reduce the load and reliance on additional services by leveraging the Remote Asset API. For example, instead of having all clients poll git services, the FetchService.FetchDirectory API is used to resolve the commit at a certain branch. As clients all track to the same revision, in cache hits are more likely for sources, artifacts and actions.

To support this we need to extend the Source Plugin API to return the list of URIs and qualifiers as needed by the FetchService. Specifically:

  • FetchDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
  • FetchDirectoryRequest.qualifiers is the minimal set of qualifiers that identifies the source. This must exclude the commit for the branch. For example:
    • vcs.branch = master
    • resource_type = application/x-git

In the response the client will learn the Digest of the source as well as all other qualifiers the service knows about. This would include identifying information the source plugin would use in its ref. For example:

  • FetchDirectoryResponse.uri is the URL that matched, that represent the content of the source. For example: git.example.com/foo/bar.git.
  • FetchDirectoryResponse.qualifiers is the complete set of qualifiers associated with the source. For example:
    • vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
    • resource_type = application/x-git
    • vcs.branch = master

Behavior should be configurable to support the following use cases:

  • client does not use the Remote Asset API to track sources, and only uses the source plugin native track
  • client uses the Remote Asset API to track sources, and falls back to using the source plugin native track
  • client uses the Remote Asset API to track sources, and does not fall back to using the source plugin native track
  • client uses the Remote Asset API to push sources, after using the source plugin native track and fetch
  • client does not use the Remote Asset API to push sources
  • client is configured to only accept results younger than a certain age.

See also:https://mail.gnome.org/archives/buildstream-list/2020-February/msg00000.html

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information