Partial, file-free local CAS
Background
This is a "WIP" issue and the description may change.
In an environment which makes heavy use of remote execution, it is not necessary to have all the files for an element on the local machine, only the directory structure to enable staging. We want to modify local CAS to make the file contents optional.
Original mailing list post which started this: https://mail.gnome.org/archives/buildstream-list/2018-November/msg00022.html
The expected behaviour of partial CAS is:
- The default behaviour will be that all runtime dependencies of a built artifact will end up on the machine which ran
bst build
, such that the user can do a checkout, shell or further local builds without any network access. - We may add an option which removes that requirement for builds.
- When pushing an artifact to an artifact cache, or pushing source directories to a remote execution service, it is still the local
bst
's responsibility to ensure all file contents are on that server and push them itself if necessary.
When pushing, we will now check our full list of expected objects against the server; if the server reports missing blobs in there and we do not have it, we have to query all remotes to obtain it. This means pushing now may involve pulling.
Task description
-
In Element
/ArtifactCache
check for the presence of all file blobs of an artifact instead of only checking for the ref, i.e., triggerpull()
on partial artifact -
Make sure pull()
can complete a partial artifact -
Add support for completing a partial artifact in SandboxRemote
-
Mandate all elements/plugins to support virtual directories when remote execution is configured (drop fallback to local builds) -
Consider artifacts with missing file blobs as cached when remote execution is configured and the element is not a runtime dependency of the build target -
Skip pulling file blobs from artifact cache when remote execution is configured and the element is not a runtime dependency of the build target -
Skip fetching output files from remote execution CAS server if the element is not a runtime dependency of the build target and the files are not required for artifact pushing
Acceptance Criteria
- The goal is to reduce network bandwidth and speed up builds. We should provide some evidence that this works.
Edited by Jürg Billeter