Reimplement support for incremental workspace builds
Reimplement support for incremental workspace builds using CAS, allowing it to work with buildbox-run
and remote execution. This is a replacement for !1726 (closed).
Background (see also #985 (closed)): We're using the CAS-based source cache for workspaces for a few months now, completely separating the sources in the workspace directories from build files such as .o
object files. !1563 (merged) introduced this but had to drop support for incremental workspace builds as that required more work in various components. All the prerequisites for reimplementing (CAS-based) incremental workspace build support have been merged into BuildBox components and BuildStream (a large part was file timestamp support, !1761 (merged) among others).
This MR completes the task by applying source changes in the workspace directory since the previous build on top of the buildtree of the previous build (using !1769 (merged)) and then performing an incremental build, if possible (previous build artifact cached and no change in dependencies).
A new mechanism is used to keep track of whether configure
has already been called. The previously used prepared
flag is difficult to support with command batching as used with REAPI. This branch introduces an empty marker file .bst-prepared
as replacement, which can also be reliably created as part of a command batch. As the marker file exists in the buildtree in the artifact, we no longer need to store the flag in the workspace YAML file.
This branch also includes an extension to the Artifact
proto to store the sources used for the build. The new field is set whenever we're caching the buildtree. We also need the buildtree for incremental workspace builds and storing the sources should be very cheap if we already store the buildtree.