Skip to content

Reimplement support for incremental workspace builds

Jürg Billeter requested to merge juerg/incremental-workspace-build into master

Reimplement support for incremental workspace builds using CAS, allowing it to work with buildbox-run and remote execution. This is a replacement for !1726 (closed).

Background (see also #985 (closed)): We're using the CAS-based source cache for workspaces for a few months now, completely separating the sources in the workspace directories from build files such as .o object files. !1563 (merged) introduced this but had to drop support for incremental workspace builds as that required more work in various components. All the prerequisites for reimplementing (CAS-based) incremental workspace build support have been merged into BuildBox components and BuildStream (a large part was file timestamp support, !1761 (merged) among others).

This MR completes the task by applying source changes in the workspace directory since the previous build on top of the buildtree of the previous build (using !1769 (merged)) and then performing an incremental build, if possible (previous build artifact cached and no change in dependencies).

A new mechanism is used to keep track of whether configure has already been called. The previously used prepared flag is difficult to support with command batching as used with REAPI. This branch introduces an empty marker file .bst-prepared as replacement, which can also be reliably created as part of a command batch. As the marker file exists in the buildtree in the artifact, we no longer need to store the flag in the workspace YAML file.

This branch also includes an extension to the Artifact proto to store the sources used for the build. The new field is set whenever we're caching the buildtree. We also need the buildtree for incremental workspace builds and storing the sources should be very cheap if we already store the buildtree.

Edited by Jürg Billeter

Merge request reports