Skip to content

Artifact blob handling

Jürg Billeter requested to merge juerg/artifact-blobs into master

This branch contains fixes and improvements related to artifact blob handling. This is preparation for blob-based expiry as used by buildbox-casd and thus, a prerequisite to !1499 (merged).

The main changes are:

  • _sandboxremote.py: Fetch blobs on artifact creation, not after command

    This fixes a bug where artifact files of a build target are not available in the local cache after a remote execution build with the default pull-artifact-files configuration. That could happen, e.g., when the build target is a compose element that doesn't run any actual commands. Another change in this branch exposed this bug.

  • _artifact.py: Update blob mtimes in cached()

    This ensures timestamps are updated for artifacts used in a bst session to allow proper LRU expiry.

  • cli.py: Require artifact file contents to be present for bst show

    With blob-based expiry (and also, partially downloaded RE artifacts) it's no longer sufficient to check for the presence of the artifact proto to determine whether an artifact is locally cached. This changes bst show to only show cached for artifacts that are complete in the local cache (with regards to file contents, buildtree is not checked). This is required for the test suite to pass with buildbox-casd and should prevent user confusion.

    Checking presence of artifact file blobs obviously has a cost. In our Debian-based 'essential-stack.bst' benchmark I see a 5% slowdown for 'bst show everything cached' (7.03s vs. master 6.73s) and I suspect there may be real world cases where the overhead is larger. If this turns out to be a real issue, one option would be to add a flag to bst show and determine artifact file blob presence only if that flag is set (or corresponding userconfig).

Edited by Jürg Billeter

Merge request reports