Skip to content

Use virtual artifact directory to stage and extract metadata

Abderrahim Kitouni requested to merge abderrahim/virtual-extract into bst-1

Description

The goal of this MR is to backport !1184 (merged) to bst-1. This eliminates extract directories, which makes bst-2 less likely to "corrupt" the cache.

The current situation is that when bst2 runs a cleanup and deletes files that belong to bst1, it (1) doesn't save space as they are still hardlinked in the extract directories (2) makes the cache unusable by bst1. This MR solves the first issue, and the second one can be easily solved once extract directories are out of the equation. It also opens the door to using the cache cleaner from bst-artifact-server, fixing #734. But these need to be separate MRs: this is already too big as is.

This additionally (partialy) backports the following MRs !445 (merged) !481 (merged) !991 (merged) !1181 (merged) !1182 (merged) !1183 (merged) !1184 (merged), as well as fixes from !953 (merged) !1647 (merged) !1101 (merged) !1190 (merged) !1878 (merged) (one commit from each MR). When reviewing, please note commits for which I'm the author: these have been substantially modified or (in the case of 4cc7eaf5) completely rewritten to remain compatible with bst-1.

I tried to make this as small as possible. In particular, I didn't port plugins: plugins should be able to keep using the old API. We can have a follow up MR for porting them (and possibly have some performance improvements)

So this introduces virtual directories to bst-1, which might not be desirable especially as the API isn't well defined yet (#1294). I see three possible solutions:

  1. make Sandbox.get_virtual_directory() private: This means it's only usable from core (and we could make an exception for core plugins)
  2. make some Directory APIs private, keeping only ones that are uncontroversial: It's okay if the resulting API is less powerful than in bst 2 as we still have the old API.
  3. Fix #1294 first. I'm planning to look into it soon anyway.

I'm leaning towards solution 2. as it makes it easier to maintain parallel branches of plugins targeting bst1 and bst2 as the diff would be smaller, but would be open to the others.

I've been using this for a couple of weeks on freedesktop-sdk and gnome-build-meta, and haven't encountered any problems. Integration tests have caught a regression (test_allow_overlaps_inside_symlink_with_dangling_target) which I plan to fix shortly.

Merge request reports