Use virtual artifact directory to stage and extract metadata

This branch eliminates extract directories (#913 (closed), #920 (closed), #921 (closed)). Artifact access for staging and extracting metadata is handled solely via CasBasedDirectory. Initial tests have shown a reduction in staging time by almost 40%.

I've verified that freedesktop-sdk still builds with this branch. It reduces total build time by 5% or 8 minutes compared to master on my system (02:39:30 vs. 02:47:44 for bst build).

added 33 commits

• b2b5cf95 - utils.py: Add filter_callback parameter to copy_files and link_files
• 29a4e73d - storage: Add filter_callback parameter to Directory.import_files
• 79362bea - element.py: Extract __split_filter() from __compute_splits()
• 3b4175f9 - element.py: Add __split_filter_func() method
• a72f7eb7 - element.py: Replace file lists with filter callbacks in stage_artifact()
• 5a91e4dd - local.py: Remove unnecessary files argument from copy_files() call
• 151f1221 - compose.py: Replace file list with filter callback
• 4339d045 - storage: Remove files parameter from Directory.import_files
• 2d12174c - utils.py: Remove unused files parameter from copy_files and link_files
• 26fa5650 - utils.py: Remove unused presorted parameter from _process_list()
• bb393b20 - utils.py: Remove unused filelist parameter from _process_list()
• 828a766c - _casbaseddirectory.py: Add result parameter to import methods
• 6801d309 - _casbaseddirectory.py: Replace file list with filter callback
• aa9dc880 - _filebaseddirectory.py: Add _get_filetype() method
• 23d6d5ad - _filebaseddirectory.py: Support importing files from CAS
• 175c69c8 - element.py: Add __get_extract_key() method
• d494874c - _artifactcache.py: Add get_artifact_directory() method
• 33d1f83b - element.py: Add __get_artifact_directory() method
• 5544031c - tests/artifactcache/expiry.py: Remove test_extract_expiry()
• 626fd7ba - tests/integration/pullbuildtrees.py: Remove extract directory checks
• acdc61a6 - tests/integration/artifact.py: Do not rely on extract directories
• 9bbcfec8 - element.py: Use virtual artifact directory to stage dependencies
• bf03f55c - element.py: Use virtual artifact directory to stage buildtree
• 1a25c57c - element.py: Use virtual artifact directory to compute manifest
• 8fba3461 - _casbaseddirectory.py: Add _objpath() method
• 6d2b387e - _casbaseddirectory.py: Add _exists() method
• e0455a94 - element.py: Use virtual artifact directory for metadata
• ccef7f80 - element.py: Remove unused __extract() method
• eaf25311 - _artifactcache.py: Remove unused extract() method
• be3992ec - cascache.py: Remove unused extract() method
• 5f2536dc - _artifactcache.py: Drop extract directory removal in remove()
• b10f89d6 - _artifactcache.py: Remove unused extractdir field
• 6e5e928c - _context.py: Remove unused extractdir field

Compare with previous version

mentioned in merge request !1185 (merged)

added 54 commits

• 6e5e928c...5a1a5814 - 32 commits from branch master
• d6088424 - _casbaseddirectory.py: Add result parameter to import methods
• 3f2b31a2 - _casbaseddirectory.py: Replace file list with filter callback
• 1775ec3a - _filebaseddirectory.py: Add _get_filetype() method
• 97ff1ee3 - _filebaseddirectory.py: Support importing files from CAS
• cf0716ac - element.py: Add __get_extract_key() method
• 5b552734 - _artifactcache.py: Add get_artifact_directory() method
• dcb93f90 - element.py: Add __get_artifact_directory() method
• 643c8e59 - tests/artifactcache/expiry.py: Remove test_extract_expiry()
• d6877749 - tests/integration/pullbuildtrees.py: Remove extract directory checks
• 651a173e - tests/integration/artifact.py: Do not rely on extract directories
• 02f3ac9e - element.py: Use virtual artifact directory to stage dependencies
• 2d1b0824 - element.py: Use virtual artifact directory to stage buildtree
• e60b6e09 - element.py: Use virtual artifact directory to compute manifest
• 4c6006f4 - _casbaseddirectory.py: Add _objpath() method
• 01f73895 - _casbaseddirectory.py: Add _exists() method
• 1320ebcc - element.py: Use virtual artifact directory for metadata
• b20f52c4 - element.py: Remove unused __extract() method
• 823c3e7b - _artifactcache.py: Remove unused extract() method
• 59659ded - cascache.py: Remove unused extract() method
• aaac7ae5 - _artifactcache.py: Drop extract directory removal in remove()
• fc80166e - _artifactcache.py: Remove unused extractdir field
• 9a11c0a3 - _context.py: Remove unused extractdir field

Compare with previous version

changed the description

added 21 commits

• afc9bd9a - _casbaseddirectory.py: Replace file list with filter callback
• b7a600af - _filebaseddirectory.py: Add _get_filetype() method
• fbda7e66 - _filebaseddirectory.py: Support importing files from CAS
• 6e09de69 - element.py: Add __get_extract_key() method
• cc90b1c9 - _artifactcache.py: Add get_artifact_directory() method
• d1807292 - element.py: Add __get_artifact_directory() method
• f6bd608a - tests/artifactcache/expiry.py: Remove test_extract_expiry()
• bcca633b - tests/integration/pullbuildtrees.py: Remove extract directory checks
• 8eccbea3 - tests/integration/artifact.py: Do not rely on extract directories
• 8a83506d - element.py: Use virtual artifact directory to stage dependencies
• 61041462 - element.py: Use virtual artifact directory to stage buildtree
• 0d78e8a3 - element.py: Use virtual artifact directory to compute manifest
• bc03a086 - _casbaseddirectory.py: Add _objpath() method
• a2200327 - _casbaseddirectory.py: Add _exists() method
• 56cf5aca - element.py: Use virtual artifact directory for metadata
• 4e76227c - element.py: Remove unused __extract() method
• 10e52f04 - _artifactcache.py: Remove unused extract() method
• d3b030c9 - cascache.py: Remove unused extract() method
• 962b6103 - _artifactcache.py: Drop extract directory removal in remove()
• 3d5833bc - _artifactcache.py: Remove unused extractdir field
• a8e32578 - _context.py: Remove unused extractdir field

Compare with previous version

This branch no longer creates the extract directory. However, it doesn't delete/cleanup the extract directory that may still exist from a previous version. It probably makes sense to simply call rmtree on startup, if BuildStream finds an extract directory (artifacts/extract or extract in cachedir). Any concerns with this approach?

added 1 commit

• 631235d5 - _context.py: Cleanup old extract directories

Compare with previous version

mentioned in merge request !1188 (merged)

added 30 commits

• 631235d5...153f81a1 - 11 commits from branch master
• 34ef71af - element.py: Add __get_extract_key() method
• 8be86f4f - _artifactcache.py: Add get_artifact_directory() method
• f4a4b384 - element.py: Add __get_artifact_directory() method
• 7ecd3410 - tests/artifactcache/expiry.py: Remove test_extract_expiry()
• 0f029b14 - tests/integration/pullbuildtrees.py: Remove extract directory checks
• e11f6691 - tests/integration/artifact.py: Do not rely on extract directories
• 45caf7e1 - element.py: Use virtual artifact directory to stage dependencies
• 6e67303a - element.py: Use virtual artifact directory to stage buildtree
• b93cd889 - element.py: Use virtual artifact directory to compute manifest
• 022311c6 - _casbaseddirectory.py: Add _objpath() method
• 5f58ec6b - _casbaseddirectory.py: Add _exists() method
• 4f565c3e - element.py: Use virtual artifact directory for metadata
• 06a84bfe - element.py: Remove unused __extract() method
• 0f4e9eb6 - _artifactcache.py: Remove unused extract() method
• b1efd53e - cascache.py: Remove unused extract() method
• a8208830 - _artifactcache.py: Drop extract directory removal in remove()
• 08d1fb6d - _artifactcache.py: Remove unused extractdir field
• 115b2db3 - _context.py: Remove unused extractdir field
• 05e26f58 - _context.py: Cleanup old extract directories

Compare with previous version

unmarked as a Work In Progress

changed the description

mentioned in merge request !1192 (merged)

resolved all discussions

Other than my comment about the pullbuildtree test this looks good to me, I've had !1175 (merged) based off the removal of extract dirs from element.py and have also noticed the speed increases.

added 22 commits

• 05e26f58...8f121ffc - 3 commits from branch master
• 8be33e19 - element.py: Add __get_extract_key() method
• 64d1e887 - _artifactcache.py: Add get_artifact_directory() method
• b9702b5a - element.py: Add __get_artifact_directory() method
• 805f02ec - tests/artifactcache/expiry.py: Remove test_extract_expiry()
• 9cb582b3 - tests/integration/pullbuildtrees.py: Remove extract directory checks
• 079139ec - tests/integration/artifact.py: Do not rely on extract directories
• 0cf62afb - element.py: Use virtual artifact directory to stage dependencies
• bed3b8fa - element.py: Use virtual artifact directory to stage buildtree
• e7b0710b - element.py: Use virtual artifact directory to compute manifest
• 296351f6 - _casbaseddirectory.py: Add _objpath() method
• 7e88f2e6 - _casbaseddirectory.py: Add _exists() method
• 581b7f99 - element.py: Use virtual artifact directory for metadata
• a596dc46 - element.py: Remove unused __extract() method
• 4427343f - _artifactcache.py: Remove unused extract() method
• 99b76bd9 - cascache.py: Remove unused extract() method
• 3fc9e559 - _artifactcache.py: Drop extract directory removal in remove()
• d7a9ba8c - _artifactcache.py: Remove unused extractdir field
• 6740b17b - _context.py: Remove unused extractdir field
• 3b54b647 - _context.py: Cleanup old extract directories

Compare with previous version

resolved all discussions

approved this merge request

resolved all discussions

assigned to @marge-bot123

merged

mentioned in commit 60e62439

mentioned in issue #920 (closed)

mentioned in issue #913 (closed)

mentioned in issue #921 (closed)

mentioned in merge request !2018 (closed)