Use buildstream 2
Context
Seems there are plans to a possible Buildstream2 release soonish. It would be great we can use it and take advantage of some of its new features
This is an issue to recollect possible blockers to use it
Blockers at the moment
-
There should be clear roadmap to have a final bst2.0 so we know at which point it's safe to start merging bst2 work into freedesktop-sdk master -
bst2 seems to be slower than bst1: - Test1 (16/01/2022):
- World rebuild in bst1: 621 min (https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/1980412118)
- World rebuild in bst2: 937 when the build failed (https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/1979958539)
- Test2 (13/07/2022), commit 362c260d:
- World rebuild bst1: 572 min (https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/2710289059)
- World rebuild bst2: 682 min (source cache already filled up)(https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/2710238197)
- Test3 (26/07/2022), commit (freedesktop-sdk-22.08beta.6 + bst2 patches)
- World rebuild bst2: 580 minutes (https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/2773932371)
- Possible causes:
-
source caching is too slow: https://github.com/apache/buildstream/issues/1317 -
using remote source cache is buggy: https://github.com/apache/buildstream/issues/1644 (we probably cannot hope for comparable/improved performance without leveraging remote source caches)
-
- PR to improve performance:
-
Don't mark layout locations as read-write directories, https://github.com/apache/buildstream/pull/1678 -
Clean the build directory after a successful build, https://github.com/apache/buildstream/pull/1677 -
Use StageTree to stage previous sources, https://github.com/apache/buildstream/pull/1680 -
https://github.com/apache/buildstream/pull/1697 -
https://github.com/apache/buildstream/pull/1698
-
- Test1 (16/01/2022):
-
Compatibility problem between ostree and buildbox-fuse (maybe that only affects g-b-m) https://github.com/apache/buildstream/issues/1660 -
notparallel variable is broken: https://github.com/apache/buildstream/issues/1360 - This is causing the build to fail at the moment: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/1979958539
-
RISC-V is not supported: -
https://github.com/apache/buildstream/issues/1578 -
BuildGrid/buildbox/buildbox-common!459 (merged) -
buildbox-run-bubblewrap needs to allow qemu-user builds, BuildGrid/buildbox/buildbox-run-bubblewrap#6 (closed)
-
-
Buildstream 2.0 plugins repo is still called experimental: https://gitlab.com/BuildStream/bst-plugins-experimental/ -
Failures when building (used to not be that common but we see it often now): https://github.com/apache/buildstream/issues/1465 -
collect_initial_scripts plugin (which exist because a limitation in buildstream) is failing (see #1446 (closed)), we need to solve: -
Fail to compile efivar, #1447 (closed) -
Fail to build librsvg (rust): -
Error tracking through junctions: https://github.com/apache/buildstream/issues/1686 -
Conditionals in the root of project.conf cause an error: https://github.com/apache/buildstream/issues/1683 -
Source.set_ref sometimes gets passed a dictionary instead of a Node instance: -
Buildstream fix: https://github.com/apache/buildstream/issues/1685 -
cargo / git plugin fix: https://github.com/apache/buildstream-plugins/pull/19
-
-
We should be able to build, at least, GNOME with bst2
Important issues but not blockers anymore
-
git_tag and derivatives put entire .git into source cache; this is a problem with bst2 as it consumes a lot of extra space, BuildStream/bst-plugins-experimental#48 (closed) - We do not consider this a blocker anymore (can be fixed later)
-
Timed out waiting for buildbox-casd to become ready: #1444 - While the issue is not fixed, we have not seen this anymore after all the optimization fixes, so not really a blocker anymore
-
https://github.com/apache/buildstream/issues/38 and BuildGrid/buildbox/buildbox-casd#69 (moved) - Workaround exist: !9088 (closed)
Acceptance Criteria
- fdsdk master is being built with buildstream2 (or beta / rc)
Edited by Javier Jardón