Distribution story for BuildStream
We would like to move away from pip
and replace this with a more distro friendly build system, possibly meson
, for the purposes of installing BuildStream on a host, such that the leg work for a distro is reduced, and the reliability of BuildStream itself remains uncompromised.
Problem statement
BuildStream currently uses a combination of manual steps and pip
for installation purposes, but this is not ultimately a suitable approach for distributing BuildStream, as we require more deep integration with the underlying system, for some reasons:
- We depend on a number of bindings and not pure python dependencies, meaning we need separate manual steps and we cannot live in clean separation inside a virtual env at all because of this.
- We need to install data outside of
/usr/lib/python3/
, which is currently supported, but thedata_files
setuptools approach is in a deprecated state of flux which I have noted upstream pypi developers to be in disagreement about, I have been told to not usedata_files
, without any clear explanation of what I should use instead to install things on the host outside of python owned directories. Files we currently need to install include:- Bash completion helper (we should be relying on a system installed
bash-completions.pc
file to discover the appropriate place to install our bash completion helper on the target host) - Man pages, which should probably also use some host system introspection to figure out the appropriate installation directory, usually in
/usr/share/man
- Bash completion helper (we should be relying on a system installed
- We should also improve our plugin story such that we no longer rely on
pip
for this, what I am hoping for is:- BuildStream expects to find plugin packages in
${prefix}/share/buildstream/plugins
- BuildStream installs a
buildstream.pc
pkg-config file, informing plugin packages where they should install - Plugins use the BuildStream installed pkg-config file to discover where they should be installing to
- BuildStream expects to find plugin packages in
Complication of versions
In order to move to a more distro friendly installation model, we need to figure out a safe way of bringing in our pure python dependencies, which is tricky.
There is a cultural difference in the python community and the distro community:
- A great many python developers expect to be using
pip
for their distribution purposes, which allows python modules to depend on very strict versions becausepip
will install them in such a way that any version of a python dependency can be parallel installed. - The distro community on the other hand usually carries with it a philosophy that a single computer should only need a single version of a given library at a time, as such the distro community expects installed modules to retain full API compatibility, and change the name of the shared library at a time when API compatibility breaks.
For BuildStream, we currently do not "pin" the exact versions of all of our pure python dependencies, but we certainly should be doing this at least in stable releases for reliability purposes - we cannot trust that pure python libraries adhere to the same standard of quality, because pip
does not make this necessary for them.
Plausible solutions
This problem needs a solution before we can think about dropping pip
and moving to distribution model more appropriate for BuildStream. Some possible solutions I can think of include:
- Leveraging
pip
during the packaging phase, where we turn the BuildStream git checkout into a consumable tarball- In this scenario, I would expect that we use
pip
to download the dependencies, and that we ensure that ourmake install
step installs BuildStream in such a way that only the pure python dependencies which we've bundled into our tarball are ever used by BuildStream, and not any random version that a distro might have considered to be correct.
- In this scenario, I would expect that we use
- Using submodules in the BuildStream repository to include the upstream dependencies in the BuildStream build directly