We would like to move away from `pip` and replace this with a more distro friendly build system, possibly `meson`, for the purposes of installing BuildStream on a host, such that the leg work for a distro is reduced, and the reliability of BuildStream itself remains uncompromised. ## Problem statement BuildStream currently uses a combination of manual steps and `pip` for installation purposes, but this is not ultimately a suitable approach for distributing BuildStream, as we require more deep integration with the underlying system, for some reasons: * We depend on a number of bindings and not pure python dependencies, meaning we need separate manual steps and we cannot live in clean separation inside a *virtual env* at all because of this. * We need to install data outside of `/usr/lib/python3/`, which is currently supported, but the `data_files` setuptools approach is in a deprecated state of flux which I have noted upstream pypi developers to be in disagreement about, I have been told to not use `data_files`, without any clear explanation of what I should use instead to install things on the host outside of python owned directories. Files we currently need to install include: * Bash completion helper (we should be relying on a system installed `bash-completions.pc` file to discover the appropriate place to install our bash completion helper on the target host) * Man pages, which should probably also use some host system introspection to figure out the appropriate installation directory, usually in `/usr/share/man` * We should also improve our plugin story such that we no longer rely on `pip` for this, what I am hoping for is: * BuildStream expects to find plugin packages in `${prefix}/share/buildstream/plugins` * BuildStream installs a `buildstream.pc` pkg-config file, informing plugin packages where they should install * Plugins use the BuildStream installed pkg-config file to discover where they should be installing to ## Complication of versions In order to move to a more distro friendly installation model, we need to figure out a safe way of bringing in our pure python dependencies, which is tricky. There is a cultural difference in the python community and the distro community: * A great many python developers *expect* to be using `pip` for their distribution purposes, which allows python modules to depend on very strict versions because `pip` will install them in such a way that any version of a python dependency can be parallel installed. * The distro community on the other hand usually carries with it a philosophy that a single computer should only need a single version of a given library at a time, as such the distro community expects installed modules to retain full API compatibility, and change the name of the shared library at a time when API compatibility breaks. For BuildStream, we currently do not *"pin"* the exact versions of all of our pure python dependencies, but we certainly should be doing this at least in stable releases for reliability purposes - we cannot trust that pure python libraries adhere to the same standard of quality, because `pip` does not make this necessary for them. ## Plausible solutions This problem needs a solution before we can think about dropping `pip` and moving to distribution model more appropriate for BuildStream. Some possible solutions I can think of include: * Leveraging `pip` during the packaging phase, where we turn the BuildStream git checkout into a consumable tarball * In this scenario, I would expect that we use `pip` to download the dependencies, and that we ensure that our `make install` step installs BuildStream in such a way that only the pure python dependencies which we've *bundled* into our tarball are ever used by BuildStream, and not any random version that a distro might have considered to be correct. * Using submodules in the BuildStream repository to include the upstream dependencies in the BuildStream build directly