Data files are missing from python package
Summary
Since !1261 (merged) was merged, the source distribution created by the standard python3 setup.py sdist
is not including the data files for the plugins. This means that BuildStream is completely unusable if installed from current state of the repository.
Seems like this is due to the include_package_date=True
option added to setup.py
in 16a28c03. Without that option, we seem to be correctly including the necessary files. More precisely, here's the list of files that are missing from the source distribution:
diff --git a/new b/old
index 872f4e59..8704414d 100644
--- a/new
+++ b/old
@@ -169,58 +169,8 @@ buildstream/_yaml.py
buildstream/buildelement.py
buildstream/data/
buildstream/data/bst
-buildstream/data/build-all.sh.in
-buildstream/data/build-module.sh.in
-buildstream/data/projectconfig.yaml
-buildstream/data/userconfig.yaml
buildstream/element.py
buildstream/plugin.py
-buildstream/plugins/
-buildstream/plugins/elements/
-buildstream/plugins/elements/__init__.py
-buildstream/plugins/elements/autotools.py
-buildstream/plugins/elements/autotools.yaml
-buildstream/plugins/elements/cmake.py
-buildstream/plugins/elements/cmake.yaml
-buildstream/plugins/elements/compose.py
-buildstream/plugins/elements/compose.yaml
-buildstream/plugins/elements/distutils.py
-buildstream/plugins/elements/distutils.yaml
-buildstream/plugins/elements/filter.py
-buildstream/plugins/elements/filter.yaml
-buildstream/plugins/elements/import.py
-buildstream/plugins/elements/import.yaml
-buildstream/plugins/elements/junction.py
-buildstream/plugins/elements/make.py
-buildstream/plugins/elements/make.yaml
-buildstream/plugins/elements/makemaker.py
-buildstream/plugins/elements/makemaker.yaml
-buildstream/plugins/elements/manual.py
-buildstream/plugins/elements/manual.yaml
-buildstream/plugins/elements/meson.py
-buildstream/plugins/elements/meson.yaml
-buildstream/plugins/elements/modulebuild.py
-buildstream/plugins/elements/modulebuild.yaml
-buildstream/plugins/elements/pip.py
-buildstream/plugins/elements/pip.yaml
-buildstream/plugins/elements/qmake.py
-buildstream/plugins/elements/qmake.yaml
-buildstream/plugins/elements/script.py
-buildstream/plugins/elements/script.yaml
-buildstream/plugins/elements/stack.py
-buildstream/plugins/sources/
-buildstream/plugins/sources/__init__.py
-buildstream/plugins/sources/_downloadablefilesource.py
-buildstream/plugins/sources/bzr.py
-buildstream/plugins/sources/deb.py
-buildstream/plugins/sources/git.py
-buildstream/plugins/sources/local.py
-buildstream/plugins/sources/ostree.py
-buildstream/plugins/sources/patch.py
-buildstream/plugins/sources/pip.py
-buildstream/plugins/sources/remote.py
-buildstream/plugins/sources/tar.py
-buildstream/plugins/sources/zip.py
buildstream/sandbox/
buildstream/sandbox/__init__.py
buildstream/sandbox/_config.py
Steps to reproduce
Run python3 setup.py sdist
and investigate the files included in the resulting tarball using something like tar tf dist/*.tar.gz
.
What is the current bug behavior?
Necessary YAML files and plugin files are missing from the source distribution.
What is the expected correct behavior?
Such files are included correctly.
Possible fixes
As mentioned in setuptools documentation:
If using the setuptools-specific include_package_data argument, files specified by package_data will not be automatically added to the manifest unless they are listed in the MANIFEST.in file
So, to fix this, we should either:
- Remove
include_package_data=True
argument, or - Remove use of
package_data
argument, and instead list those files inMANIFEST.in
Additionally, it seems rather concerning that such a major issue went unnoticed by our testing and CI setup. Since the current directory is always added to sys.path
in Python, BuildStream is able to find these files correctly during the tests, even though they are not included in the installed package. To mitigate this in future, I would suggest moving our source code inside a src
directory. This layout has several advantages as mentioned here.