## Summary Since !1261 was merged, the source distribution created by the standard `python3 setup.py sdist` is not including the data files for the plugins. This means that BuildStream is completely unusable if installed from current state of the repository. Seems like this is due to the `include_package_date=True` option added to `setup.py` in https://gitlab.com/BuildStream/buildstream/commit/16a28c0316a64189cb8c396c43b570764a83d33b. Without that option, we seem to be correctly including the necessary files. More precisely, here's the list of files that are missing from the source distribution: ```diff diff --git a/new b/old index 872f4e59..8704414d 100644 --- a/new +++ b/old @@ -169,58 +169,8 @@ buildstream/_yaml.py buildstream/buildelement.py buildstream/data/ buildstream/data/bst -buildstream/data/build-all.sh.in -buildstream/data/build-module.sh.in -buildstream/data/projectconfig.yaml -buildstream/data/userconfig.yaml buildstream/element.py buildstream/plugin.py -buildstream/plugins/ -buildstream/plugins/elements/ -buildstream/plugins/elements/__init__.py -buildstream/plugins/elements/autotools.py -buildstream/plugins/elements/autotools.yaml -buildstream/plugins/elements/cmake.py -buildstream/plugins/elements/cmake.yaml -buildstream/plugins/elements/compose.py -buildstream/plugins/elements/compose.yaml -buildstream/plugins/elements/distutils.py -buildstream/plugins/elements/distutils.yaml -buildstream/plugins/elements/filter.py -buildstream/plugins/elements/filter.yaml -buildstream/plugins/elements/import.py -buildstream/plugins/elements/import.yaml -buildstream/plugins/elements/junction.py -buildstream/plugins/elements/make.py -buildstream/plugins/elements/make.yaml -buildstream/plugins/elements/makemaker.py -buildstream/plugins/elements/makemaker.yaml -buildstream/plugins/elements/manual.py -buildstream/plugins/elements/manual.yaml -buildstream/plugins/elements/meson.py -buildstream/plugins/elements/meson.yaml -buildstream/plugins/elements/modulebuild.py -buildstream/plugins/elements/modulebuild.yaml -buildstream/plugins/elements/pip.py -buildstream/plugins/elements/pip.yaml -buildstream/plugins/elements/qmake.py -buildstream/plugins/elements/qmake.yaml -buildstream/plugins/elements/script.py -buildstream/plugins/elements/script.yaml -buildstream/plugins/elements/stack.py -buildstream/plugins/sources/ -buildstream/plugins/sources/__init__.py -buildstream/plugins/sources/_downloadablefilesource.py -buildstream/plugins/sources/bzr.py -buildstream/plugins/sources/deb.py -buildstream/plugins/sources/git.py -buildstream/plugins/sources/local.py -buildstream/plugins/sources/ostree.py -buildstream/plugins/sources/patch.py -buildstream/plugins/sources/pip.py -buildstream/plugins/sources/remote.py -buildstream/plugins/sources/tar.py -buildstream/plugins/sources/zip.py buildstream/sandbox/ buildstream/sandbox/__init__.py buildstream/sandbox/_config.py ``` ## Steps to reproduce Run `python3 setup.py sdist` and investigate the files included in the resulting tarball using something like `tar tf dist/*.tar.gz`. ## What is the current bug behavior? Necessary YAML files and plugin files are missing from the source distribution. ## What is the expected correct behavior? Such files are included correctly. ## Possible fixes As mentioned in [setuptools documentation](https://setuptools.readthedocs.io/en/latest/setuptools.html?highlight=include_package_data#including-data-files): > If using the setuptools-specific include_package_data argument, files specified by package_data will not be automatically added to the manifest unless they are listed in the MANIFEST.in file So, to fix this, we should either: * Remove `include_package_data=True` argument, or * Remove use of `package_data` argument, and instead list those files in `MANIFEST.in` --- Additionally, it seems rather concerning that such a major issue went unnoticed by our testing and CI setup. Since the current directory is always added to `sys.path` in Python, BuildStream is able to find these files correctly during the tests, even though they are not included in the installed package. To mitigate this in future, I would suggest moving our source code inside a `src` directory. This layout has several advantages as mentioned [here](https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure).