Providing a consistent runtime environment
One significant difference between Portmod and other source-based package managers (portage, the arch build system, nix, etc.) is that portmod doesn't provide its own runtime environment. This is because it's not a system package manager, and by necessity it relies on an external system (otherwise we couldn't run things such as tes3cmd
, or the project atlas generator scripts).
Without a Consistent Environment
This has led to a number of issues. A good recent example is that the python
command is not necessarily usable, as there may be multiple python versions on the system and the one that is running portmod is not necessarily the one used when python
is called. Instead, the only reliable way of spawning a new python shell is to run the executable defined by sys.executable
.
Another is that sh
doesn't have consistent behaviour across systems, as it could either be strict sh
, or support bash
syntax.
These problems could extend to basically every external tool that packages rely on (also currently includes: VirtualEnv (where you must specify the python executable to use if you want to avoid problems on a system with python2 as the default), Perl, ImageMagick and Java (where we could certainly run into problems if we ever need different versions of the java runtime)), and while not all of these have had problems yet, it's likely that the number of such problems will only increase as portmod's userbase increases, and the number of packages increases.
Gererally, the lack of a common environment means that there are a number of rules (currently undocumented, and many undiscovered) which need to be followed if you want pybuild
files to be portable. Discovering and documenting these constraints may be possible but also may result in an extremely complex, cumbersome and unintuitive system, as opposed to one where the environment is identical regardless of your distribution or OS version (and ideally also OS).
Security
The lack of a consistent environment also means that we have to trade some degree of security for usability (as a side note, security is not a significant issue at the moment, as all packages currently require approval by me, so they are not significantly less secure than portmod itself. In future there might be package repositories where packages don't require approval and could be malicious).
Ideally, portmod would only provide read access to parts of the system which it needs, however, since it's hard for portmod to figure out where everything is on an arbitrary system, the most reliable way is to provide read access everywhere.
Currently portmod restricts packages in a half-hearted manner, only limiting read access during package loading and when network access is allowed (src_unpack
), and given the problems this has caused, I'm tempted to slacken this too until a better way of handling the runtime environment is implemented.
Possible Alternatives
Containerization
The main, full-featured, alternative that occurs to me is that we could bundle our own execution environment using a container system such as docker or podman (we could even support both, as they are not very different from the perspective of invoking them).
This would also mean that we could integrate with the package manager in the container to allow portmod's packages to depend on system packages in the container, which could then be installed automatically.
The downside is that installation gets an extra step, and possible non-userspace requirements like docker. Fortunately podman is a userspace alternative that should be more or less interchangeable, but it is somewhat complicated to get it working on MacOS or Windows as it would basically require a running Linux VM of some sort, or WSL2.
It also would require non-trivial storage for the container, but as a quick test I found that the clearlinux container on docker hub is 65MB to download, and 174MB storage, and python requires a 55MB download for a total container size of 221MB after python is installed (which would be sufficient as a base environment). While multiple prefixes would need multiple containers, they could still use the same base image. The containers could easily get very big though.
Simplification
A second, simpler, option would be to remove the runtime environment entirely and just allow the python contained within the package file.
While this is not entirely desirable, it would still allow about 95% of the current packages to work, and I can see a benefit to having packages use this by default and opt-in to a more complex environment such as containerization.
As python scripts still allow filesystem access, this would also still require maintaining the current sandbox to handle filesystem permissions, though we could even have something like !304 (closed) safely interpret packages which don't define any of the installation phase functions (src_*
, etc.), and require containerization even for those which define phase functions.
Request for Comments
Any opinions, particularly on the usability side of things? I've never tried installing Docker on Windows, and I am concerned that the runtime requirements of Portmod might get too unwieldy, noting that it looks like running Docker images on windows will require one of:
- A up to date version of Windows 10 (any type) with WSL2 (allowing a linux container to be used): https://docs.docker.com/docker-for-windows/install-windows-home/
- Windows 10 Pro/Enterprise/Education with Hyper-V enabled: https://docs.docker.com/docker-for-windows/install/ (which I think also allows Linux containers, but it's less clear).
Plus it's probably impossible on older versions of Windows.
Mac requirements for docker are a lot less stringent, but still could be limiting compared to the current requirements: https://docs.docker.com/docker-for-mac/install/
Linux of course would have great support for it, but I feel like it's barely worth the effort of trying to keep things cross-platform if getting Portmod working is complex on every platform other than Linux (where users are usually more technically-minded and could manage complex setup).