Out of process pybuild sandboxing
I feel like this idea is coming a little late (though it's still before committing to a stable 2.0 API), but previously my thoughts of out of process sandboxing were only in the context of switching to bash as the build file language mostly since that's portage's system and everything is strings and is easy to communicate, and I hadn't really given much though to doing out of process sandboxing using python.
It occurs to me that there's little reason why we couldn't do out of process sandboxing in our existing python system using a wrapper script that, for the purpose of pulling information out of the pybuild, turns it into json (like we already do for the cache) and sends it to the non-sandboxed main portmod process via stdout.
That being said, there are a number of benefits to using RestrictedPython for in-process sandboxing. By using a restricted subset of python we can enforce that pybuilds are relatively portable and future-proof by limiting the modules that can be used, and blocking the usage of private interfaces (i.e. fields and functions starting with underscores which aren't part of public APIs).
It seems to me that the best idea is to use both. We can use out of process sandboxing for security reasons, and RestrictedPython for portability reasons. Unfortunately this will likely be slightly slower than the current implementation, given that this won't resolve any of our current performance problems, and may add additional overhead (E.g. when doing out of process sandboxing, we'll have the same performance drain caused by the import cache not being shared, and still have to use RestrictedPython to process the files).
- Security: Out of process sandboxing is the most secure.
- Simplicity: We can throw out a lot of ugly and hard to maintain sandboxing code, while being able to much more easily provide access to builtin modules, and functions without worrying about their security.
In terms of the potential overhead, on my machine I'd previously measured an average time of about 0.1s to load a pybuild using the sandbox, while on the other hand with a wrapper script I quickly threw together, it increases to about 0.7s. We still do have the cache to mitigate this though, and it also seems that importing the pybuild module is a significant part of the slowdown, taking 500ms by itself, due to the fact that the pybuild import also imports portmod, and portmod is slow to import. Refactoring portmod's structure and removing global code to improve import time (also necessary for #160 (closed)) will help a lot with speeding this up, though the 200ms unaccounted for makes it unlikely we will ever get such a system to be as fast as the current implementation.
It's also worth noting that the lack of stdout from Sandboxie until very recently also meant that implementing this would have meant that the Windows version would be either completely broken (there would have been no way of getting information out of pybuilds if they were wrapped by sandboxie) or very insecure (given that part of this proposal includes throwing away most of the python sandboxing code, which currently provides reasonably good security).