Python adventures, again

I have been having a love-hate relationship with python virtual environments, like everyone.

What I think I have decided tonight is:

1. Avoid using "system" pythons for programming projects, including Homebrew — that's a system python. System pythons are for utilities and applications managed for you, not for doing programming work: they can and do change and break dependants.
2. Use conda (Anaconda) for setting up anything "complicated" (Scientific stuff, Jupyter, or using pygame on a Mac, or using python at all on Windows).
   • It supports using multiple environments, and just works.
   • You can use pip in a conda environment, for packages which aren't managed by anaconda, or conda forge.
   • But for other development projects (or Linux server deployments) it may be easier to use "traditional" light-weight python environments (miniconda is lightweight, but more difficult to set up than anaconda full, at least in Mac/Windows, and less common in servers).
3. Use ASDF to manage multiple pythons (but not venvs) for development and testing.
   • ASDF uses pyenv under the hood. Pyenv compiles python runtimes, Unix Ports-style.
   • Some versions of python can't be compiled on newer macOS (e.g. Python 3.6).
4. As a last resort, use Docker Containers, or virtual machines, to host python environments.
   • These run on the Linux kernel in the container/virtual machine.
   • You can install any version of python, typically as part of the container image, or as a package of the hosted OS.
   • These can be more difficult to manage, and require virtualization hardware and software.
   • Access to the host environment (e.g. to interface with games running on the host) is more difficult.

Virtual environments:

1. In unixen, Pipx is a nice way to install PyPI applications "stand-alone" — for bpytop and virtualenv, for instance. These dedicated environments are lightweight and linked to your default python: that would be a system python, Anaconda base, or a global ASDF python, depending on which python your system shell will run when you type python.
   • Pipx is installed as a Homebrew package on macOS, and a pip package elsewhere (see my environment variable settings below, first): brew install pipx, or python -m pip install --user pipx
2. For python virtual environments, if you can't use Conda environments (because you are not using Conda), then use virtualenv — it's the next most mature option.
   • You can install it with pipx: pipx install --python $(asdf which python) virtualenv
     • If you installed pipx with Homebrew, then this means that the default python for any virtualenv-managed environment is also Homebrew's python.
     • Since this is changeable on Homebrew's whim, you need to override it with --python $(asdf which python) to use ASDF.
3. Python's built-in venv module is the PEP 405 standard, but it's less capable than virtualenv and slower
4. I've always disliked virtualenvwrapper, but VirtualFish (despite the unhelpful name) provides the same functionality, with better names, and Tide prompt support.
   • Install virtualfish: pipx install --python $(asdf which python) virtualfish, (to avoid Homebrew as the default python for venvs)
   • Then install it into fish with vf install; alias vp=vf; funcsave vp.
5. Avoid using pipenv. It only offers dependency locking and some convenience (e.g. "automatic" activation via pipenv shell), at the cost of brokenness. These are better provided by other means:
   • pip-tools pinned/locked pip dependencies with the ability to keep them fresh
   • VirtualFish has a plugin for automatic environment activation, or else use direnv.

Configuring default locations for venvs:

#virtualenv / virtualfish base directory
#https://pipenv.pypa.io/en/latest/advanced/#custom-virtual-environment-location
WORKON_HOME=~/lib/venv
#https://github.com/justinmayer/virtualfish/blob/main/virtualfish/virtual.fish
VIRTUALFISH_HOME=~/lib/venv

#PIPX https://pypa.github.io/pipx/docs/
PIPX_HOME=~/lib/pipx
PIPX_BIN_DIR=~/bin

What about Ansible?

I'm still deciding. I would like to set up an ASDF-managed python, then install ansible into it. However there are easier alternatives:

• Ansible is available as an OS package, though usually an older version.
• I could use pipx, but this requires a python anyway, and since pipx is instaled with homebrew, then by default all the installed applications also use Homebrew's python. This is probably probably okay for ansible on a management node.
• Actually in many cases, the version of Ansible is project-dependant anyway. So that it is installed into a venv with pip, and probably I don't want a global ansible, except as part of a workstation bootstrap.

On the (ansible-managed) server side, just install python as an OS package in most cases, or as part of the container image (along with SSH), and then Ansible can control it. Most come with python 2.7 which is "good enough".