local venvs use an outdated version of pip
One of our taps has a transitive dependency on grpcio
.
meltano install
works fine on developer machines, but when building our docker image it would hang and eventually fail:
15 134.9 Installing extractor 'tap-crashlytics'...
#15 650.2 Extractor 'tap-crashlytics' could not be installed: failed to install plugin 'tap-crashlytics'.
#15 650.2 Failed building wheel for tap-bigquery
#15 650.2 Failed building wheel for singer-python
#15 650.2 Failed building wheel for proto-plus
#15 650.2 Failed building wheel for ciso8601
#15 650.2 Failed building wheel for simplejson
#15 650.2 Failed building wheel for grpcio
#15 650.2 Failed building wheel for pyyaml
#15 650.2 Command "/project/.meltano/extractors/tap-crashlytics/venv/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-04q9q1qe/grpcio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-88i4nve7/install-record.txt --single-version-externally-managed --compile --install-headers /project/.meltano/extractors/tap-crashlytics/venv/include/site/python3.7/grpcio" failed with error code 1 in /tmp/pip-install-04q9q1qe/grpcio/
#15 650.2
#15 650.2
...
#15 809.8 Failed to install plugin(s)
#15 ERROR: executor failed running [/bin/sh -c meltano install]: runc did not terminate sucessfully
------
> [stage-3 7/9] RUN meltano install:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c meltano install]: runc did not terminate sucessfully
It took quite a while to figure out what was going on, but it seems that grpcio
only reliably installs with the latest version of pip. So it would install fine on 20.1, but the python:3.7
docker image ships with 18.1.
I tried upgrading pip in the image before installing meltano, but even when meltano was using 20.1, the plugin venvs were still using 18.1. This seems to be a quirk of the way python -m venv
works. The suggested solution is to upgrade pip after installing venv, which is tricky, since meltano manages creating venvs at plugin install time.