Handle python depencies for the Meltano and its components
This stems out of the following discussion that happened after two production jobs started to fail.
15h]
tlapiana @jschatz dependency issue?
[14h]
jschatz @Micaël Bergeron was going to look at this. I am not sure what the issue is at the moment.
[14h]
Micaël Bergeron yes it is, there's an MR for it in "target-snowflake"
[14h]
Micaël Bergeron its a deps conflict for 'idna'
[14h]
tlapiana should this have been caught in a build process during the release? do more checks need to get run before release or is this unrelated?
[12h]
jschatz I ran into it in general python. It's a shortcoming in pip. Not sure exactly how it went down but the main problem is that our dependencies in setup py are not frozen. Which is the way it should be but it doesnt read from requirements.txt. so in the end we need to lock the dependencies somehow. That way right now seems to be with setup.py
[3m]
Micaël Bergeron basically one of our sub-deps (azure-common) is not pinning its deps and pip has not deps resolver.
> should this have been caught in a build process during the release? do more checks need to get run before release or is this unrelated?
basically this might just happen anytime you `pip install X`, if there is a sub-deps that has an unbounded dep. I'm actually very surprised `pip` doesn't resolve dependencies. What now? As we manage the venv in meltano, we could do `pip download ... --no-deps`, extract the `requirements.txt` then `pip install -r requirements.txt (if it exists)` then `pip install ...`
[<1m]
Micaël Bergeron @tlapiana tl;dr; the build process might go on just fine then 2 months later you can't install the package properly.
It seems pip
by itself is not a viable solution for deployment unless you are certain that your dependency tree is completely pinned.
Solution 1
The process I describe above:
- pip download --no-deps -d /tmp/...
- (for package url)
-tar xzf /tmp/... $(basename -s .tar.gz /tmp/.../requirements.txt)
- [optional] pip install -r /tmp/.../requirements.txt
- (for package git repo)
- unzip /tmp/... /requirements.txt
- [optional] pip install -r /tmp/.../requirements.txt
- pip install /tmp/...
I think it seems weird to use that to deploy but
Scrappy solution
Manually resolve conflicts in the setup.py
of each project.
Edited by Taylor A Murphy