Let plugin `pip_url` take into account Python version
The pip_url
for airflow
is currently apache-airflow==1.10.14 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt
, using a constraints file built specifically for Python 3.6, regardless of the version that's actually used.
With Airflow 1.10.14, this is OK, because the constraints file is actually valid on 3.7 and 3.8 as well, but this is not the case for Airflow 2.0.1's https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.6.txt.
This can be seen in !2032 (merged) and pipelines https://gitlab.com/michelrado/meltano/-/jobs/1026252711 (3.7) and https://gitlab.com/michelrado/meltano/-/jobs/1025856502 (3.8).
One way to solve this would be to let a PYTHON_VERSION
environment variable to referenced from pip_url
, like so:
pip_url: 'apache-airflow==2.0.1 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-${PYTHON_VERSION}.txt'