Add support for Airflow 2.0
Problem to solve
It's not possible to use Airflow 2.0 as an orchestrator out of the box with Meltano today. This is due to Airflow's change of the CLI commands for database initialization. Airflow 1.x uses airflow initdb
and 2.x uses the new format airflow db init
.
Target audience
Data engineers who want to use the new features in Airflow 2.0 for their pipelines.
Proposal
As for airflow db init
, we should be able to branch here based on the result of airflow --version: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/airflow.py#L80
If there are also changes in how DAGs are defined, we'd also need changes in https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py
What does success look like, and how can we measure that?
Both Airflow 1.x & Airflow 2.x are supported by the Meltano CLI.
Regression test
(Ensure the feature doesn't cause any regressions)
-
Write adequate test cases and submit test results -
Test results should be reviewed by a person from the team