Add support for Airflow 2.0
Problem to solve
It's not possible to use Airflow 2.0 as an orchestrator out of the box with Meltano today. This is due to Airflow's change of the CLI commands for database initialization. Airflow 1.x uses
airflow initdb and 2.x uses the new format
airflow db init.
Data engineers who want to use the new features in Airflow 2.0 for their pipelines.
airflow db init, we should be able to branch here based on the result of airflow --version: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/airflow.py#L80
If there are also changes in how DAGs are defined, we'd also need changes in https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py
What does success look like, and how can we measure that?
Both Airflow 1.x & Airflow 2.x are supported by the Meltano CLI.
(Ensure the feature doesn't cause any regressions)
- Write adequate test cases and submit test results
- Test results should be reviewed by a person from the team