Document how to use an existing Airflow installation to orchestrate Meltano

Or Barda asked on Slack:

Hi guys, is there a way to use already running AirFlow in Meltano as the orchestrator?

I responded:

Absolutely! Using Meltano's "built-in" Airflow orchestrator really just means installing it into a Meltano-managed Python venv inside .meltano, and invoking it through meltano invoke so that some default Meltano-compatible configuration can be dynamically applied at invocation time: https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L840-855

That config points Airflow to $MELTANO_PROJECT_ROOT/orchestrate/dags to look for dags, which is where you'll find https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py (since meltano add orchestrator airflow also addd the Airflow file bundle from that repo).

That DAG should work with any Airflow installation as long as the MELTANO_PROJECT_ROOT env var is set to point at the Meltano project directory, and as long as Meltano itself is installed and meltano install has been run inside the project.

That setup assumes you'll use meltano schedule to schedule your meltano elt pipelines, since it iterates over the result of meltano schedule list --format=json to create DAGs for each, but you can also create your own DAGs for any pipeline you fancy using BashOperator + meltano elt, or DockerOperator + docker run <your project image> elt ... (see https://meltano.com/#containerization)