Skip to content

Update to support per task job executions

With meltano!2652 (closed) as part of meltano#3479 (closed) the meltano schedule command gains add's support for jobs (https://docs.meltano.com/reference/command-line-interface#job).

  1. meltano schedule run schedule-name will behave as it does today when calling a legacy elt job
  2. meltano schedule run schedule-name will call meltano run under the hood when the schedule is a job and effectively just be an alias for meltano run in that scenario.
    • It will not call meltano run per task , it will simply call meltano run <job_name>
  3. The new schema output from meltano schedule list --format=json will clearly distinguish between legacy elt schedules and new jobs with tasks.
  4. files-airflow will be updated and temporarily accommodate but the existing json format and the new format.
    • files-airflow will determine which format is in use either by calling meltano --version first, or potentially by looking for a top-level embedded version field in the json output.
    • files-airflow will continue to use a BashOperators calling meltano schedule run <schedule name> for legacy ELT schedules
    • files-airflow will create a dag/BashOperator per task when encountering a job. Which in turn will call meltano run <the task blocks...> directly rather than a singular meltano schedule <job_name>