Schedule UI (distinct Meltano layer on top of Orchestration)
Problem to solve
- Currently ELT Orchestration can only run via the CLI (
meltano schedule carbon__sqlite tap-carbon-intensity target-sqlite @daily
) and not the UI. Without the UI scheduling, the Airflow Orchestration UI is empty
Target audience
- UI users
Further details
- Enable users to run ELT on a schedule
- This UI and feature-set is the precursor of the "Profiles" concept @mbergeron from which fine grained Orchestration and Scheduling can occur
Proposal
- Provide a lightweight UI that calls to existing CLI commands for scheduling
What does success look like, and how can we measure that?
- The user can Schedule ELT runs in a dedicated UI and also see them in Airflow
Regression test
(Ensure the feature doesn't cause any regressions)
-
Write adequate test cases and submit test results -
Test results should be reviewed by a person from the team
Notes
Schedule -> E, L, T (default vs select?), Schedule time
meltano schedule carbon__sqlite tap-carbon-intensity target-sqlite @daily
- default to first installed for onboarding vs requiring manual selection (manual overrideable though)
- facilitates @mbergeron's "Profiles" concept
- Use Airflow tut in docs as helper/guide
TODOs
-
base UI -
ELT selectors -
schedule name + interval selector -
optional Catchup Date selector -
FE -> BE -> FE loop -
prefill vs manual selection solution -
schedule persistence -
UI inflation -
changelog + docs update
Edited by Derek Knox