Orchestration MVC: Solo Piano - automatically refresh data daily
In the exploratory/planning sprint for orchestration, we established a vision for the MVC of orchestration... which we are calling "Solo Piano" (more details in the opportunity assessment)
While “Orchestration” is a large and complex feature, there are smaller wins we can offer users that will improve retention and support our goal to increase MAUI.
Solo Piano (MVC)
- User is able to toggle automatic refresh on/off at the single tap level
- User is able to choose the specific time when their daily refresh takes place
- User receives clear error messages and graceful error handling
- User expects to be able to come back daily and see up-to-date dashboards
@valexieva & @mbergeron to add)
Work Items to Deliver MVC prototype (MVC would be CLI focused meltano.yml should be the most basic config one wants, but we should allow users to configure on top as much as needed
-
Migrate Meltano to Airflow flask version -
Install Airflow with Meltano as a dependancy -
Get Meltano ELT running as a DAG - have E&L as 1 task, Transform as a second task -
meltano orchestrate tap-salesforce target-postgresql
- add the expression of scheduling meltano.yml (one/off would probably be a flag), we can give time, interval, similarly to cron and run by this command [ ]meltano orchestrate list
- read meltano.yml and show what is currently scheduled
Edited by Virjinia Alexieva