Airflow is not always installed properly, so let's move the "meltano first run" behaviors to `meltano init`
The current behavior is that meltano init
only seeds some files, then you are ready to start playing with the tool. We designed it that way because users had to go through it when on-boarding, and we wanted to get them on the UI faster.
Instead, we now have first run
behaviors, where we expect the meltano ui
run to do some work before/after starting the Meltano UI. This logic is brittle, and leads to issues where:
- Airflow is not properly installed
- dbt is not properly installed
- Huge spike in resource consumption when
meltano ui
is run - Meltano UI secrets are not generated automatically
meltano ui setup
(flowBlocked by #1668 (closed)) - Database migrations are expected to run at each invocation, which is an anti-pattern
I think we should improve the meltano init
command, so that it does most of this.
The new flow
Hosted instances
On our hosted instances, meltano init
has already run and they should be good to go already.
Self-Hosted
pip install meltano
meltano init test-server
> Installing airflow
> Installing dbt
> Generating application secrets
> Creating the system database
> …
> Your project is ready at: http://localhost:5000
If we still want the "no-airflow" use-case, then we already have MELTANO_DISABLE_AIRFLOW
env variable that we can use, but I'm not sure it's worth to implement right now.
Drawbacks
meltano init
takes more time, which I think is totally fine as long as we properly explain to users why.
Implementation
Add each behavior described above in the meltano init
command.
Follow-up with #1157 (closed) in meltano upgrade