Inconsistent Airflow Initialization
After a comment from @dmor about Airflow not showing up for her/Sid in running v1.0.1, I started a fresh project to test for myself:
- Run 1
- Run 2
meltano init test-1-0-1
- Went straight to Orchestration and didn't see preloader as is expected, instead saw (same as above screenshot)
- Hit "Refresh Airflow", changed routes, and hard refresh to no avail. I never saw my CLI update with Airflow-related status
- Airflow did not display
- Run 3
meltano init test-1-0-1
- Got below error pertaining to SQLite3 and Airflow:
(meltano-1-0-1) ✘ dknox-gitlab@Dereks-MacBook-Pro ~/Documents/Projects/test-1-0-1 meltano ui
[MainThread][INFO][2019-10-07 14:42:05,058] Context impl SQLiteImpl.
[MainThread][INFO][2019-10-07 14:42:05,059] Will assume non-transactional DDL.
System database up-to-date.
[MainThread][INFO][2019-10-07 14:42:05,081] Auto-compiling models in '/Users/dknox-gitlab/Documents/Projects/test-1-0-1/model'
[MainThread][INFO][2019-10-07 14:42:05,084] All workers started.
[2019-10-07 14:42:05 -0700] [77478] [INFO] Starting gunicorn 19.9.0
[2019-10-07 14:42:05 -0700] [77478] [INFO] Listening at: http://0.0.0.0:5000 (77478)
[2019-10-07 14:42:05 -0700] [77478] [INFO] Using worker: sync
[2019-10-07 14:42:05 -0700] [77481] [INFO] Booting worker with pid: 77481
[2019-10-07 14:42:05 -0700] [77482] [INFO] Booting worker with pid: 77482
[2019-10-07 14:42:05 -0700] [77483] [INFO] Booting worker with pid: 77483
[2019-10-07 14:42:05 -0700] [77484] [INFO] Booting worker with pid: 77484
Traceback (most recent call last):
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: job
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/bin/airflow", line 32, in <module>
args.func(args)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/bin/cli.py", line 991, in scheduler
job.run()
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/jobs.py", line 196, in run
session.commit()
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1023, in commit
self.transaction.commit()
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 487, in commit
self._prepare_impl()
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 466, in _prepare_impl
self.session.flush()
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2446, in flush
self._flush(objects)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2584, in _flush
transaction.rollback(_capture_exception=True)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
raise value
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2544, in _flush
flush_context.execute()
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 416, in execute
rec.execute(self)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 583, in execute
uow,
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
insert,
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1116, in _emit_insert_statements
statement, params
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute
return meth(self, multiparams, params)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
distilled_params,
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
e, statement, parameters, cursor, context
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
raise value.with_traceback(tb)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: job [SQL: 'INSERT INTO job (dag_id, state, job_type, start_date, end_date, latest_heartbeat, executor_class, hostname, unixname) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: (None, 'running', 'SchedulerJob', '2019-10-07 21:42:07.105541', None, '2019-10-07 21:42:07.105554', 'SequentialExecutor', 'Dereks-MacBook-Pro.local', 'dknox-gitlab')] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: connection
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/bin/airflow", line 32, in <module>
args.func(args)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/bin/cli.py", line 855, in webserver
app = cached_app_rbac(conf) if settings.RBAC else cached_app(conf)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/www/app.py", line 183, in cached_app
app = create_app(config, testing)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/www/app.py", line 72, in create_app
from airflow.www import views
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/www/views.py", line 2355, in <module>
class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/www/views.py", line 2446, in ChartModelView
'conn_id': _connection_ids()
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/utils/db.py", line 73, in wrapper
return func(*args, **kwargs)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/www/views.py", line 2352, in _connection_ids
.group_by(models.Connection.conn_id))]
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
return self._execute_and_instances(context)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute
return meth(self, multiparams, params)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
distilled_params,
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
e, statement, parameters, cursor, context
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
raise value.with_traceback(tb)
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/Users/dknox-gitlab/Documents/Projects/test-1-0-1/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: 'SELECT connection.conn_id AS connection_conn_id \nFROM connection GROUP BY connection.conn_id'] (Background on this error at: http://sqlalche.me/e/e3q8)
[http://0.0.0.0:5000/orchestrate] as <meltano.api.security.identity.FreeUser object at 0x120c51160>