airflow fails to start in v1.37.0
What is the current bug behavior?
What is happening now?
FileNotFound
error for .meltano/run/airflow/airflow.cfg
when invoking the airflow server on a new project.
Rolling back to 1.36.1, installing airbrake, and then upgrading again fixed the issue.
What is the expected correct behavior?
What should be happening?
The Airflow server should start
Steps to reproduce
How one can reproduce the issue?
meltano init sampleproj
cd sampleproj/
meltano add orchestrator airflow
meltano invoke airflow webserver -D
Relevant logs and/or screenshots
Please use code blocks (```) to format console output
[2020-06-25 21:42:30,152|W|MainThread|root] before_configure hook 'before_configure' has failed: [Errno 2] No such file or directory: '/Users/cjk/fixd/projects/meltano/sampleproj/.meltano/run/airflow/airflow.cfg'
[2020-06-25 21:42:30,152|E|MainThread|root] [Errno 2] No such file or directory: '/Users/cjk/fixd/projects/meltano/sampleproj/.meltano/run/airflow/airflow.cfg'
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 566, in move
os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/cjk/fixd/projects/meltano/sampleproj/.meltano/run/airflow/airflow.cfg' -> '/Users/cjk/fixd/projects/meltano/sampleproj/.meltano/orchestrators/airflow/airflow.cfg'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/cjk/fixd/meltano/venv/lib/python3.7/site-packages/meltano/cli/invoke.py", line 31, in invoke
service = invoker_factory(project, plugin, prepare_with_session=session)
File "/Users/cjk/fixd/meltano/venv/lib/python3.7/site-packages/meltano/core/plugin_invoker.py", line 27, in invoker_factory
invoker.prepare(prepare_with_session)
File "/Users/cjk/fixd/meltano/venv/lib/python3.7/site-packages/meltano/core/plugin_invoker.py", line 96, in prepare
with self.plugin.trigger_hooks("configure", self, session):
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/Users/cjk/fixd/meltano/venv/lib/python3.7/site-packages/meltano/core/behavior/hookable.py", line 68, in trigger_hooks
self.__class__.trigger(self, f"before_{hook_name}", *args, **kwargs)
File "/Users/cjk/fixd/meltano/venv/lib/python3.7/site-packages/meltano/core/behavior/hookable.py", line 93, in trigger
raise err
File "/Users/cjk/fixd/meltano/venv/lib/python3.7/site-packages/meltano/core/behavior/hookable.py", line 87, in trigger
hook_func(target, *args, **kwargs)
File "/Users/cjk/fixd/meltano/venv/lib/python3.7/site-packages/meltano/core/plugin/airflow.py", line 75, in before_configure
shutil.move(airflow_cfg_path, stub_path)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 580, in move
copy_function(src, real_dst)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 266, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/cjk/fixd/projects/meltano/sampleproj/.meltano/run/airflow/airflow.cfg'
An error occured: [Errno 2] No such file or directory: '/Users/cjk/fixd/projects/meltano/sampleproj/.meltano/run/airflow/airflow.cfg'.
Aborted!
Possible fixes
If you can, link to the line of code that might be responsible for the problem or suggest a fix
I suspect something in !1711 (merged) triggered this.
Further regression test
Ensure we automatically catch similar issues in the future
-
Write additional adequate test cases and submit test results -
Test results should be reviewed by a person from the team